($INBOX_DIR/description missing)help / color / mirror / Atom feed
Re: [HACKERS] MERGE SQL Statement for PG11 81+ messages / 6 participants [nested] [flat]
* Re: [HACKERS] MERGE SQL Statement for PG11 @ 2018-03-22 23:13 Peter Geoghegan <[email protected]> 2018-03-23 11:56 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 0 siblings, 2 replies; 81+ messages in thread From: Peter Geoghegan @ 2018-03-22 23:13 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Thu, Mar 22, 2018 at 11:42 AM, Pavan Deolasee <[email protected]> wrote: > A slightly improved version attached. You still need to remove this change: > diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h > index a4574cd533..dbfb5d2a1a 100644 > --- a/src/include/miscadmin.h > +++ b/src/include/miscadmin.h > @@ -444,5 +444,6 @@ extern bool has_rolreplication(Oid roleid); > /* in access/transam/xlog.c */ > extern bool BackupInProgress(void); > extern void CancelBackup(void); > +extern int64 GetXactWALBytes(void); I see that we're still using two target RTEs in this latest revision, v23e -- the new approach to partitioning, which I haven't had time to study in more detail, has not produced a change there. This causes weird effects, such as the following: """ pg@~[20658]=# create table foo(bar int4); CREATE TABLE pg@~[20658]=# merge into foo f using (select 1 col) dd on f.bar=dd.col when matched then update set bar = f.barr + 1; ERROR: column f.barr does not exist LINE 1: ...n f.bar=dd.col when matched then update set bar = f.barr + 1... ^ HINT: Perhaps you meant to reference the column "f.bar" or the column "f.bar". """ While I think this this particular HINT buglet is pretty harmless, I continue to be concerned about the unintended consequences of having multiple RTEs for MERGE's target table. Each RTE comes from a different lookup path -- the first one goes through setTargetTable()'s parserOpenTable() + addRangeTableEntryForRelation() calls. The second one goes through transformFromClauseItem(), for the join, which ultimately ends up calling transformTableEntry()/addRangeTableEntry(). Consider commit 5f173040, which fixed a privilege escalation security bug around multiple name lookup. Could the approach taken by MERGE here introduce a similar security issue? Using GDB, I see two calls to RangeVarGetRelidExtended() when a simple MERGE is executed. They both have identical relation arguments, that look like this: (gdb) p *relation $4 = { type = T_RangeVar, catalogname = 0x0, schemaname = 0x0, relname = 0x5600ebdcafb0 "foo", inh = 1 '\001', relpersistence = 112 'p', alias = 0x5600ebdcb048, location = 11 } This seems like something that needs to be explained, at a minimum. Even if I'm completely wrong about there being a security hazard, maybe the suggestion that there might be still gives you some idea of what I mean about unintended consequences. -- Peter Geoghegan ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> @ 2018-03-23 11:56 ` Pavan Deolasee <[email protected]> 2018-03-24 06:15 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 1 sibling, 1 reply; 81+ messages in thread From: Pavan Deolasee @ 2018-03-23 11:56 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Fri, Mar 23, 2018 at 4:43 AM, Peter Geoghegan <[email protected]> wrote: > On Thu, Mar 22, 2018 at 11:42 AM, Pavan Deolasee > <[email protected]> wrote: > > A slightly improved version attached. > > You still need to remove this change: > > > diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h > > index a4574cd533..dbfb5d2a1a 100644 > > --- a/src/include/miscadmin.h > > +++ b/src/include/miscadmin.h > > @@ -444,5 +444,6 @@ extern bool has_rolreplication(Oid roleid); > > /* in access/transam/xlog.c */ > > extern bool BackupInProgress(void); > > extern void CancelBackup(void); > > +extern int64 GetXactWALBytes(void); > Sigh. Fixed in the recent version. > > I see that we're still using two target RTEs in this latest revision, > v23e -- the new approach to partitioning, which I haven't had time to > study in more detail, has not produced a change there. Yes, we continue to use two RTEs because I don't have any brighter idea than that to handle the case of partitioned table and right outer join. As I explained sometime back, this is necessary to ensure that we don't produce duplicate rows when a partition is joined with the source and then a second partition is joined again with the source. Now I don't know if we can run a join query and still have a single RTE, but that looks improbable and wrong. This causes > weird effects, such as the following: > > """ > pg@~[20658]=# create table foo(bar int4); > CREATE TABLE > pg@~[20658]=# merge into foo f using (select 1 col) dd on f.bar=dd.col > when matched then update set bar = f.barr + 1; > ERROR: column f.barr does not exist > LINE 1: ...n f.bar=dd.col when matched then update set bar = f.barr + 1... > ^ > HINT: Perhaps you meant to reference the column "f.bar" or the column > "f.bar". > > """ > > While I think this this particular HINT buglet is pretty harmless, I > continue to be concerned about the unintended consequences of having > multiple RTEs for MERGE's target table. Each RTE comes from a > different lookup path -- the first one goes through setTargetTable()'s > parserOpenTable() + addRangeTableEntryForRelation() calls. The second > one goes through transformFromClauseItem(), for the join, which > ultimately ends up calling transformTableEntry()/addRangeTableEntry(). > How's it different than running a INSERT query with the target table again specified in a subquery producing the rows to be inserted? For example, postgres=# insert into target as t select sid from source s join target t on t.ttid = s.sid; ERROR: column t.ttid does not exist LINE 1: ...rget as t select sid from source join target t on t.ttid = s... ^ HINT: Perhaps you meant to reference the column "t.tid" or the column "t.tid". postgres=# This produces a very similar looking HINT as your test above. I am certain that "target" table gets two RTEs, exactly via the same code paths as you discussed above. So if this is not a problem for INSERT, why it would be a problem for MERGE? May be I am missing a point here. > Consider commit 5f173040, which fixed a privilege escalation security > bug around multiple name lookup. Could the approach taken by MERGE > here introduce a similar security issue? > > Using GDB, I see two calls to RangeVarGetRelidExtended() when a simple > MERGE is executed. They both have identical relation arguments, that > look like this: > > (gdb) p *relation > $4 = { > type = T_RangeVar, > catalogname = 0x0, > schemaname = 0x0, > relname = 0x5600ebdcafb0 "foo", > inh = 1 '\001', > relpersistence = 112 'p', > alias = 0x5600ebdcb048, > location = 11 > } > > This seems like something that needs to be explained, at a minimum. > Even if I'm completely wrong about there being a security hazard, > maybe the suggestion that there might be still gives you some idea of > what I mean about unintended consequences. > Ok. I will try to explain it better and also think about the security hazards. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-23 11:56 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> @ 2018-03-24 06:15 ` Peter Geoghegan <[email protected]> 2018-03-24 06:35 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Peter Geoghegan @ 2018-03-24 06:15 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Fri, Mar 23, 2018 at 4:56 AM, Pavan Deolasee <[email protected]> wrote: > postgres=# insert into target as t select sid from source s join target t on > t.ttid = s.sid; > ERROR: column t.ttid does not exist > LINE 1: ...rget as t select sid from source join target t on t.ttid = s... > ^ > HINT: Perhaps you meant to reference the column "t.tid" or the column > "t.tid". > postgres=# > > This produces a very similar looking HINT as your test above. I am certain > that "target" table gets two RTEs, exactly via the same code paths as you > discussed above. So if this is not a problem for INSERT, why it would be a > problem for MERGE? May be I am missing a point here. I agree that this is very similar, as far as the RTEs go. What is dissimilar is the fact that there is hard-coded knowledge of both through parsing, planning, and execution. It's everything, taken together. ResultRelInfo has a ri_mergeTargetRTI field, which seems to be used instead of ri_RangeTableIndex in some contexts but not others. What might the interactions with something like GetInsertedColumns() and GetUpdatedColumns() be? Is that explained anywhere? In general, I think that there is potential for things to break in subtle ways. >> This seems like something that needs to be explained, at a minimum. >> Even if I'm completely wrong about there being a security hazard, >> maybe the suggestion that there might be still gives you some idea of >> what I mean about unintended consequences. > > > Ok. I will try to explain it better and also think about the security > hazards. I realize that I'm giving you a somewhat vague problem, without offering any real help on a solution. For what it's worth, I don't feel great about that, but I don't know enough about partitioning in general and your approach to partitioning for MERGE in particular to be more constructive. That said, checking that an issue like the one fixed by 5f173040 cannot recur here is one concrete thing you could do. Documenting/explaining the ri_RangeTableIndex/ri_mergeTargetRTI divide is another. The comment above ri_mergeTargetRTI is totally inadequate. -- Peter Geoghegan ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-23 11:56 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-24 06:15 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> @ 2018-03-24 06:35 ` Peter Geoghegan <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Peter Geoghegan @ 2018-03-24 06:35 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Fri, Mar 23, 2018 at 11:15 PM, Peter Geoghegan <[email protected]> wrote: > I agree that this is very similar, as far as the RTEs go. What is > dissimilar is the fact that there is hard-coded knowledge of both > through parsing, planning, and execution. It's everything, taken > together. > > ResultRelInfo has a ri_mergeTargetRTI field, which seems to be used > instead of ri_RangeTableIndex in some contexts but not others. What > might the interactions with something like GetInsertedColumns() and > GetUpdatedColumns() be? Is that explained anywhere? In general, I > think that there is potential for things to break in subtle ways. I just realized that there were no tests added to privileges.sql. You only have a small number of GRANT tests in merge.sql, for relation-level privileges, not column-level privileges. IOW, this area is totally untested. -- Peter Geoghegan ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> @ 2018-03-24 12:16 ` Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:28 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 1 sibling, 2 replies; 81+ messages in thread From: Robert Haas @ 2018-03-24 12:16 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Simon Riggs <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Thu, Mar 22, 2018 at 7:13 PM, Peter Geoghegan <[email protected]> wrote: > While I think this this particular HINT buglet is pretty harmless, I > continue to be concerned about the unintended consequences of having > multiple RTEs for MERGE's target table. Each RTE comes from a > different lookup path -- the first one goes through setTargetTable()'s > parserOpenTable() + addRangeTableEntryForRelation() calls. The second > one goes through transformFromClauseItem(), for the join, which > ultimately ends up calling transformTableEntry()/addRangeTableEntry(). > Consider commit 5f173040, which fixed a privilege escalation security > bug around multiple name lookup. Could the approach taken by MERGE > here introduce a similar security issue? Yeah, that seems really bad. I don't think there is a huge problem with having multiple RTEs; for example, we very commonly end up with both rte->inh and !rte->inh RTEs for the same table, and that is OK. However, generating those RTEs by doing multiple name lookups for the same table is a big problem. Imagine, for example, that a user has a search_path of a, b and that there is a table b.foo. The user does a merge on foo. Between the first name lookup and the second, somebody creates a.foo. Now, potentially, half of the MERGE statement is going to be running against b.foo and the other half against a.foo. I don't know whether that will crash or bomb out with a strange error or just make some unexpected modification to one of those tables, but the behavior, even if not insecure, will certainly be wrong. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> @ 2018-03-24 12:27 ` Robert Haas <[email protected]> 2018-03-24 17:48 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 1 sibling, 2 replies; 81+ messages in thread From: Robert Haas @ 2018-03-24 12:27 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Simon Riggs <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Sat, Mar 24, 2018 at 8:16 AM, Robert Haas <[email protected]> wrote: > On Thu, Mar 22, 2018 at 7:13 PM, Peter Geoghegan <[email protected]> wrote: >> While I think this this particular HINT buglet is pretty harmless, I >> continue to be concerned about the unintended consequences of having >> multiple RTEs for MERGE's target table. Each RTE comes from a >> different lookup path -- the first one goes through setTargetTable()'s >> parserOpenTable() + addRangeTableEntryForRelation() calls. The second >> one goes through transformFromClauseItem(), for the join, which >> ultimately ends up calling transformTableEntry()/addRangeTableEntry(). >> Consider commit 5f173040, which fixed a privilege escalation security >> bug around multiple name lookup. Could the approach taken by MERGE >> here introduce a similar security issue? > > Yeah, that seems really bad. I don't think there is a huge problem > with having multiple RTEs; for example, we very commonly end up with > both rte->inh and !rte->inh RTEs for the same table, and that is OK. > However, generating those RTEs by doing multiple name lookups for the > same table is a big problem. Imagine, for example, that a user has a > search_path of a, b and that there is a table b.foo. The user does a > merge on foo. Between the first name lookup and the second, somebody > creates a.foo. Now, potentially, half of the MERGE statement is going > to be running against b.foo and the other half against a.foo. I don't > know whether that will crash or bomb out with a strange error or just > make some unexpected modification to one of those tables, but the > behavior, even if not insecure, will certainly be wrong. If it's possible to identify the two OIDs that are supposed to match and cross-check that the OIDs are the same, then we could just bomb out with an error if they aren't. That's not lovely, and is basically a hack, but it's possible that no better fix is possible in the time we have, and it's wouldn't be any worse than this crock from copy.c: if (!list_member_oid(plan->relationOids, queryRelId)) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), errmsg("relation referenced by COPY statement has changed"))); Uggh, that code makes me hold my nose every time I look at it. But it's a cheap fix. (Hmm... I wonder if that's really an adequate fix for the problem in COPY, if we can't verify that the OID in question plays the right role in the query, rather than just that it's there somewhere. Anyway, if we can reliably identify the two OIDs to be compared, that's certainly good enough.) -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> @ 2018-03-24 17:48 ` Peter Geoghegan <[email protected]> 1 sibling, 0 replies; 81+ messages in thread From: Peter Geoghegan @ 2018-03-24 17:48 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Simon Riggs <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Sat, Mar 24, 2018 at 5:27 AM, Robert Haas <[email protected]> wrote: > If it's possible to identify the two OIDs that are supposed to match > and cross-check that the OIDs are the same, then we could just bomb > out with an error if they aren't. That's not lovely, and is basically > a hack, but it's possible that no better fix is possible in the time > we have, and it's wouldn't be any worse than this crock from copy.c: > > if (!list_member_oid(plan->relationOids, queryRelId)) > ereport(ERROR, > (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > errmsg("relation referenced by COPY statement > has changed"))); That's definitely all we have time for. The only alternative is to rip out support for partitioning, as partitioning is the only thing that necessitates the use of multiple RTEs. I don't think it would make sense to use a second RTE only when needed. -- Peter Geoghegan ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> @ 2018-03-26 09:53 ` Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 15:09 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 1 sibling, 2 replies; 81+ messages in thread From: Simon Riggs @ 2018-03-26 09:53 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 24 March 2018 at 12:27, Robert Haas <[email protected]> wrote: > On Sat, Mar 24, 2018 at 8:16 AM, Robert Haas <[email protected]> wrote: >> On Thu, Mar 22, 2018 at 7:13 PM, Peter Geoghegan <[email protected]> wrote: >>> While I think this this particular HINT buglet is pretty harmless, I >>> continue to be concerned about the unintended consequences of having >>> multiple RTEs for MERGE's target table. Each RTE comes from a >>> different lookup path -- the first one goes through setTargetTable()'s >>> parserOpenTable() + addRangeTableEntryForRelation() calls. The second >>> one goes through transformFromClauseItem(), for the join, which >>> ultimately ends up calling transformTableEntry()/addRangeTableEntry(). >>> Consider commit 5f173040, which fixed a privilege escalation security >>> bug around multiple name lookup. Could the approach taken by MERGE >>> here introduce a similar security issue? >> >> Yeah, that seems really bad. I don't think there is a huge problem >> with having multiple RTEs; for example, we very commonly end up with >> both rte->inh and !rte->inh RTEs for the same table, and that is OK. >> However, generating those RTEs by doing multiple name lookups for the >> same table is a big problem. Imagine, for example, that a user has a >> search_path of a, b and that there is a table b.foo. The user does a >> merge on foo. Between the first name lookup and the second, somebody >> creates a.foo. Now, potentially, half of the MERGE statement is going >> to be running against b.foo and the other half against a.foo. I don't >> know whether that will crash or bomb out with a strange error or just >> make some unexpected modification to one of those tables, but the >> behavior, even if not insecure, will certainly be wrong. > > If it's possible to identify the two OIDs that are supposed to match > and cross-check that the OIDs are the same, then we could just bomb > out with an error if they aren't. Since we now have MVCC catalog scans, all the name lookups are performed using the same snapshot so in the above scenario the newly created object would be invisible to the second name lookup. So I don't see anyway for the ERROR to occur and hence no need for a cross check, for UPDATE or MERGE. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-26 14:39 ` Pavan Deolasee <[email protected]> 2018-03-26 15:05 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 1 sibling, 2 replies; 81+ messages in thread From: Pavan Deolasee @ 2018-03-26 14:39 UTC (permalink / raw) To: Amit Langote <[email protected]>; +Cc: Simon Riggs <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Fri, Mar 23, 2018 at 4:37 PM, Pavan Deolasee <[email protected]> wrote: > > > On Fri, Mar 23, 2018 at 12:57 PM, Amit Langote < > [email protected]> wrote: > >> >> Also, it seems that the delta patch I sent in the last email didn't >> contain all the changes I had to make. It didn't contain, for example, >> replacing adjust_and_expand_inherited_tlist() with >> adjust_partition_tlist(). I guess you'll know when you rebase anyway. >> > > Yes, I am planning to fix that once the ON CONFLICT patch is > ready/committed. > > Now that ON CONFLICT patch is in, here are rebased patches. The second patch is to add support for CTE (thanks Peter). Apart from rebase, the following things are fixed/improved: - Added test cases for column level privileges as suggested by Peter. One problem got discovered during the process. Since we expand and track source relation targetlist, the exiting code was demanding SELECT privileges on all attributes, even though MERGE is only referencing a few attributes on which the user has privilege. Fixed that by disassociating expansion from the actual referencing. - Added a test case for RLS where SELECT policy actually hides some rows, as suggested by Stephen in the past - Added check to compare result relation's and merge target relation's OIDs, as suggested by Robert. Simon thinks it's not necessary given that we now scan catalog using MVCC snapshot. So will leave it to his discretion when he takes it up for commit - Improved explanation regarding why we need a second RTE for merge target relation and general cleanup/improvements in that area I think it will be a good idea to send any further patches as add-on patches for reviewer/committer's sake. I will do that unless someone disagrees. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services Attachments: [application/octet-stream] v25-0002-Add-support-for-CTE.patch (13.5K, ../../CABOikdM9hvMCt_nsunLpk1KhQQVwW+-cGH7BfMoh4HqMdbbn-g@mail.gmail.com/3-v25-0002-Add-support-for-CTE.patch) download | inline diff: From 99fda84854b6f25ae573202d2803ff96e39719ea Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Mon, 26 Mar 2018 18:56:50 +0530 Subject: [PATCH v25 2/2] Add support for CTE --- src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/parser/gram.y | 11 +-- src/backend/parser/parse_merge.c | 9 +++ src/include/nodes/parsenodes.h | 1 + src/test/regress/expected/merge.out | 3 - src/test/regress/expected/with.out | 132 ++++++++++++++++++++++++++++++++++++ src/test/regress/sql/with.sql | 51 ++++++++++++++ 9 files changed, 203 insertions(+), 8 deletions(-) diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 770ed3b1a8..c3efca3c45 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -3055,6 +3055,7 @@ _copyMergeStmt(const MergeStmt *from) COPY_NODE_FIELD(source_relation); COPY_NODE_FIELD(join_condition); COPY_NODE_FIELD(mergeActionList); + COPY_NODE_FIELD(withClause); return newnode; } diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 5a0151eece..45ceba2830 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -1051,6 +1051,7 @@ _equalMergeStmt(const MergeStmt *a, const MergeStmt *b) COMPARE_NODE_FIELD(source_relation); COMPARE_NODE_FIELD(join_condition); COMPARE_NODE_FIELD(mergeActionList); + COMPARE_NODE_FIELD(withClause); return true; } diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 68e2cec66e..7106765e2b 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3446,6 +3446,8 @@ raw_expression_tree_walker(Node *node, return true; if (walker(stmt->mergeActionList, context)) return true; + if (walker(stmt->withClause, context)) + return true; } break; case T_SelectStmt: diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index ebca5f3eb7..2f21571915 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -11105,17 +11105,18 @@ set_target_list: *****************************************************************************/ MergeStmt: - MERGE INTO relation_expr_opt_alias + opt_with_clause MERGE INTO relation_expr_opt_alias USING table_ref ON a_expr merge_when_list { MergeStmt *m = makeNode(MergeStmt); - m->relation = $3; - m->source_relation = $5; - m->join_condition = $7; - m->mergeActionList = $8; + m->withClause = $1; + m->relation = $4; + m->source_relation = $6; + m->join_condition = $8; + m->mergeActionList = $9; $$ = (Node *)m; } diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c index b6e0c46656..26a6692231 100644 --- a/src/backend/parser/parse_merge.c +++ b/src/backend/parser/parse_merge.c @@ -24,6 +24,7 @@ #include "parser/parsetree.h" #include "parser/parser.h" #include "parser/parse_clause.h" +#include "parser/parse_cte.h" #include "parser/parse_merge.h" #include "parser/parse_relation.h" #include "parser/parse_target.h" @@ -203,6 +204,14 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) qry->commandType = CMD_MERGE; + /* process the WITH clause independently of all else */ + if (stmt->withClause) + { + qry->hasRecursive = stmt->withClause->recursive; + qry->cteList = transformWithClause(pstate, stmt->withClause); + qry->hasModifyingCTE = pstate->p_hasModifyingCTE; + } + /* * Check WHEN clauses for permissions and sanity */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 0c904f4d7f..36e6e2e976 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -1519,6 +1519,7 @@ typedef struct MergeStmt Node *source_relation; /* source relation */ Node *join_condition; /* join condition between source and target */ List *mergeActionList; /* list of MergeAction(s) */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeAction diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out index 05c4287078..411ccd0e66 100644 --- a/src/test/regress/expected/merge.out +++ b/src/test/regress/expected/merge.out @@ -1191,9 +1191,6 @@ WHEN NOT MATCHED THEN WHEN MATCHED AND tid < 2 THEN DELETE ; -ERROR: syntax error at or near "MERGE" -LINE 4: MERGE INTO sq_target t - ^ ROLLBACK; -- RETURNING BEGIN; diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out index 2a2085556b..543ca4f272 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -1904,6 +1904,138 @@ RETURNING k, v; (0 rows) DROP TABLE withz; +-- WITH referenced by MERGE statement +CREATE TABLE m AS SELECT i AS k, (i || ' v')::text v FROM generate_series(1, 16, 3) i; +ALTER TABLE m ADD UNIQUE (k); +-- Basic: +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 0; + k | v +---+---------------------- + 0 | merge source SubPlan +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + QUERY PLAN +------------------------------------------------------------------- + Merge on public.m + CTE cte_basic + -> Result + Output: 1, 'cte_basic val'::text + -> Hash Right Join + Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = o.k) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: o.k, o.v, o.* + -> Subquery Scan on o + Output: o.k, o.v, o.* + -> Result + Output: 0, 'merge source SubPlan'::text + SubPlan 2 + -> Limit + Output: ((cte_basic.b || ' merge update'::text)) + -> CTE Scan on cte_basic + Output: (cte_basic.b || ' merge update'::text) + Filter: (cte_basic.a = m.k) +(21 rows) + +-- InitPlan +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 1; + k | v +---+--------------------------- + 1 | cte_init val merge update +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + QUERY PLAN +-------------------------------------------------------------------- + Merge on public.m + CTE cte_init + -> Result + Output: 1, 'cte_init val'::text + InitPlan 2 (returns $1) + -> Limit + Output: ((cte_init.b || ' merge update'::text)) + -> CTE Scan on cte_init + Output: (cte_init.b || ' merge update'::text) + Filter: (cte_init.a = 1) + -> Hash Right Join + Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = o.k) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: o.k, o.v, o.* + -> Subquery Scan on o + Output: o.k, o.v, o.* + -> Result + Output: 1, 'merge source InitPlan'::text +(21 rows) + +-- MERGE source comes from CTE: +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); +-- Examine +SELECT * FROM m where k = 15; + k | v +----+-------------------------------------------------------------- + 15 | merge_source_cte val(15,"merge_source_cte val") merge insert +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------- + Merge on public.m + CTE merge_source_cte + -> Result + Output: 15, 'merge_source_cte val'::text + InitPlan 2 (returns $1) + -> CTE Scan on merge_source_cte merge_source_cte_1 + Output: ((merge_source_cte_1.b || (merge_source_cte_1.*)::text) || ' merge update'::text) + Filter: (merge_source_cte_1.a = 15) + InitPlan 3 (returns $2) + -> CTE Scan on merge_source_cte merge_source_cte_2 + Output: ((merge_source_cte_2.*)::text || ' merge insert'::text) + -> Hash Right Join + Output: merge_source_cte.a, merge_source_cte.b, ROW(merge_source_cte.a, merge_source_cte.b), m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = merge_source_cte.a) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: merge_source_cte.a, merge_source_cte.b + -> CTE Scan on merge_source_cte + Output: merge_source_cte.a, merge_source_cte.b +(20 rows) + +DROP TABLE m; -- check that run to completion happens in proper ordering TRUNCATE TABLE y; INSERT INTO y SELECT generate_series(1, 3); diff --git a/src/test/regress/sql/with.sql b/src/test/regress/sql/with.sql index f85645efde..dd73b334de 100644 --- a/src/test/regress/sql/with.sql +++ b/src/test/regress/sql/with.sql @@ -862,6 +862,57 @@ RETURNING k, v; DROP TABLE withz; +-- WITH referenced by MERGE statement +CREATE TABLE m AS SELECT i AS k, (i || ' v')::text v FROM generate_series(1, 16, 3) i; +ALTER TABLE m ADD UNIQUE (k); + +-- Basic: +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 0; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + +-- InitPlan +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 1; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + +-- MERGE source comes from CTE: +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); +-- Examine +SELECT * FROM m where k = 15; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); + +DROP TABLE m; + -- check that run to completion happens in proper ordering TRUNCATE TABLE y; -- 2.14.3 (Apple Git-98) [application/octet-stream] v25-0001-Version-25-of-MERGE-patch-based-on-ON-CONFLICT-D.patch (322.6K, ../../CABOikdM9hvMCt_nsunLpk1KhQQVwW+-cGH7BfMoh4HqMdbbn-g@mail.gmail.com/4-v25-0001-Version-25-of-MERGE-patch-based-on-ON-CONFLICT-D.patch) download | inline diff: From ea30e14dd97248420f847ba532d33c21d1922f5b Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Sat, 24 Mar 2018 17:25:04 +0530 Subject: [PATCH v25 1/2] Version 25 of MERGE patch, based on ON CONFLICT DO UPDATE work Add a check for consistent lookup for result and mergeTarget relation, some comments and fixes Add tests to privileges.sql and ensure that we don't demand SELECT privileges on columns from source side Add additional test case for RLS --- contrib/test_decoding/expected/ddl.out | 46 + contrib/test_decoding/sql/ddl.sql | 16 + doc/src/sgml/libpq.sgml | 8 +- doc/src/sgml/mvcc.sgml | 28 +- doc/src/sgml/plpgsql.sgml | 3 +- doc/src/sgml/ref/allfiles.sgml | 1 + doc/src/sgml/ref/insert.sgml | 11 +- doc/src/sgml/reference.sgml | 1 + doc/src/sgml/trigger.sgml | 20 + src/backend/access/heap/heapam.c | 3 + src/backend/catalog/sql_features.txt | 6 +- src/backend/commands/explain.c | 30 + src/backend/commands/prepare.c | 1 + src/backend/commands/trigger.c | 156 +- src/backend/executor/Makefile | 2 +- src/backend/executor/README | 10 + src/backend/executor/execMain.c | 17 + src/backend/executor/execPartition.c | 116 ++ src/backend/executor/execReplication.c | 4 +- src/backend/executor/nodeMerge.c | 566 +++++++ src/backend/executor/nodeModifyTable.c | 378 ++++- src/backend/executor/spi.c | 3 + src/backend/nodes/copyfuncs.c | 40 + src/backend/nodes/equalfuncs.c | 32 + src/backend/nodes/nodeFuncs.c | 48 +- src/backend/nodes/outfuncs.c | 25 + src/backend/nodes/readfuncs.c | 6 + src/backend/optimizer/plan/createplan.c | 22 +- src/backend/optimizer/plan/planner.c | 29 +- src/backend/optimizer/plan/setrefs.c | 59 + src/backend/optimizer/prep/preptlist.c | 40 +- src/backend/optimizer/util/pathnode.c | 11 +- src/backend/optimizer/util/plancat.c | 4 + src/backend/optimizer/util/relnode.c | 1 - src/backend/parser/Makefile | 2 +- src/backend/parser/analyze.c | 18 +- src/backend/parser/gram.y | 158 +- src/backend/parser/parse_agg.c | 10 + src/backend/parser/parse_clause.c | 39 +- src/backend/parser/parse_collate.c | 1 + src/backend/parser/parse_expr.c | 3 + src/backend/parser/parse_func.c | 3 + src/backend/parser/parse_merge.c | 670 ++++++++ src/backend/parser/parse_relation.c | 10 + src/backend/rewrite/rewriteHandler.c | 112 +- src/backend/rewrite/rowsecurity.c | 97 ++ src/backend/tcop/pquery.c | 5 + src/backend/tcop/utility.c | 16 + src/include/access/heapam.h | 1 + src/include/commands/trigger.h | 6 +- src/include/executor/execPartition.h | 1 + src/include/executor/nodeMerge.h | 22 + src/include/executor/nodeModifyTable.h | 21 + src/include/executor/spi.h | 1 + src/include/nodes/execnodes.h | 64 +- src/include/nodes/nodes.h | 6 +- src/include/nodes/parsenodes.h | 39 +- src/include/nodes/plannodes.h | 8 +- src/include/nodes/relation.h | 7 +- src/include/optimizer/pathnode.h | 7 +- src/include/parser/analyze.h | 5 + src/include/parser/kwlist.h | 2 + src/include/parser/parse_clause.h | 5 +- src/include/parser/parse_merge.h | 19 + src/include/parser/parse_node.h | 6 +- src/include/rewrite/rewriteHandler.h | 1 + src/interfaces/libpq/fe-exec.c | 9 +- src/pl/plpgsql/src/pl_exec.c | 5 +- src/pl/plpgsql/src/pl_gram.y | 8 + src/pl/plpgsql/src/pl_scanner.c | 1 + src/pl/plpgsql/src/plpgsql.h | 4 +- src/test/isolation/expected/merge-delete.out | 97 ++ .../isolation/expected/merge-insert-update.out | 84 + .../isolation/expected/merge-match-recheck.out | 106 ++ src/test/isolation/expected/merge-update.out | 213 +++ src/test/isolation/isolation_schedule | 4 + src/test/isolation/specs/merge-delete.spec | 51 + src/test/isolation/specs/merge-insert-update.spec | 52 + src/test/isolation/specs/merge-match-recheck.spec | 79 + src/test/isolation/specs/merge-update.spec | 132 ++ src/test/regress/expected/identity.out | 55 + src/test/regress/expected/merge.out | 1599 ++++++++++++++++++++ src/test/regress/expected/privileges.out | 98 ++ src/test/regress/expected/rowsecurity.out | 182 +++ src/test/regress/expected/rules.out | 31 + src/test/regress/expected/triggers.out | 48 + src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/identity.sql | 45 + src/test/regress/sql/merge.sql | 1068 +++++++++++++ src/test/regress/sql/privileges.sql | 108 ++ src/test/regress/sql/rowsecurity.sql | 156 ++ src/test/regress/sql/rules.sql | 33 + src/test/regress/sql/triggers.sql | 47 + src/tools/pgindent/typedefs.list | 3 + 95 files changed, 7280 insertions(+), 149 deletions(-) create mode 100644 src/backend/executor/nodeMerge.c create mode 100644 src/backend/parser/parse_merge.c create mode 100644 src/include/executor/nodeMerge.h create mode 100644 src/include/parser/parse_merge.h create mode 100644 src/test/isolation/expected/merge-delete.out create mode 100644 src/test/isolation/expected/merge-insert-update.out create mode 100644 src/test/isolation/expected/merge-match-recheck.out create mode 100644 src/test/isolation/expected/merge-update.out create mode 100644 src/test/isolation/specs/merge-delete.spec create mode 100644 src/test/isolation/specs/merge-insert-update.spec create mode 100644 src/test/isolation/specs/merge-match-recheck.spec create mode 100644 src/test/isolation/specs/merge-update.spec create mode 100644 src/test/regress/expected/merge.out create mode 100644 src/test/regress/sql/merge.sql diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index b7c76469fc..79c359d6e3 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -192,6 +192,52 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (33 rows) +-- MERGE support +BEGIN; +MERGE INTO replication_example t + USING (SELECT i as id, i as data, i as num FROM generate_series(-20, 5) i) s + ON t.id = s.id + WHEN MATCHED AND t.id < 0 THEN + UPDATE SET somenum = somenum + 1 + WHEN MATCHED AND t.id >= 0 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.*); +COMMIT; +/* display results */ +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); + data +-------------------------------------------------------------------------------------------------------------------------------------------------- + BEGIN + table public.replication_example: INSERT: id[integer]:-20 somedata[integer]:-20 somenum[integer]:-20 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-19 somedata[integer]:-19 somenum[integer]:-19 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-18 somedata[integer]:-18 somenum[integer]:-18 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-17 somedata[integer]:-17 somenum[integer]:-17 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-16 somedata[integer]:-16 somenum[integer]:-16 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-15 somedata[integer]:-15 somenum[integer]:-14 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-14 somedata[integer]:-14 somenum[integer]:-13 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-13 somedata[integer]:-13 somenum[integer]:-12 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-12 somedata[integer]:-12 somenum[integer]:-11 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-11 somedata[integer]:-11 somenum[integer]:-10 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-10 somedata[integer]:-10 somenum[integer]:-9 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-9 somedata[integer]:-9 somenum[integer]:-8 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-8 somedata[integer]:-8 somenum[integer]:-7 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-7 somedata[integer]:-7 somenum[integer]:-6 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-6 somedata[integer]:-6 somenum[integer]:-5 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-5 somedata[integer]:-5 somenum[integer]:-4 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-4 somedata[integer]:-4 somenum[integer]:-3 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-3 somedata[integer]:-3 somenum[integer]:-2 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-2 somedata[integer]:-2 somenum[integer]:-1 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-1 somedata[integer]:-1 somenum[integer]:0 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: DELETE: id[integer]:0 + table public.replication_example: DELETE: id[integer]:1 + table public.replication_example: DELETE: id[integer]:2 + table public.replication_example: DELETE: id[integer]:3 + table public.replication_example: DELETE: id[integer]:4 + table public.replication_example: DELETE: id[integer]:5 + COMMIT +(28 rows) + CREATE TABLE tr_unique(id2 serial unique NOT NULL, data int); INSERT INTO tr_unique(data) VALUES(10); ALTER TABLE tr_unique RENAME TO tr_pkey; diff --git a/contrib/test_decoding/sql/ddl.sql b/contrib/test_decoding/sql/ddl.sql index c4b10a4cf9..0e608b252f 100644 --- a/contrib/test_decoding/sql/ddl.sql +++ b/contrib/test_decoding/sql/ddl.sql @@ -93,6 +93,22 @@ COMMIT; /* display results */ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- MERGE support +BEGIN; +MERGE INTO replication_example t + USING (SELECT i as id, i as data, i as num FROM generate_series(-20, 5) i) s + ON t.id = s.id + WHEN MATCHED AND t.id < 0 THEN + UPDATE SET somenum = somenum + 1 + WHEN MATCHED AND t.id >= 0 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.*); +COMMIT; + +/* display results */ +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); + CREATE TABLE tr_unique(id2 serial unique NOT NULL, data int); INSERT INTO tr_unique(data) VALUES(10); ALTER TABLE tr_unique RENAME TO tr_pkey; diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml index 1fd5dd9fca..de03046f5c 100644 --- a/doc/src/sgml/libpq.sgml +++ b/doc/src/sgml/libpq.sgml @@ -3873,9 +3873,11 @@ char *PQcmdTuples(PGresult *res); <structname>PGresult</structname>. This function can only be used following the execution of a <command>SELECT</command>, <command>CREATE TABLE AS</command>, <command>INSERT</command>, <command>UPDATE</command>, <command>DELETE</command>, - <command>MOVE</command>, <command>FETCH</command>, or <command>COPY</command> statement, - or an <command>EXECUTE</command> of a prepared query that contains an - <command>INSERT</command>, <command>UPDATE</command>, or <command>DELETE</command> statement. + <command>MERGE</command>, <command>MOVE</command>, <command>FETCH</command>, + or <command>COPY</command> statement, or an <command>EXECUTE</command> of a + prepared query that contains an <command>INSERT</command>, + <command>UPDATE</command>, <command>DELETE</command> + or <command>MERGE</command> statement. If the command that generated the <structname>PGresult</structname> was anything else, <function>PQcmdTuples</function> returns an empty string. The caller should not free the return value directly. It will be freed when diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 24613e3c75..0e3e89af56 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -422,6 +422,31 @@ COMMIT; <literal>11</literal>, which no longer matches the criteria. </para> + <para> + The <command>MERGE</command> allows the user to specify various combinations + of <command>INSERT</command>, <command>UPDATE</command> or + <command>DELETE</command> subcommands. A <command>MERGE</command> command + with both <command>INSERT</command> and <command>UPDATE</command> + subcommands looks similar to <command>INSERT</command> with an + <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee + that either <command>INSERT</command> and <command>UPDATE</command> will occur. + + If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + but the join condition still passes for the current target and the current + source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + and perform its action on the latest version of the row, using standard + EvalPlanQual. MERGE actions can be conditional, so conditions must be + re-evaluated on the latest row, starting from the first action. + + On the other hand, if the row is concurrently updated or deleted so that + the join condition fails, then MERGE will execute a NOT MATCHED action, if it + exists and the AND WHEN qual evaluates to true. + + If MERGE attempts an INSERT and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. MERGE + does not attempt to avoid the ERROR by attempting an UPDATE. + </para> + <para> Because Read Committed mode starts each command with a new snapshot that includes all transactions committed up to that instant, @@ -900,7 +925,8 @@ ERROR: could not serialize access due to read/write dependencies among transact <para> The commands <command>UPDATE</command>, - <command>DELETE</command>, and <command>INSERT</command> + <command>DELETE</command>, <command>INSERT</command> and + <command>MERGE</command> acquire this lock mode on the target table (in addition to <literal>ACCESS SHARE</literal> locks on any other referenced tables). In general, this lock mode will be acquired by any diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml index 7ed926fd51..67b22a0d04 100644 --- a/doc/src/sgml/plpgsql.sgml +++ b/doc/src/sgml/plpgsql.sgml @@ -1246,7 +1246,7 @@ EXECUTE format('SELECT count(*) FROM %I ' </programlisting> Another restriction on parameter symbols is that they only work in <command>SELECT</command>, <command>INSERT</command>, <command>UPDATE</command>, and - <command>DELETE</command> commands. In other statement + <command>DELETE</command> and <command>MERGE</command> commands. In other statement types (generically called utility statements), you must insert values textually even if they are just data values. </para> @@ -1529,6 +1529,7 @@ GET DIAGNOSTICS integer_var = ROW_COUNT; <listitem> <para> <command>UPDATE</command>, <command>INSERT</command>, and <command>DELETE</command> + and <command>MERGE</command> statements set <literal>FOUND</literal> true if at least one row is affected, false if no row is affected. </para> diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 22e6893211..4e01e5641c 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -159,6 +159,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY load SYSTEM "load.sgml"> <!ENTITY lock SYSTEM "lock.sgml"> <!ENTITY move SYSTEM "move.sgml"> +<!ENTITY merge SYSTEM "merge.sgml"> <!ENTITY notify SYSTEM "notify.sgml"> <!ENTITY prepare SYSTEM "prepare.sgml"> <!ENTITY prepareTransaction SYSTEM "prepare_transaction.sgml"> diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml index 62e142fd8e..da294aaa46 100644 --- a/doc/src/sgml/ref/insert.sgml +++ b/doc/src/sgml/ref/insert.sgml @@ -579,6 +579,13 @@ INSERT <replaceable>oid</replaceable> <replaceable class="parameter">count</repl is a partition, an error will occur if one of the input rows violates the partition constraint. </para> + + <para> + You may also wish to consider using <command>MERGE</command>, since that + allows mixed <command>INSERT</command>, <command>UPDATE</command> and + <command>DELETE</command> within a single statement. + See <xref linkend="sql-merge"/>. + </para> </refsect1> <refsect1> @@ -749,7 +756,9 @@ INSERT INTO distributors (did, dname) VALUES (10, 'Conrad International') Also, the case in which a column name list is omitted, but not all the columns are filled from the <literal>VALUES</literal> clause or <replaceable>query</replaceable>, - is disallowed by the standard. + is disallowed by the standard. If you prefer a more SQL Standard + conforming statement than <literal>ON CONFLICT</literal>, see + <xref linkend="sql-merge"/>. </para> <para> diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index d27fb414f7..ef2270c467 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -186,6 +186,7 @@ &listen; &load; &lock; + &merge; &move; ¬ify; &prepare; diff --git a/doc/src/sgml/trigger.sgml b/doc/src/sgml/trigger.sgml index c43dbc9786..ac662bc64d 100644 --- a/doc/src/sgml/trigger.sgml +++ b/doc/src/sgml/trigger.sgml @@ -182,6 +182,26 @@ will be fired. </para> + <para> + No separate triggers are defined for <command>MERGE</command>. Instead, + statement-level or row-level <command>UPDATE</command>, + <command>DELETE</command> and <command>INSERT</command> triggers are fired + depedning on what actions are specified in the <command>MERGE</command> query + and what actions are activated. + </para> + + <para> + While running a <command>MERGE</command> command, statement-level + <literal>BEFORE</literal> and <literal>AFTER</literal> triggers are fired for + specific actions specified in the <command>MERGE</command>, irrespective of + whether the action is finally activated or not. This is same as + an <command>UPDATE</command> statement that updates no rows, yet + statement-level triggers are fired. The row-level triggers are fired only + when a row is actually updated, inserted or deleted. So it's perfectly legal + that while statement-level triggers are fired for certain type of action, no + row-level triggers are fired for the same kind of action. + </para> + <para> Trigger functions invoked by per-statement triggers should always return <symbol>NULL</symbol>. Trigger functions invoked by per-row diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index c08ab14c02..dec97a7328 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -3241,6 +3241,7 @@ l1: result == HeapTupleUpdated || result == HeapTupleBeingUpdated); Assert(!(tp.t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = tp.t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(tp.t_data); if (result == HeapTupleSelfUpdated) @@ -3882,6 +3883,7 @@ l2: result == HeapTupleUpdated || result == HeapTupleBeingUpdated); Assert(!(oldtup.t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = oldtup.t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(oldtup.t_data); if (result == HeapTupleSelfUpdated) @@ -5086,6 +5088,7 @@ failed: Assert(result == HeapTupleSelfUpdated || result == HeapTupleUpdated || result == HeapTupleWouldBlock); Assert(!(tuple->t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = tuple->t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(tuple->t_data); if (result == HeapTupleSelfUpdated) diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt index 20d61f3780..9612c135da 100644 --- a/src/backend/catalog/sql_features.txt +++ b/src/backend/catalog/sql_features.txt @@ -229,9 +229,9 @@ F311 Schema definition statement 02 CREATE TABLE for persistent base tables YES F311 Schema definition statement 03 CREATE VIEW YES F311 Schema definition statement 04 CREATE VIEW: WITH CHECK OPTION YES F311 Schema definition statement 05 GRANT statement YES -F312 MERGE statement NO consider INSERT ... ON CONFLICT DO UPDATE -F313 Enhanced MERGE statement NO -F314 MERGE statement with DELETE branch NO +F312 MERGE statement YES consider INSERT ... ON CONFLICT DO UPDATE +F313 Enhanced MERGE statement YES +F314 MERGE statement with DELETE branch YES F321 User authorization YES F341 Usage tables NO no ROUTINE_*_USAGE tables F361 Subprogram support YES diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index c38d178cd9..dc2f727d21 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -887,6 +887,9 @@ ExplainNode(PlanState *planstate, List *ancestors, case CMD_DELETE: pname = operation = "Delete"; break; + case CMD_MERGE: + pname = operation = "Merge"; + break; default: pname = "???"; break; @@ -2948,6 +2951,10 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, operation = "Delete"; foperation = "Foreign Delete"; break; + case CMD_MERGE: + operation = "Merge"; + foperation = "Foreign Merge"; + break; default: operation = "???"; foperation = "Foreign ???"; @@ -3070,6 +3077,29 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, other_path, 0, es); } } + else if (node->operation == CMD_MERGE) + { + /* EXPLAIN ANALYZE display of actual outcome for each tuple proposed */ + if (es->analyze && mtstate->ps.instrument) + { + double total; + double insert_path; + double update_path; + double delete_path; + + InstrEndLoop(mtstate->mt_plans[0]->instrument); + + /* count the number of source rows */ + total = mtstate->mt_plans[0]->instrument->ntuples; + update_path = mtstate->ps.instrument->nfiltered1; + delete_path = mtstate->ps.instrument->nfiltered2; + insert_path = total - update_path - delete_path; + + ExplainPropertyFloat("Tuples Inserted", NULL, insert_path, 0, es); + ExplainPropertyFloat("Tuples Updated", NULL, update_path, 0, es); + ExplainPropertyFloat("Tuples Deleted", NULL, delete_path, 0, es); + } + } if (labeltargets) ExplainCloseGroup("Target Tables", "Target Tables", false, es); diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c index b945b1556a..c3610b1874 100644 --- a/src/backend/commands/prepare.c +++ b/src/backend/commands/prepare.c @@ -151,6 +151,7 @@ PrepareQuery(PrepareStmt *stmt, const char *queryString, case CMD_INSERT: case CMD_UPDATE: case CMD_DELETE: + case CMD_MERGE: /* OK */ break; default: diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c index 9d8df5986e..331c37ad67 100644 --- a/src/backend/commands/trigger.c +++ b/src/backend/commands/trigger.c @@ -85,7 +85,8 @@ static HeapTuple GetTupleForTrigger(EState *estate, ResultRelInfo *relinfo, ItemPointer tid, LockTupleMode lockmode, - TupleTableSlot **newSlot); + TupleTableSlot **newSlot, + HeapUpdateFailureData *hufdp); static bool TriggerEnabled(EState *estate, ResultRelInfo *relinfo, Trigger *trigger, TriggerEvent event, Bitmapset *modifiedCols, @@ -2729,7 +2730,8 @@ bool ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, - HeapTuple fdw_trigtuple) + HeapTuple fdw_trigtuple, + HeapUpdateFailureData *hufdp) { TriggerDesc *trigdesc = relinfo->ri_TrigDesc; bool result = true; @@ -2743,7 +2745,7 @@ ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, if (fdw_trigtuple == NULL) { trigtuple = GetTupleForTrigger(estate, epqstate, relinfo, tupleid, - LockTupleExclusive, &newSlot); + LockTupleExclusive, &newSlot, hufdp); if (trigtuple == NULL) return false; } @@ -2814,6 +2816,7 @@ ExecARDeleteTriggers(EState *estate, ResultRelInfo *relinfo, relinfo, tupleid, LockTupleExclusive, + NULL, NULL); else trigtuple = fdw_trigtuple; @@ -2951,7 +2954,8 @@ ExecBRUpdateTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, HeapTuple fdw_trigtuple, - TupleTableSlot *slot) + TupleTableSlot *slot, + HeapUpdateFailureData *hufdp) { TriggerDesc *trigdesc = relinfo->ri_TrigDesc; HeapTuple slottuple = ExecMaterializeSlot(slot); @@ -2972,7 +2976,7 @@ ExecBRUpdateTriggers(EState *estate, EPQState *epqstate, { /* get a copy of the on-disk tuple we are planning to update */ trigtuple = GetTupleForTrigger(estate, epqstate, relinfo, tupleid, - lockmode, &newSlot); + lockmode, &newSlot, hufdp); if (trigtuple == NULL) return NULL; /* cancel the update action */ } @@ -3092,6 +3096,7 @@ ExecARUpdateTriggers(EState *estate, ResultRelInfo *relinfo, relinfo, tupleid, LockTupleExclusive, + NULL, NULL); else trigtuple = fdw_trigtuple; @@ -3240,7 +3245,8 @@ GetTupleForTrigger(EState *estate, ResultRelInfo *relinfo, ItemPointer tid, LockTupleMode lockmode, - TupleTableSlot **newSlot) + TupleTableSlot **newSlot, + HeapUpdateFailureData *hufdp) { Relation relation = relinfo->ri_RelationDesc; HeapTupleData tuple; @@ -3266,6 +3272,11 @@ ltrmark:; estate->es_output_cid, lockmode, LockWaitBlock, false, &buffer, &hufd); + + /* Let the caller know about failure reason, if any. */ + if (hufdp) + *hufdp = hufd; + switch (test) { case HeapTupleSelfUpdated: @@ -3302,10 +3313,17 @@ ltrmark:; /* it was updated, so look at the updated version */ TupleTableSlot *epqslot; + /* + * If we're running MERGE then we must install the + * new tuple in the slot of the underlying join query and + * not the result relation itself. If the join does not + * yeild any tuple, the caller will take the necessary + * action. + */ epqslot = EvalPlanQual(estate, epqstate, relation, - relinfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(relinfo), lockmode, &hufd.ctid, hufd.xmax); @@ -3828,8 +3846,14 @@ struct AfterTriggersTableData bool before_trig_done; /* did we already queue BS triggers? */ bool after_trig_done; /* did we already queue AS triggers? */ AfterTriggerEventList after_trig_events; /* if so, saved list pointer */ - Tuplestorestate *old_tuplestore; /* "old" transition table, if any */ - Tuplestorestate *new_tuplestore; /* "new" transition table, if any */ + /* "old" transition table for UPDATE, if any */ + Tuplestorestate *old_upd_tuplestore; + /* "new" transition table for UPDATE, if any */ + Tuplestorestate *new_upd_tuplestore; + /* "old" transition table for DELETE, if any */ + Tuplestorestate *old_del_tuplestore; + /* "new" transition table INSERT, if any */ + Tuplestorestate *new_ins_tuplestore; }; static AfterTriggersData afterTriggers; @@ -4296,13 +4320,19 @@ AfterTriggerExecute(AfterTriggerEvent event, { if (LocTriggerData.tg_trigger->tgoldtable) { - LocTriggerData.tg_oldtable = evtshared->ats_table->old_tuplestore; + if (TRIGGER_FIRED_BY_UPDATE(evtshared->ats_event)) + LocTriggerData.tg_oldtable = evtshared->ats_table->old_upd_tuplestore; + else + LocTriggerData.tg_oldtable = evtshared->ats_table->old_del_tuplestore; evtshared->ats_table->closed = true; } if (LocTriggerData.tg_trigger->tgnewtable) { - LocTriggerData.tg_newtable = evtshared->ats_table->new_tuplestore; + if (TRIGGER_FIRED_BY_INSERT(evtshared->ats_event)) + LocTriggerData.tg_newtable = evtshared->ats_table->new_ins_tuplestore; + else + LocTriggerData.tg_newtable = evtshared->ats_table->new_upd_tuplestore; evtshared->ats_table->closed = true; } } @@ -4637,8 +4667,10 @@ TransitionCaptureState * MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) { TransitionCaptureState *state; - bool need_old, - need_new; + bool need_old_upd, + need_new_upd, + need_old_del, + need_new_ins; AfterTriggersTableData *table; MemoryContext oldcxt; ResourceOwner saveResourceOwner; @@ -4650,23 +4682,31 @@ MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) switch (cmdType) { case CMD_INSERT: - need_old = false; - need_new = trigdesc->trig_insert_new_table; + need_old_upd = need_old_del = need_new_upd = false; + need_new_ins = trigdesc->trig_insert_new_table; break; case CMD_UPDATE: - need_old = trigdesc->trig_update_old_table; - need_new = trigdesc->trig_update_new_table; + need_old_upd = trigdesc->trig_update_old_table; + need_new_upd = trigdesc->trig_update_new_table; + need_old_del = need_new_ins = false; break; case CMD_DELETE: - need_old = trigdesc->trig_delete_old_table; - need_new = false; + need_old_del = trigdesc->trig_delete_old_table; + need_old_upd = need_new_upd = need_new_ins = false; + break; + case CMD_MERGE: + need_old_upd = trigdesc->trig_update_old_table; + need_new_upd = trigdesc->trig_update_new_table; + need_old_del = trigdesc->trig_delete_old_table; + need_new_ins = trigdesc->trig_insert_new_table; break; default: elog(ERROR, "unexpected CmdType: %d", (int) cmdType); - need_old = need_new = false; /* keep compiler quiet */ + /* keep compiler quiet */ + need_old_upd = need_new_upd = need_old_del = need_new_ins = false; break; } - if (!need_old && !need_new) + if (!need_old_upd && !need_new_upd && !need_new_ins && !need_old_del) return NULL; /* Check state, like AfterTriggerSaveEvent. */ @@ -4696,10 +4736,14 @@ MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) saveResourceOwner = CurrentResourceOwner; CurrentResourceOwner = CurTransactionResourceOwner; - if (need_old && table->old_tuplestore == NULL) - table->old_tuplestore = tuplestore_begin_heap(false, false, work_mem); - if (need_new && table->new_tuplestore == NULL) - table->new_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_old_upd && table->old_upd_tuplestore == NULL) + table->old_upd_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_new_upd && table->new_upd_tuplestore == NULL) + table->new_upd_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_old_del && table->old_del_tuplestore == NULL) + table->old_del_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_new_ins && table->new_ins_tuplestore == NULL) + table->new_ins_tuplestore = tuplestore_begin_heap(false, false, work_mem); CurrentResourceOwner = saveResourceOwner; MemoryContextSwitchTo(oldcxt); @@ -4888,12 +4932,20 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs) { AfterTriggersTableData *table = (AfterTriggersTableData *) lfirst(lc); - ts = table->old_tuplestore; - table->old_tuplestore = NULL; + ts = table->old_upd_tuplestore; + table->old_upd_tuplestore = NULL; if (ts) tuplestore_end(ts); - ts = table->new_tuplestore; - table->new_tuplestore = NULL; + ts = table->new_upd_tuplestore; + table->new_upd_tuplestore = NULL; + if (ts) + tuplestore_end(ts); + ts = table->old_del_tuplestore; + table->old_del_tuplestore = NULL; + if (ts) + tuplestore_end(ts); + ts = table->new_ins_tuplestore; + table->new_ins_tuplestore = NULL; if (ts) tuplestore_end(ts); } @@ -5744,12 +5796,11 @@ AfterTriggerSaveEvent(EState *estate, ResultRelInfo *relinfo, newtup == NULL)); if (oldtup != NULL && - ((event == TRIGGER_EVENT_DELETE && delete_old_table) || - (event == TRIGGER_EVENT_UPDATE && update_old_table))) + (event == TRIGGER_EVENT_DELETE && delete_old_table)) { Tuplestorestate *old_tuplestore; - old_tuplestore = transition_capture->tcs_private->old_tuplestore; + old_tuplestore = transition_capture->tcs_private->old_del_tuplestore; if (map != NULL) { @@ -5761,13 +5812,48 @@ AfterTriggerSaveEvent(EState *estate, ResultRelInfo *relinfo, else tuplestore_puttuple(old_tuplestore, oldtup); } + if (oldtup != NULL && + (event == TRIGGER_EVENT_UPDATE && update_old_table)) + { + Tuplestorestate *old_tuplestore; + + old_tuplestore = transition_capture->tcs_private->old_upd_tuplestore; + + if (map != NULL) + { + HeapTuple converted = do_convert_tuple(oldtup, map); + + tuplestore_puttuple(old_tuplestore, converted); + pfree(converted); + } + else + tuplestore_puttuple(old_tuplestore, oldtup); + } + if (newtup != NULL && + (event == TRIGGER_EVENT_INSERT && insert_new_table)) + { + Tuplestorestate *new_tuplestore; + + new_tuplestore = transition_capture->tcs_private->new_ins_tuplestore; + + if (original_insert_tuple != NULL) + tuplestore_puttuple(new_tuplestore, original_insert_tuple); + else if (map != NULL) + { + HeapTuple converted = do_convert_tuple(newtup, map); + + tuplestore_puttuple(new_tuplestore, converted); + pfree(converted); + } + else + tuplestore_puttuple(new_tuplestore, newtup); + } if (newtup != NULL && - ((event == TRIGGER_EVENT_INSERT && insert_new_table) || - (event == TRIGGER_EVENT_UPDATE && update_new_table))) + (event == TRIGGER_EVENT_UPDATE && update_new_table)) { Tuplestorestate *new_tuplestore; - new_tuplestore = transition_capture->tcs_private->new_tuplestore; + new_tuplestore = transition_capture->tcs_private->new_upd_tuplestore; if (original_insert_tuple != NULL) tuplestore_puttuple(new_tuplestore, original_insert_tuple); diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefile index cc09895fa5..68675f9796 100644 --- a/src/backend/executor/Makefile +++ b/src/backend/executor/Makefile @@ -22,7 +22,7 @@ OBJS = execAmi.o execCurrent.o execExpr.o execExprInterp.o \ nodeCustom.o nodeFunctionscan.o nodeGather.o \ nodeHash.o nodeHashjoin.o nodeIndexscan.o nodeIndexonlyscan.o \ nodeLimit.o nodeLockRows.o nodeGatherMerge.o \ - nodeMaterial.o nodeMergeAppend.o nodeMergejoin.o nodeModifyTable.o \ + nodeMaterial.o nodeMergeAppend.o nodeMergejoin.o nodeMerge.o nodeModifyTable.o \ nodeNestloop.o nodeProjectSet.o nodeRecursiveunion.o nodeResult.o \ nodeSamplescan.o nodeSeqscan.o nodeSetOp.o nodeSort.o nodeUnique.o \ nodeValuesscan.o \ diff --git a/src/backend/executor/README b/src/backend/executor/README index 0d7cd552eb..05769772b7 100644 --- a/src/backend/executor/README +++ b/src/backend/executor/README @@ -37,6 +37,16 @@ the plan tree returns the computed tuples to be updated, plus a "junk" one. For DELETE, the plan tree need only deliver a CTID column, and the ModifyTable node visits each of those rows and marks the row deleted. +MERGE runs one generic plan that returns candidate target rows. Each row +consists of a super-row that contains all the columns needed by any of the +individual actions, plus a CTID and a TABLEOID junk columns. The CTID column is +required to know if a matching target row was found or not and the TABLEOID +column is needed to find the underlying target partition, in case when the +target table is a partition table. If the CTID column is set we attempt to +activate WHEN MATCHED actions, or if it is NULL then we will attempt to +activate WHEN NOT MATCHED actions. Once we know which action is activated we +form the final result row and apply only those changes. + XXX a great deal more documentation needs to be written here... diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 68f6450ee6..11e1d11ef3 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -233,6 +233,7 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags) case CMD_INSERT: case CMD_DELETE: case CMD_UPDATE: + case CMD_MERGE: estate->es_output_cid = GetCurrentCommandId(true); break; @@ -1357,6 +1358,9 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_onConflictArbiterIndexes = NIL; resultRelInfo->ri_onConflict = NULL; + resultRelInfo->ri_mergeTargetRTI = 0; + resultRelInfo->ri_mergeState = (MergeState *) palloc0(sizeof (MergeState)); + /* * Partition constraint, which also includes the partition constraint of * all the ancestors that are partitions. Note that it will be checked @@ -2205,6 +2209,19 @@ ExecWithCheckOptions(WCOKind kind, ResultRelInfo *resultRelInfo, errmsg("new row violates row-level security policy for table \"%s\"", wco->relname))); break; + case WCO_RLS_MERGE_UPDATE_CHECK: + case WCO_RLS_MERGE_DELETE_CHECK: + if (wco->polname != NULL) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("target row violates row-level security policy \"%s\" (USING expression) for table \"%s\"", + wco->polname, wco->relname))); + else + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("target row violates row-level security policy (USING expression) for table \"%s\"", + wco->relname))); + break; case WCO_RLS_CONFLICT_CHECK: if (wco->polname != NULL) ereport(ERROR, diff --git a/src/backend/executor/execPartition.c b/src/backend/executor/execPartition.c index 9a13188649..a6a7885abd 100644 --- a/src/backend/executor/execPartition.c +++ b/src/backend/executor/execPartition.c @@ -67,6 +67,8 @@ ExecSetupPartitionTupleRouting(ModifyTableState *mtstate, Relation rel) ResultRelInfo *update_rri = NULL; int num_update_rri = 0, update_rri_index = 0; + bool is_update = false; + bool is_merge = false; PartitionTupleRouting *proute; int nparts; ModifyTable *node = mtstate ? (ModifyTable *) mtstate->ps.plan : NULL; @@ -89,13 +91,22 @@ ExecSetupPartitionTupleRouting(ModifyTableState *mtstate, Relation rel) /* Set up details specific to the type of tuple routing we are doing. */ if (node && node->operation == CMD_UPDATE) + is_update = true; + else if (node && node->operation == CMD_MERGE) + is_merge = true; + + if (is_update) { update_rri = mtstate->resultRelInfo; num_update_rri = list_length(node->plans); proute->subplan_partition_offsets = palloc(num_update_rri * sizeof(int)); proute->num_subplan_partition_offsets = num_update_rri; + } + + if (is_update || is_merge) + { /* * We need an additional tuple slot for storing transient tuples that * are converted to the root table descriptor. @@ -299,6 +310,25 @@ ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, return result; } +/* + * Given OID of the partition leaf, return the index of the leaf in the + * partition hierarchy. + */ +int +ExecFindPartitionByOid(PartitionTupleRouting *proute, Oid partoid) +{ + int i; + + for (i = 0; i < proute->num_partitions; i++) + { + if (proute->partition_oids[i] == partoid) + break; + } + + Assert(i < proute->num_partitions); + return i; +} + /* * ExecInitPartitionInfo * Initialize ResultRelInfo and other information for a partition if not @@ -337,6 +367,8 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, rootrel, estate->es_instrument); + leaf_part_rri->ri_PartitionLeafIndex = partidx; + /* * Verify result relation is a valid target for an INSERT. An UPDATE of a * partition-key becomes a DELETE+INSERT operation, so this check is still @@ -625,6 +657,90 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, Assert(proute->partitions[partidx] == NULL); proute->partitions[partidx] = leaf_part_rri; + /* + * Initialize information about this partition that's needed to handle + * MERGE. + */ + if (node && node->operation == CMD_MERGE) + { + TupleDesc partrelDesc = RelationGetDescr(partrel); + TupleConversionMap *map = proute->parent_child_tupconv_maps[partidx]; + int firstVarno = mtstate->resultRelInfo[0].ri_RangeTableIndex; + Relation firstResultRel = mtstate->resultRelInfo[0].ri_RelationDesc; + + /* + * If the root parent and partition have the same tuple + * descriptor, just reuse the original MERGE state for partition. + */ + if (map == NULL) + { + leaf_part_rri->ri_mergeState = resultRelInfo->ri_mergeState; + } + else + { + /* Convert expressions contain partition's attnos. */ + List *conv_tl, *conv_qual; + ListCell *l; + List *matchedActionStates = NIL; + List *notMatchedActionStates = NIL; + + foreach (l, node->mergeActionList) + { + MergeAction *action = lfirst_node(MergeAction, l); + MergeActionState *action_state = makeNode(MergeActionState); + TupleDesc tupDesc; + ExprContext *econtext; + + action_state->matched = action->matched; + action_state->commandType = action->commandType; + + conv_qual = (List *) action->qual; + conv_qual = map_partition_varattnos(conv_qual, + firstVarno, partrel, + firstResultRel, NULL); + + action_state->whenqual = ExecInitQual(conv_qual, &mtstate->ps); + + conv_tl = (List *) action->targetList; + conv_tl = map_partition_varattnos(conv_tl, + firstVarno, partrel, + firstResultRel, NULL); + + conv_tl = adjust_partition_tlist( conv_tl, map); + + tupDesc = ExecTypeFromTL(conv_tl, partrelDesc->tdhasoid); + action_state->tupDesc = tupDesc; + + /* build action projection state */ + econtext = mtstate->ps.ps_ExprContext; + action_state->proj = + ExecBuildProjectionInfo(conv_tl, econtext, + mtstate->mt_mergeproj, + &mtstate->ps, + partrelDesc); + + if (action_state->matched) + matchedActionStates = + lappend(matchedActionStates, action_state); + else + notMatchedActionStates = + lappend(notMatchedActionStates, action_state); + } + leaf_part_rri->ri_mergeState->matchedActionStates = + matchedActionStates; + leaf_part_rri->ri_mergeState->notMatchedActionStates = + notMatchedActionStates; + } + + /* + * get_partition_dispatch_recurse() and expand_partitioned_rtentry() + * fetch the leaf OIDs in the same order. So we can safely derive the + * index of the merge target relation corresponding to this partition + * by simply adding partidx + 1 to the root's merge target relation. + */ + leaf_part_rri->ri_mergeTargetRTI = node->mergeTargetRelation + + partidx + 1; + } MemoryContextSwitchTo(oldContext); return leaf_part_rri; diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 32891abbdf..971f92a938 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -454,7 +454,7 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate, { slot = ExecBRUpdateTriggers(estate, epqstate, resultRelInfo, &searchslot->tts_tuple->t_self, - NULL, slot); + NULL, slot, NULL); if (slot == NULL) /* "do nothing" */ skip_tuple = true; @@ -515,7 +515,7 @@ ExecSimpleRelationDelete(EState *estate, EPQState *epqstate, { skip_tuple = !ExecBRDeleteTriggers(estate, epqstate, resultRelInfo, &searchslot->tts_tuple->t_self, - NULL); + NULL, NULL); } if (!skip_tuple) diff --git a/src/backend/executor/nodeMerge.c b/src/backend/executor/nodeMerge.c new file mode 100644 index 0000000000..e633af7704 --- /dev/null +++ b/src/backend/executor/nodeMerge.c @@ -0,0 +1,566 @@ +/*------------------------------------------------------------------------- + * + * nodeMerge.c + * routines to handle Merge nodes. + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/executor/nodeMerge.c + * + *------------------------------------------------------------------------- + */ + + +#include "postgres.h" + +#include "access/htup_details.h" +#include "access/xact.h" +#include "commands/trigger.h" +#include "executor/execPartition.h" +#include "executor/executor.h" +#include "executor/nodeModifyTable.h" +#include "executor/nodeMerge.h" +#include "miscadmin.h" +#include "nodes/nodeFuncs.h" +#include "storage/bufmgr.h" +#include "storage/lmgr.h" +#include "utils/builtins.h" +#include "utils/memutils.h" +#include "utils/rel.h" +#include "utils/tqual.h" + + +/* + * Check and execute the first qualifying MATCHED action. The current target + * tuple is identified by tupleid. + * + * We start from the first WHEN MATCHED action and check if the additional WHEN + * quals pass. If the additional quals for the first action do not pass, we + * check the second, then the third and so on. If we reach to the end, no + * action is taken and we return true, indicating that no further action is + * required for this tuple. + * + * If we do find a qualifying action, then we attempt to execute the action. In + * case the tuple is concurrently updated, EvalPlanQual is run with the updated + * tuple to recheck the join quals. Note that the additional quals associated + * with individual actions are evaluated separately by the MERGE code, while + * EvalPlanQual checks for the join quals. If EvalPlanQual tells us that the + * updated tuple still passes the join quals, then we restart from the top and + * again look for a qualifying action. Otherwise, we return false indicating + * that a NOT MATCHED action must now be executed for the current source tuple. + */ +static bool +ExecMergeMatched(ModifyTableState *mtstate, EState *estate, + TupleTableSlot *slot, JunkFilter *junkfilter, + ItemPointer tupleid) +{ + ExprContext *econtext = mtstate->ps.ps_ExprContext; + bool isNull; + Datum datum; + Oid tableoid = InvalidOid; + List *mergeMatchedActionStates = NIL; + HeapUpdateFailureData hufd; + bool tuple_updated, + tuple_deleted; + Buffer buffer; + HeapTupleData tuple; + EPQState *epqstate = &mtstate->mt_epqstate; + ResultRelInfo *saved_resultRelInfo; + ResultRelInfo *resultRelInfo = estate->es_result_relation_info; + ListCell *l; + TupleTableSlot *saved_slot = slot; + + + /* + * We always fetch the tableoid while performing MATCHED MERGE action. + * This is strictly not required if the target table is not a partitioned + * table. But we are not yet optimising for that case. + */ + datum = ExecGetJunkAttribute(slot, junkfilter->jf_otherJunkAttNo, + &isNull); + Assert(!isNull); + tableoid = DatumGetObjectId(datum); + + if (mtstate->mt_partition_tuple_routing) + { + int leaf_part_index; + PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing; + + /* + * If we're dealing with a MATCHED tuple, then tableoid must have been + * set correctly. In case of partitioned table, we must now fetch the + * correct result relation corresponding to the child table emitting + * the matching target row. For normal table, there is just one result + * relation and it must be the one emitting the matching row. + */ + leaf_part_index = ExecFindPartitionByOid(proute, tableoid); + + resultRelInfo = proute->partitions[leaf_part_index]; + if (resultRelInfo == NULL) + { + resultRelInfo = ExecInitPartitionInfo(mtstate, + mtstate->resultRelInfo, + proute, estate, leaf_part_index); + Assert(resultRelInfo != NULL); + } + } + + /* + * Save the current information and work with the correct result relation. + */ + saved_resultRelInfo = resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + + /* + * And get the correct action lists. + */ + mergeMatchedActionStates = + resultRelInfo->ri_mergeState->matchedActionStates; + + /* + * If there are not WHEN MATCHED actions, we are done. + */ + if (mergeMatchedActionStates == NIL) + return true; + + /* + * Make tuple and any needed join variables available to ExecQual and + * ExecProject. The target's existing tuple is installed in the scantuple. + * Again, this target relation's slot is required only in the case of a + * MATCHED tuple and UPDATE/DELETE actions. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_existing, + resultRelInfo->ri_RelationDesc->rd_att); + econtext->ecxt_scantuple = mtstate->mt_existing; + econtext->ecxt_innertuple = slot; + econtext->ecxt_outertuple = NULL; + +lmerge_matched:; + slot = saved_slot; + buffer = InvalidBuffer; + + /* + * UPDATE/DELETE is only invoked for matched rows. And we must have found + * the tupleid of the target row in that case. We fetch using SnapshotAny + * because we might get called again after EvalPlanQual returns us a new + * tuple. This tuple may not be visible to our MVCC snapshot. + */ + Assert(tupleid != NULL); + + tuple.t_self = *tupleid; + if (!heap_fetch(resultRelInfo->ri_RelationDesc, SnapshotAny, &tuple, + &buffer, true, NULL)) + elog(ERROR, "Failed to fetch the target tuple"); + + /* Store target's existing tuple in the state's dedicated slot */ + ExecStoreTuple(&tuple, mtstate->mt_existing, buffer, false); + + foreach(l, mergeMatchedActionStates) + { + MergeActionState *action = (MergeActionState *) lfirst(l); + + /* + * Test condition, if any + * + * In the absence of a condition we perform the action unconditionally + * (no need to check separately since ExecQual() will return true if + * there are no conditions to evaluate). + */ + if (!ExecQual(action->whenqual, econtext)) + continue; + + /* + * Check if the existing target tuple meet the USING checks of + * UPDATE/DELETE RLS policies. If those checks fail, we throw an + * error. + * + * The WITH CHECK quals are applied in ExecUpdate() and hence we need + * not do anything special to handle them. + * + * NOTE: We must do this after WHEN quals are evaluated so that we + * check policies only when they matter. + */ + if (resultRelInfo->ri_WithCheckOptions) + { + ExecWithCheckOptions(action->commandType == CMD_UPDATE ? + WCO_RLS_MERGE_UPDATE_CHECK : WCO_RLS_MERGE_DELETE_CHECK, + resultRelInfo, + mtstate->mt_existing, + mtstate->ps.state); + } + + /* Perform stated action */ + switch (action->commandType) + { + case CMD_UPDATE: + + /* + * We set up the projection earlier, so all we do here is + * Project, no need for any other tasks prior to the + * ExecUpdate. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_mergeproj, action->tupDesc); + ExecProject(action->proj); + + /* + * We don't call ExecFilterJunk() because the projected tuple + * using the UPDATE action's targetlist doesn't have a junk + * attribute. + */ + slot = ExecUpdate(mtstate, tupleid, NULL, + mtstate->mt_mergeproj, + slot, epqstate, estate, + &tuple_updated, &hufd, + action, mtstate->canSetTag); + break; + + case CMD_DELETE: + /* Nothing to Project for a DELETE action */ + slot = ExecDelete(mtstate, tupleid, NULL, + slot, epqstate, estate, + &tuple_deleted, false, &hufd, action, + mtstate->canSetTag); + + break; + + default: + elog(ERROR, "unknown action in MERGE WHEN MATCHED clause"); + + } + + /* + * Check for any concurrent update/delete operation which may have + * prevented our update/delete. We also check for situations where we + * might be trying to update/delete the same tuple twice. + */ + if ((action->commandType == CMD_UPDATE && !tuple_updated) || + (action->commandType == CMD_DELETE && !tuple_deleted)) + + { + switch (hufd.result) + { + case HeapTupleMayBeUpdated: + break; + case HeapTupleInvisible: + + /* + * This state should never be reached since the underlying + * JOIN runs with a MVCC snapshot and should only return + * rows visible to us. + */ + elog(ERROR, "unexpected invisible tuple"); + break; + + case HeapTupleSelfUpdated: + + /* + * SQLStandard disallows this for MERGE. + */ + if (TransactionIdIsCurrentTransactionId(hufd.xmax)) + ereport(ERROR, + (errcode(ERRCODE_CARDINALITY_VIOLATION), + errmsg("MERGE command cannot affect row a second time"), + errhint("Ensure that not more than one source rows match any one target row"))); + /* This shouldn't happen */ + elog(ERROR, "attempted to update or delete invisible tuple"); + break; + + case HeapTupleUpdated: + + /* + * The target tuple was concurrently updated/deleted by + * some other transaction. + * + * If the current tuple is that last tuple in the update + * chain, then we know that the tuple was concurrently + * deleted. Just return and let the caller try NOT MATCHED + * actions. + * + * If the current tuple was concurrently updated, then we + * must run the EvalPlanQual() with the new version of the + * tuple. If EvalPlanQual() does not return a tuple (can + * that happen?), then again we switch to NOT MATCHED + * action. If it does return a tuple and the join qual is + * still satified, then we just need to recheck the + * MATCHED actions, starting from the top, and execute the + * first qualifying action. + */ + if (!ItemPointerEquals(tupleid, &hufd.ctid)) + { + TupleTableSlot *epqslot; + + /* + * Since we generate a JOIN query with a target table + * RTE different than the result relation RTE, we must + * pass in the RTI of the relation used in the join + * query and not the one from result relation. + */ + Assert(resultRelInfo->ri_mergeTargetRTI > 0); + epqslot = EvalPlanQual(estate, + epqstate, + resultRelInfo->ri_RelationDesc, + GetEPQRangeTableIndex(resultRelInfo), + LockTupleExclusive, + &hufd.ctid, + hufd.xmax); + + if (!TupIsNull(epqslot)) + { + (void) ExecGetJunkAttribute(epqslot, + resultRelInfo->ri_junkFilter->jf_junkAttNo, + &isNull); + + /* + * A valid ctid means that we are still dealing + * with MATCHED case. But we must retry from the + * start with the updated tuple to ensure that the + * first qualifying WHEN MATCHED action is + * executed. + * + * We don't use the new slot returned by + * EvalPlanQual because we anyways re-install the + * new target tuple in econtext->ecxt_scantuple + * before re-evaluating WHEN AND conditions and + * re-projecting the update targetlists. The + * source side tuple does not change and hence we + * can safely continue to use the old slot. + */ + if (!isNull) + { + /* + * Must update *tupleid to the TID of the + * newer tuple found in the update chain. + */ + *tupleid = hufd.ctid; + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + goto lmerge_matched; + } + } + } + + /* + * Tell the caller about the updated TID, restore the + * state back and return. + */ + *tupleid = hufd.ctid; + estate->es_result_relation_info = saved_resultRelInfo; + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + return false; + + default: + break; + + } + } + + if (action->commandType == CMD_UPDATE && tuple_updated) + InstrCountFiltered1(&mtstate->ps, 1); + if (action->commandType == CMD_DELETE && tuple_deleted) + InstrCountFiltered2(&mtstate->ps, 1); + + /* + * We've activated one of the WHEN clauses, so we don't search + * further. This is required behaviour, not an optimisation. + */ + estate->es_result_relation_info = saved_resultRelInfo; + break; + } + + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + + /* + * Successfully executed an action or no qualifying action was found. + */ + return true; +} + +/* + * Execute the first qualifying NOT MATCHED action. + */ +static void +ExecMergeNotMatched(ModifyTableState *mtstate, EState *estate, + TupleTableSlot *slot) +{ + PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing; + ExprContext *econtext = mtstate->ps.ps_ExprContext; + List *mergeNotMatchedActionStates = NIL; + ResultRelInfo *resultRelInfo; + ListCell *l; + TupleTableSlot *myslot; + + /* + * We are dealing with NOT MATCHED tuple. Since for MERGE, partition tree + * is not expanded for the result relation, we continue to work with the + * currently active result relation, which should be of the root of the + * partition tree. + */ + resultRelInfo = mtstate->resultRelInfo; + + /* + * For INSERT actions, root relation's merge action is OK since the the + * INSERT's targetlist and the WHEN conditions can only refer to the + * source relation and hence it does not matter which result relation we + * work with. + */ + mergeNotMatchedActionStates = + resultRelInfo->ri_mergeState->notMatchedActionStates; + + /* + * Make source tuple available to ExecQual and ExecProject. We don't need + * the target tuple since the WHEN quals and the targetlist can't refer to + * the target columns. + */ + econtext->ecxt_scantuple = NULL; + econtext->ecxt_innertuple = slot; + econtext->ecxt_outertuple = NULL; + + foreach(l, mergeNotMatchedActionStates) + { + MergeActionState *action = (MergeActionState *) lfirst(l); + + /* + * Test condition, if any + * + * In the absence of a condition we perform the action unconditionally + * (no need to check separately since ExecQual() will return true if + * there are no conditions to evaluate). + */ + if (!ExecQual(action->whenqual, econtext)) + continue; + + /* Perform stated action */ + switch (action->commandType) + { + case CMD_INSERT: + + /* + * We set up the projection earlier, so all we do here is + * Project, no need for any other tasks prior to the + * ExecInsert. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_mergeproj, action->tupDesc); + ExecProject(action->proj); + + /* + * ExecPrepareTupleRouting may modify the passed-in slot. Hence + * pass a local reference so that action->slot is not modified. + */ + myslot = mtstate->mt_mergeproj; + + /* Prepare for tuple routing if needed. */ + if (proute) + myslot = ExecPrepareTupleRouting(mtstate, estate, proute, + resultRelInfo, myslot); + slot = ExecInsert(mtstate, myslot, slot, + estate, action, + mtstate->canSetTag); + /* Revert ExecPrepareTupleRouting's state change. */ + if (proute) + estate->es_result_relation_info = resultRelInfo; + break; + case CMD_NOTHING: + /* Do Nothing */ + break; + default: + elog(ERROR, "unknown action in MERGE WHEN NOT MATCHED clause"); + } + + break; + } +} + +/* + * Perform MERGE. + */ +void +ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, + JunkFilter *junkfilter, ResultRelInfo *resultRelInfo) +{ + ItemPointer tupleid; + ItemPointerData tuple_ctid; + bool matched = false; + char relkind; + Datum datum; + bool isNull; + + relkind = resultRelInfo->ri_RelationDesc->rd_rel->relkind; + Assert(relkind == RELKIND_RELATION || + relkind == RELKIND_MATVIEW || + relkind == RELKIND_PARTITIONED_TABLE); + + /* + * We run a JOIN between the target relation and the source relation to + * find a set of candidate source rows that has matching row in the target + * table and a set of candidate source rows that does not have matching + * row in the target table. If the join returns us a tuple with target + * relation's tid set, that implies that the join found a matching row for + * the given source tuple. This case triggers the WHEN MATCHED clause of + * the MERGE. Whereas a NULL in the target relation's ctid column + * indicates a NOT MATCHED case. + */ + datum = ExecGetJunkAttribute(slot, junkfilter->jf_junkAttNo, &isNull); + + if (!isNull) + { + matched = true; + tupleid = (ItemPointer) DatumGetPointer(datum); + tuple_ctid = *tupleid; /* be sure we don't free ctid!! */ + tupleid = &tuple_ctid; + } + else + { + matched = false; + tupleid = NULL; /* we don't need it for INSERT actions */ + } + + /* + * If we are dealing with a WHEN MATCHED case, we look at the given WHEN + * MATCHED actions in an order and execute the first action which also + * satisfies the additional WHEN MATCHED AND quals. If an action without + * any additional quals is found, that action is executed. + * + * Similarly, if we are dealing with WHEN NOT MATCHED case, we look at the + * given WHEN NOT MATCHED actions in an ordr and execute the first + * qualifying action. + * + * Things get interesting in case of concurrent update/delete of the + * target tuple. Such concurrent update/delete is detected while we are + * executing a WHEN MATCHED action. + * + * A concurrent update for example can: + * + * 1. modify the target tuple so that it no longer satisfies the + * additional quals attached to the current WHEN MATCHED action OR 2. + * modify the target tuple so that the join quals no longer pass and hence + * the source tuple no longer has a match. + * + * In the first case, we are still dealing with a WHEN MATCHED case, but + * we should recheck the list of WHEN MATCHED actions and choose the first + * one that satisfies the new target tuple. In the second case, since the + * source tuple no longer matches the target tuple, we now instead find a + * qualifying WHEN NOT MATCHED action and execute that. + * + * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. + * + * ExecMergeMatched takes care of following the update chain and + * re-finding the qualifying WHEN MATCHED action, as long as the updated + * target tuple still satisfies the join quals i.e. it still remains a + * WHEN MATCHED case. If the tuple gets deleted or the join quals fail, it + * returns and we try ExecMergeNotMatched. Given that ExecMergeMatched + * always make progress by following the update chain and we never switch + * from ExecMergeNotMatched to ExecMergeMatched, there is no risk of a + * livelock. + */ + if (!matched || + !ExecMergeMatched(mtstate, estate, slot, junkfilter, tupleid)) + ExecMergeNotMatched(mtstate, estate, slot); +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 1b09868ff8..def7eec223 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -42,6 +42,7 @@ #include "commands/trigger.h" #include "executor/execPartition.h" #include "executor/executor.h" +#include "executor/nodeMerge.h" #include "executor/nodeModifyTable.h" #include "foreign/fdwapi.h" #include "miscadmin.h" @@ -62,17 +63,17 @@ static bool ExecOnConflictUpdate(ModifyTableState *mtstate, EState *estate, bool canSetTag, TupleTableSlot **returning); -static TupleTableSlot *ExecPrepareTupleRouting(ModifyTableState *mtstate, - EState *estate, - PartitionTupleRouting *proute, - ResultRelInfo *targetRelInfo, - TupleTableSlot *slot); static ResultRelInfo *getTargetResultRelInfo(ModifyTableState *node); static void ExecSetupChildParentMapForTcs(ModifyTableState *mtstate); static void ExecSetupChildParentMapForSubplan(ModifyTableState *mtstate); static TupleConversionMap *tupconv_map_for_subplan(ModifyTableState *node, int whichplan); +/* flags for mt_merge_subcommands */ +#define MERGE_INSERT 0x01 +#define MERGE_UPDATE 0x02 +#define MERGE_DELETE 0x04 + /* * Verify that the tuples to be produced by INSERT or UPDATE match the * target relation's rowtype @@ -259,11 +260,12 @@ ExecCheckTIDVisible(EState *estate, * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +extern TupleTableSlot * ExecInsert(ModifyTableState *mtstate, TupleTableSlot *slot, TupleTableSlot *planSlot, EState *estate, + MergeActionState *actionState, bool canSetTag) { HeapTuple tuple; @@ -390,9 +392,17 @@ ExecInsert(ModifyTableState *mtstate, * partition, we should instead check UPDATE policies, because we are * executing policies defined on the target table, and not those * defined on the child partitions. + * + * If we're running MERGE, we refer to the action that we're executing + * to know if we're doing an INSERT or UPDATE to a partition table. */ - wco_kind = (mtstate->operation == CMD_UPDATE) ? - WCO_RLS_UPDATE_CHECK : WCO_RLS_INSERT_CHECK; + if (mtstate->operation == CMD_UPDATE) + wco_kind = WCO_RLS_UPDATE_CHECK; + else if (mtstate->operation == CMD_INSERT) + wco_kind = WCO_RLS_INSERT_CHECK; + else if (mtstate->operation == CMD_MERGE) + wco_kind = (actionState->commandType == CMD_UPDATE) ? + WCO_RLS_UPDATE_CHECK : WCO_RLS_INSERT_CHECK; /* * ExecWithCheckOptions() will skip any WCOs which are not of the kind @@ -617,10 +627,19 @@ ExecInsert(ModifyTableState *mtstate, * passed to foreign table triggers; it is NULL when the foreign * table has no relevant triggers. * + * MERGE passes actionState of the action it's currently executing; + * regular DELETE passes NULL. This is used by ExecDelete to know if it's + * being called from MERGE or regular DELETE operation. + * + * If the DELETE fails because the tuple is concurrently updated/deleted + * by this or some other transaction, hufdp is filled with the reason as + * well as other important information. Currently only MERGE needs this + * information. + * * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +TupleTableSlot * ExecDelete(ModifyTableState *mtstate, ItemPointer tupleid, HeapTuple oldtuple, @@ -629,6 +648,8 @@ ExecDelete(ModifyTableState *mtstate, EState *estate, bool *tupleDeleted, bool processReturning, + HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag) { ResultRelInfo *resultRelInfo; @@ -641,6 +662,14 @@ ExecDelete(ModifyTableState *mtstate, if (tupleDeleted) *tupleDeleted = false; + /* + * Initialize hufdp. Since the caller is only interested in the failure + * status, initialize with the state that is used to indicate successful + * operation. + */ + if (hufdp) + hufdp->result = HeapTupleMayBeUpdated; + /* * get information on the (current) result relation */ @@ -654,7 +683,7 @@ ExecDelete(ModifyTableState *mtstate, bool dodelete; dodelete = ExecBRDeleteTriggers(estate, epqstate, resultRelInfo, - tupleid, oldtuple); + tupleid, oldtuple, hufdp); if (!dodelete) /* "do nothing" */ return NULL; @@ -721,6 +750,15 @@ ldelete:; estate->es_crosscheck_snapshot, true /* wait for commit */ , &hufd); + + /* + * Copy the necessary information, if the caller has asked for it. We + * must do this irrespective of whether the tuple was updated or + * deleted. + */ + if (hufdp) + *hufdp = hufd; + switch (result) { case HeapTupleSelfUpdated: @@ -755,7 +793,11 @@ ldelete:; errmsg("tuple to be updated was already modified by an operation triggered by the current command"), errhint("Consider using an AFTER trigger instead of a BEFORE trigger to propagate changes to other rows."))); - /* Else, already deleted by self; nothing to do */ + /* + * Else, already deleted by self; nothing to do but inform + * MERGE about it anyways so that it can take necessary + * action. + */ return NULL; case HeapTupleMayBeUpdated: @@ -766,14 +808,24 @@ ldelete:; ereport(ERROR, (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE), errmsg("could not serialize access due to concurrent update"))); + if (!ItemPointerEquals(tupleid, &hufd.ctid)) { TupleTableSlot *epqslot; + /* + * If we're executing MERGE, then the onus of running + * EvalPlanQual() and handling its outcome lies with the + * caller. + */ + if (actionState != NULL) + return NULL; + + /* Normal DELETE path. */ epqslot = EvalPlanQual(estate, epqstate, resultRelationDesc, - resultRelInfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(resultRelInfo), LockTupleExclusive, &hufd.ctid, hufd.xmax); @@ -783,7 +835,12 @@ ldelete:; goto ldelete; } } - /* tuple already deleted; nothing to do */ + + /* + * tuple already deleted; nothing to do. But MERGE might want + * to handle it differently. We've already filled-in hufdp + * with sufficient information for MERGE to look at. + */ return NULL; default: @@ -911,10 +968,21 @@ ldelete:; * foreign table triggers; it is NULL when the foreign table has * no relevant triggers. * + * MERGE passes actionState of the action it's currently executing; + * regular UPDATE passes NULL. This is used by ExecUpdate to know if it's + * being called from MERGE or regular UPDATE operation. ExecUpdate may + * pass this information to ExecInsert if it ends up running DELETE+INSERT + * for partition key updates. + * + * If the UPDATE fails because the tuple is concurrently updated/deleted + * by this or some other transaction, hufdp is filled with the reason as + * well as other important information. Currently only MERGE needs this + * information. + * * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +extern TupleTableSlot * ExecUpdate(ModifyTableState *mtstate, ItemPointer tupleid, HeapTuple oldtuple, @@ -922,6 +990,9 @@ ExecUpdate(ModifyTableState *mtstate, TupleTableSlot *planSlot, EPQState *epqstate, EState *estate, + bool *tuple_updated, + HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag) { HeapTuple tuple; @@ -938,6 +1009,17 @@ ExecUpdate(ModifyTableState *mtstate, if (IsBootstrapProcessingMode()) elog(ERROR, "cannot UPDATE during bootstrap"); + if (tuple_updated) + *tuple_updated = false; + + /* + * Initialize hufdp. Since the caller is only interested in the failure + * status, initialize with the state that is used to indicate successful + * operation. + */ + if (hufdp) + hufdp->result = HeapTupleMayBeUpdated; + /* * get the heap tuple out of the tuple table slot, making sure we have a * writable copy @@ -955,7 +1037,7 @@ ExecUpdate(ModifyTableState *mtstate, resultRelInfo->ri_TrigDesc->trig_update_before_row) { slot = ExecBRUpdateTriggers(estate, epqstate, resultRelInfo, - tupleid, oldtuple, slot); + tupleid, oldtuple, slot, hufdp); if (slot == NULL) /* "do nothing" */ return NULL; @@ -1079,8 +1161,9 @@ lreplace:; * Row movement, part 1. Delete the tuple, but skip RETURNING * processing. We want to return rows from INSERT. */ - ExecDelete(mtstate, tupleid, oldtuple, planSlot, epqstate, estate, - &tuple_deleted, false, false); + ExecDelete(mtstate, tupleid, oldtuple, planSlot, epqstate, + estate, &tuple_deleted, false, hufdp, NULL, + false); /* * For some reason if DELETE didn't happen (e.g. trigger prevented @@ -1116,16 +1199,36 @@ lreplace:; saved_tcs_map = mtstate->mt_transition_capture->tcs_map; /* - * resultRelInfo is one of the per-subplan resultRelInfos. So we - * should convert the tuple into root's tuple descriptor, since - * ExecInsert() starts the search from root. The tuple conversion - * map list is in the order of mtstate->resultRelInfo[], so to - * retrieve the one for this resultRel, we need to know the - * position of the resultRel in mtstate->resultRelInfo[]. + * We should convert the tuple into root's tuple descriptor, since + * ExecInsert() starts the search from root. To do that, we need to + * retrieve the tuple conversion map for this resultRelInfo. + * + * If we're running MERGE then resultRelInfo is per-partition + * resultRelInfo as initialised in ExecInitPartitionInfo(). Note + * that we don't expand inheritance for the resultRelation in case + * of MERGE and hence there is just one subplan. Whereas for + * regular UPDATE, resultRelInfo is one of the per-subplan + * resultRelInfos. In either case the position of this partition in + * tracked in ri_PartitionLeafIndex; + * + * Retrieve the map either by looking at the resultRelInfo's + * position in mtstate->resultRelInfo[] (for UPDATE) or by simply + * using the ri_PartitionLeafIndex value (for MERGE). */ - map_index = resultRelInfo - mtstate->resultRelInfo; - Assert(map_index >= 0 && map_index < mtstate->mt_nplans); - tupconv_map = tupconv_map_for_subplan(mtstate, map_index); + if (mtstate->operation == CMD_MERGE) + { + map_index = resultRelInfo->ri_PartitionLeafIndex; + Assert(mtstate->rootResultRelInfo == NULL); + tupconv_map = TupConvMapForLeaf(proute, + mtstate->resultRelInfo, + map_index); + } + else + { + map_index = resultRelInfo - mtstate->resultRelInfo; + Assert(map_index >= 0 && map_index < mtstate->mt_nplans); + tupconv_map = tupconv_map_for_subplan(mtstate, map_index); + } tuple = ConvertPartitionTupleSlot(tupconv_map, tuple, proute->root_tuple_slot, @@ -1135,12 +1238,16 @@ lreplace:; * Prepare for tuple routing, making it look like we're inserting * into the root. */ - Assert(mtstate->rootResultRelInfo != NULL); slot = ExecPrepareTupleRouting(mtstate, estate, proute, - mtstate->rootResultRelInfo, slot); + getTargetResultRelInfo(mtstate), + slot); ret_slot = ExecInsert(mtstate, slot, planSlot, - estate, canSetTag); + estate, actionState, canSetTag); + + /* Update is successful. */ + if (tuple_updated) + *tuple_updated = true; /* Revert ExecPrepareTupleRouting's node change. */ estate->es_result_relation_info = resultRelInfo; @@ -1179,6 +1286,15 @@ lreplace:; estate->es_crosscheck_snapshot, true /* wait for commit */ , &hufd, &lockmode); + + /* + * Copy the necessary information, if the caller has asked for it. We + * must do this irrespective of whether the tuple was updated or + * deleted. + */ + if (hufdp) + *hufdp = hufd; + switch (result) { case HeapTupleSelfUpdated: @@ -1223,26 +1339,42 @@ lreplace:; ereport(ERROR, (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE), errmsg("could not serialize access due to concurrent update"))); + if (!ItemPointerEquals(tupleid, &hufd.ctid)) { TupleTableSlot *epqslot; + /* + * If we're executing MERGE, then the onus of running + * EvalPlanQual() and handling its outcome lies with the + * caller. + */ + if (actionState != NULL) + return NULL; + + /* Regular UPDATE path. */ epqslot = EvalPlanQual(estate, epqstate, resultRelationDesc, - resultRelInfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(resultRelInfo), lockmode, &hufd.ctid, hufd.xmax); if (!TupIsNull(epqslot)) { *tupleid = hufd.ctid; + /* Normal UPDATE path */ slot = ExecFilterJunk(resultRelInfo->ri_junkFilter, epqslot); tuple = ExecMaterializeSlot(slot); goto lreplace; } } - /* tuple already deleted; nothing to do */ + + /* + * tuple already deleted; nothing to do. But MERGE might want + * to handle it differently. We've already filled-in hufdp + * with sufficient information for MERGE to look at. + */ return NULL; default: @@ -1271,6 +1403,9 @@ lreplace:; estate, false, NULL, NIL); } + if (tuple_updated) + *tuple_updated = true; + if (canSetTag) (estate->es_processed)++; @@ -1365,9 +1500,9 @@ ExecOnConflictUpdate(ModifyTableState *mtstate, * there's no historical behavior to break. * * It is the user's responsibility to prevent this situation from - * occurring. These problems are why SQL-2003 similarly specifies - * that for SQL MERGE, an exception must be raised in the event of - * an attempt to update the same row twice. + * occurring. These problems are why SQL Standard similarly + * specifies that for SQL MERGE, an exception must be raised in + * the event of an attempt to update the same row twice. */ if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(tuple.t_data))) ereport(ERROR, @@ -1489,7 +1624,7 @@ ExecOnConflictUpdate(ModifyTableState *mtstate, *returning = ExecUpdate(mtstate, &tuple.t_self, NULL, mtstate->mt_conflproj, planSlot, &mtstate->mt_epqstate, mtstate->ps.state, - canSetTag); + NULL, NULL, NULL, canSetTag); ReleaseBuffer(buffer); return true; @@ -1527,6 +1662,14 @@ fireBSTriggers(ModifyTableState *node) case CMD_DELETE: ExecBSDeleteTriggers(node->ps.state, resultRelInfo); break; + case CMD_MERGE: + if (node->mt_merge_subcommands & MERGE_INSERT) + ExecBSInsertTriggers(node->ps.state, resultRelInfo); + if (node->mt_merge_subcommands & MERGE_UPDATE) + ExecBSUpdateTriggers(node->ps.state, resultRelInfo); + if (node->mt_merge_subcommands & MERGE_DELETE) + ExecBSDeleteTriggers(node->ps.state, resultRelInfo); + break; default: elog(ERROR, "unknown operation"); break; @@ -1582,6 +1725,17 @@ fireASTriggers(ModifyTableState *node) ExecASDeleteTriggers(node->ps.state, resultRelInfo, node->mt_transition_capture); break; + case CMD_MERGE: + if (node->mt_merge_subcommands & MERGE_DELETE) + ExecASDeleteTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + if (node->mt_merge_subcommands & MERGE_UPDATE) + ExecASUpdateTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + if (node->mt_merge_subcommands & MERGE_INSERT) + ExecASInsertTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + break; default: elog(ERROR, "unknown operation"); break; @@ -1644,7 +1798,7 @@ ExecSetupTransitionCaptureState(ModifyTableState *mtstate, EState *estate) * * Returns a slot holding the tuple of the partition rowtype. */ -static TupleTableSlot * +TupleTableSlot * ExecPrepareTupleRouting(ModifyTableState *mtstate, EState *estate, PartitionTupleRouting *proute, @@ -1967,6 +2121,7 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; + if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2015,6 +2170,12 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + if (operation == CMD_MERGE) + { + ExecMerge(node, estate, slot, junkfilter, resultRelInfo); + continue; + } + tupleid = NULL; oldtuple = NULL; if (junkfilter != NULL) @@ -2096,19 +2257,20 @@ ExecModifyTable(PlanState *pstate) slot = ExecPrepareTupleRouting(node, estate, proute, resultRelInfo, slot); slot = ExecInsert(node, slot, planSlot, - estate, node->canSetTag); + estate, NULL, node->canSetTag); /* Revert ExecPrepareTupleRouting's state change. */ if (proute) estate->es_result_relation_info = resultRelInfo; break; case CMD_UPDATE: slot = ExecUpdate(node, tupleid, oldtuple, slot, planSlot, - &node->mt_epqstate, estate, node->canSetTag); + &node->mt_epqstate, estate, + NULL, NULL, NULL, node->canSetTag); break; case CMD_DELETE: slot = ExecDelete(node, tupleid, oldtuple, planSlot, &node->mt_epqstate, estate, - NULL, true, node->canSetTag); + NULL, true, NULL, NULL, node->canSetTag); break; default: elog(ERROR, "unknown operation"); @@ -2198,6 +2360,16 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) saved_resultRelInfo = estate->es_result_relation_info; resultRelInfo = mtstate->resultRelInfo; + + /* + * mergeTargetRelation must be set if we're running MERGE and mustn't be + * set if we're not. + */ + Assert(operation != CMD_MERGE || node->mergeTargetRelation > 0); + Assert(operation == CMD_MERGE || node->mergeTargetRelation == 0); + + resultRelInfo->ri_mergeTargetRTI = node->mergeTargetRelation; + i = 0; foreach(l, node->plans) { @@ -2276,7 +2448,8 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) * partition key. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE && - (operation == CMD_INSERT || update_tuple_routing_needed)) + (operation == CMD_INSERT || operation == CMD_MERGE || + update_tuple_routing_needed)) mtstate->mt_partition_tuple_routing = ExecSetupPartitionTupleRouting(mtstate, rel); @@ -2287,6 +2460,15 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) if (!(eflags & EXEC_FLAG_EXPLAIN_ONLY)) ExecSetupTransitionCaptureState(mtstate, estate); + /* + * If we are doing MERGE then setup child-parent mapping. This will be + * required in case we end up doing a partition-key update, triggering a + * tuple routing. + */ + if (mtstate->operation == CMD_MERGE && + mtstate->mt_partition_tuple_routing != NULL) + ExecSetupChildParentMapForLeaf(mtstate->mt_partition_tuple_routing); + /* * Construct mapping from each of the per-subplan partition attnos to the * root attno. This is required when during update row movement the tuple @@ -2478,6 +2660,102 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) } } + resultRelInfo = mtstate->resultRelInfo; + + if (node->mergeActionList) + { + ListCell *l; + ExprContext *econtext; + List *mergeMatchedActionStates = NIL; + List *mergeNotMatchedActionStates = NIL; + TupleDesc relationDesc = resultRelInfo->ri_RelationDesc->rd_att; + + mtstate->mt_merge_subcommands = 0; + + if (mtstate->ps.ps_ExprContext == NULL) + ExecAssignExprContext(estate, &mtstate->ps); + + econtext = mtstate->ps.ps_ExprContext; + + /* initialize slot for the existing tuple */ + Assert(mtstate->mt_existing == NULL); + mtstate->mt_existing = + ExecInitExtraTupleSlot(mtstate->ps.state, + mtstate->mt_partition_tuple_routing ? + NULL : relationDesc); + + /* initialise slot for merge actions */ + Assert(mtstate->mt_mergeproj == NULL); + mtstate->mt_mergeproj = + ExecInitExtraTupleSlot(mtstate->ps.state, + mtstate->mt_partition_tuple_routing ? + NULL : relationDesc); + + /* + * Create a MergeActionState for each action on the mergeActionList + * and add it to either a list of matched actions or not-matched + * actions. + */ + foreach(l, node->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + MergeActionState *action_state = makeNode(MergeActionState); + TupleDesc tupDesc; + + action_state->matched = action->matched; + action_state->commandType = action->commandType; + action_state->whenqual = ExecInitQual((List *) action->qual, + &mtstate->ps); + + /* create target slot for this action's projection */ + tupDesc = ExecTypeFromTL((List *) action->targetList, + resultRelInfo->ri_RelationDesc->rd_rel->relhasoids); + action_state->tupDesc = tupDesc; + + /* build action projection state */ + action_state->proj = + ExecBuildProjectionInfo(action->targetList, econtext, + mtstate->mt_mergeproj, &mtstate->ps, + resultRelInfo->ri_RelationDesc->rd_att); + + /* + * We create two lists - one for WHEN MATCHED actions and one + * for WHEN NOT MATCHED actions - and stick the + * MergeActionState into the appropriate list. + */ + if (action_state->matched) + mergeMatchedActionStates = + lappend(mergeMatchedActionStates, action_state); + else + mergeNotMatchedActionStates = + lappend(mergeNotMatchedActionStates, action_state); + + switch (action->commandType) + { + case CMD_INSERT: + mtstate->mt_merge_subcommands |= MERGE_INSERT; + break; + case CMD_UPDATE: + mtstate->mt_merge_subcommands |= MERGE_UPDATE; + break; + case CMD_DELETE: + mtstate->mt_merge_subcommands |= MERGE_DELETE; + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown operation"); + break; + } + + resultRelInfo->ri_mergeState->matchedActionStates = + mergeMatchedActionStates; + resultRelInfo->ri_mergeState->notMatchedActionStates = + mergeNotMatchedActionStates; + + } + } + /* select first subplan */ mtstate->mt_whichplan = 0; subplan = (Plan *) linitial(node->plans); @@ -2491,7 +2769,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) * --- no need to look first. Typically, this will be a 'ctid' or * 'wholerow' attribute, but in the case of a foreign data wrapper it * might be a set of junk attributes sufficient to identify the remote - * row. + * row. We follow this logic for MERGE, so it always has a junk 'ctid'. * * If there are multiple result relations, each one needs its own junk * filter. Note multiple rels are only possible for UPDATE/DELETE, so we @@ -2519,6 +2797,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) break; case CMD_UPDATE: case CMD_DELETE: + case CMD_MERGE: junk_filter_needed = true; break; default: @@ -2534,6 +2813,11 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) JunkFilter *j; subplan = mtstate->mt_plans[i]->plan; + + /* + * XXX we probably need to check plan output for CMD_MERGE + * also + */ if (operation == CMD_INSERT || operation == CMD_UPDATE) ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, subplan->targetlist); @@ -2542,7 +2826,9 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) resultRelInfo->ri_RelationDesc->rd_att->tdhasoid, ExecInitExtraTupleSlot(estate, NULL)); - if (operation == CMD_UPDATE || operation == CMD_DELETE) + if (operation == CMD_UPDATE || + operation == CMD_DELETE || + operation == CMD_MERGE) { /* For UPDATE/DELETE, find the appropriate junk attr now */ char relkind; @@ -2555,6 +2841,14 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) j->jf_junkAttNo = ExecFindJunkAttribute(j, "ctid"); if (!AttributeNumberIsValid(j->jf_junkAttNo)) elog(ERROR, "could not find junk ctid column"); + + if (operation == CMD_MERGE) + { + j->jf_otherJunkAttNo = ExecFindJunkAttribute(j, "tableoid"); + if (!AttributeNumberIsValid(j->jf_otherJunkAttNo)) + elog(ERROR, "could not find junk tableoid column"); + + } } else if (relkind == RELKIND_FOREIGN_TABLE) { diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c index 9fc4431b80..e050a317c0 100644 --- a/src/backend/executor/spi.c +++ b/src/backend/executor/spi.c @@ -2404,6 +2404,9 @@ _SPI_pquery(QueryDesc *queryDesc, bool fire_triggers, uint64 tcount) else res = SPI_OK_UPDATE; break; + case CMD_MERGE: + res = SPI_OK_MERGE; + break; default: return SPI_ERROR_OPUNKNOWN; } diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index c7293a60d7..770ed3b1a8 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -207,6 +207,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(partitioned_rels); COPY_SCALAR_FIELD(partColsUpdated); COPY_NODE_FIELD(resultRelations); + COPY_SCALAR_FIELD(mergeTargetRelation); COPY_SCALAR_FIELD(resultRelIndex); COPY_SCALAR_FIELD(rootResultRelIndex); COPY_NODE_FIELD(plans); @@ -222,6 +223,8 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(onConflictWhere); COPY_SCALAR_FIELD(exclRelRTI); COPY_NODE_FIELD(exclRelTlist); + COPY_NODE_FIELD(mergeSourceTargetList); + COPY_NODE_FIELD(mergeActionList); return newnode; } @@ -2977,6 +2980,9 @@ _copyQuery(const Query *from) COPY_NODE_FIELD(setOperations); COPY_NODE_FIELD(constraintDeps); COPY_NODE_FIELD(withCheckOptions); + COPY_SCALAR_FIELD(mergeTarget_relation); + COPY_NODE_FIELD(mergeSourceTargetList); + COPY_NODE_FIELD(mergeActionList); COPY_LOCATION_FIELD(stmt_location); COPY_LOCATION_FIELD(stmt_len); @@ -3040,6 +3046,34 @@ _copyUpdateStmt(const UpdateStmt *from) return newnode; } +static MergeStmt * +_copyMergeStmt(const MergeStmt *from) +{ + MergeStmt *newnode = makeNode(MergeStmt); + + COPY_NODE_FIELD(relation); + COPY_NODE_FIELD(source_relation); + COPY_NODE_FIELD(join_condition); + COPY_NODE_FIELD(mergeActionList); + + return newnode; +} + +static MergeAction * +_copyMergeAction(const MergeAction *from) +{ + MergeAction *newnode = makeNode(MergeAction); + + COPY_SCALAR_FIELD(matched); + COPY_SCALAR_FIELD(commandType); + COPY_NODE_FIELD(condition); + COPY_NODE_FIELD(qual); + COPY_NODE_FIELD(stmt); + COPY_NODE_FIELD(targetList); + + return newnode; +} + static SelectStmt * _copySelectStmt(const SelectStmt *from) { @@ -5102,6 +5136,12 @@ copyObjectImpl(const void *from) case T_UpdateStmt: retval = _copyUpdateStmt(from); break; + case T_MergeStmt: + retval = _copyMergeStmt(from); + break; + case T_MergeAction: + retval = _copyMergeAction(from); + break; case T_SelectStmt: retval = _copySelectStmt(from); break; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 765b1be74b..5a0151eece 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -987,6 +987,8 @@ _equalQuery(const Query *a, const Query *b) COMPARE_NODE_FIELD(setOperations); COMPARE_NODE_FIELD(constraintDeps); COMPARE_NODE_FIELD(withCheckOptions); + COMPARE_NODE_FIELD(mergeSourceTargetList); + COMPARE_NODE_FIELD(mergeActionList); COMPARE_LOCATION_FIELD(stmt_location); COMPARE_LOCATION_FIELD(stmt_len); @@ -1042,6 +1044,30 @@ _equalUpdateStmt(const UpdateStmt *a, const UpdateStmt *b) return true; } +static bool +_equalMergeStmt(const MergeStmt *a, const MergeStmt *b) +{ + COMPARE_NODE_FIELD(relation); + COMPARE_NODE_FIELD(source_relation); + COMPARE_NODE_FIELD(join_condition); + COMPARE_NODE_FIELD(mergeActionList); + + return true; +} + +static bool +_equalMergeAction(const MergeAction *a, const MergeAction *b) +{ + COMPARE_SCALAR_FIELD(matched); + COMPARE_SCALAR_FIELD(commandType); + COMPARE_NODE_FIELD(condition); + COMPARE_NODE_FIELD(qual); + COMPARE_NODE_FIELD(stmt); + COMPARE_NODE_FIELD(targetList); + + return true; +} + static bool _equalSelectStmt(const SelectStmt *a, const SelectStmt *b) { @@ -3233,6 +3259,12 @@ equal(const void *a, const void *b) case T_UpdateStmt: retval = _equalUpdateStmt(a, b); break; + case T_MergeStmt: + retval = _equalMergeStmt(a, b); + break; + case T_MergeAction: + retval = _equalMergeAction(a, b); + break; case T_SelectStmt: retval = _equalSelectStmt(a, b); break; diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 6c76c41ebe..68e2cec66e 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -2146,6 +2146,16 @@ expression_tree_walker(Node *node, return true; } break; + case T_MergeAction: + { + MergeAction *action = (MergeAction *) node; + + if (walker(action->targetList, context)) + return true; + if (walker(action->qual, context)) + return true; + } + break; case T_JoinExpr: { JoinExpr *join = (JoinExpr *) node; @@ -2255,6 +2265,10 @@ query_tree_walker(Query *query, return true; if (walker((Node *) query->onConflict, context)) return true; + if (walker((Node *) query->mergeSourceTargetList, context)) + return true; + if (walker((Node *) query->mergeActionList, context)) + return true; if (walker((Node *) query->returningList, context)) return true; if (walker((Node *) query->jointree, context)) @@ -2932,6 +2946,18 @@ expression_tree_mutator(Node *node, return (Node *) newnode; } break; + case T_MergeAction: + { + MergeAction *action = (MergeAction *) node; + MergeAction *newnode; + + FLATCOPY(newnode, action, MergeAction); + MUTATE(newnode->qual, action->qual, Node *); + MUTATE(newnode->targetList, action->targetList, List *); + + return (Node *) newnode; + } + break; case T_JoinExpr: { JoinExpr *join = (JoinExpr *) node; @@ -3083,6 +3109,8 @@ query_tree_mutator(Query *query, MUTATE(query->targetList, query->targetList, List *); MUTATE(query->withCheckOptions, query->withCheckOptions, List *); MUTATE(query->onConflict, query->onConflict, OnConflictExpr *); + MUTATE(query->mergeSourceTargetList, query->mergeSourceTargetList, List *); + MUTATE(query->mergeActionList, query->mergeActionList, List *); MUTATE(query->returningList, query->returningList, List *); MUTATE(query->jointree, query->jointree, FromExpr *); MUTATE(query->setOperations, query->setOperations, Node *); @@ -3224,9 +3252,9 @@ query_or_expression_tree_mutator(Node *node, * boundaries: we descend to everything that's possibly interesting. * * Currently, the node type coverage here extends only to DML statements - * (SELECT/INSERT/UPDATE/DELETE) and nodes that can appear in them, because - * this is used mainly during analysis of CTEs, and only DML statements can - * appear in CTEs. + * (SELECT/INSERT/UPDATE/DELETE/MERGE) and nodes that can appear in them, + * because this is used mainly during analysis of CTEs, and only DML + * statements can appear in CTEs. */ bool raw_expression_tree_walker(Node *node, @@ -3406,6 +3434,20 @@ raw_expression_tree_walker(Node *node, return true; } break; + case T_MergeStmt: + { + MergeStmt *stmt = (MergeStmt *) node; + + if (walker(stmt->relation, context)) + return true; + if (walker(stmt->source_relation, context)) + return true; + if (walker(stmt->join_condition, context)) + return true; + if (walker(stmt->mergeActionList, context)) + return true; + } + break; case T_SelectStmt: { SelectStmt *stmt = (SelectStmt *) node; diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f61ae03ac5..9ebea55048 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -375,6 +375,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(partitioned_rels); WRITE_BOOL_FIELD(partColsUpdated); WRITE_NODE_FIELD(resultRelations); + WRITE_INT_FIELD(mergeTargetRelation); WRITE_INT_FIELD(resultRelIndex); WRITE_INT_FIELD(rootResultRelIndex); WRITE_NODE_FIELD(plans); @@ -390,6 +391,21 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(onConflictWhere); WRITE_UINT_FIELD(exclRelRTI); WRITE_NODE_FIELD(exclRelTlist); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); +} + +static void +_outMergeAction(StringInfo str, const MergeAction *node) +{ + WRITE_NODE_TYPE("MERGEACTION"); + + WRITE_BOOL_FIELD(matched); + WRITE_ENUM_FIELD(commandType, CmdType); + WRITE_NODE_FIELD(condition); + WRITE_NODE_FIELD(qual); + /* We don't dump the stmt node */ + WRITE_NODE_FIELD(targetList); } static void @@ -2114,6 +2130,7 @@ _outModifyTablePath(StringInfo str, const ModifyTablePath *node) WRITE_NODE_FIELD(partitioned_rels); WRITE_BOOL_FIELD(partColsUpdated); WRITE_NODE_FIELD(resultRelations); + WRITE_INT_FIELD(mergeTargetRelation); WRITE_NODE_FIELD(subpaths); WRITE_NODE_FIELD(subroots); WRITE_NODE_FIELD(withCheckOptionLists); @@ -2121,6 +2138,8 @@ _outModifyTablePath(StringInfo str, const ModifyTablePath *node) WRITE_NODE_FIELD(rowMarks); WRITE_NODE_FIELD(onconflict); WRITE_INT_FIELD(epqParam); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); } static void @@ -2942,6 +2961,9 @@ _outQuery(StringInfo str, const Query *node) WRITE_NODE_FIELD(setOperations); WRITE_NODE_FIELD(constraintDeps); /* withCheckOptions intentionally omitted, see comment in parsenodes.h */ + WRITE_INT_FIELD(mergeTarget_relation); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); WRITE_LOCATION_FIELD(stmt_location); WRITE_LOCATION_FIELD(stmt_len); } @@ -3657,6 +3679,9 @@ outNode(StringInfo str, const void *obj) case T_ModifyTable: _outModifyTable(str, obj); break; + case T_MergeAction: + _outMergeAction(str, obj); + break; case T_Append: _outAppend(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index fd4586e73d..3b8071d056 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -270,6 +270,9 @@ _readQuery(void) READ_NODE_FIELD(setOperations); READ_NODE_FIELD(constraintDeps); /* withCheckOptions intentionally omitted, see comment in parsenodes.h */ + READ_INT_FIELD(mergeTarget_relation); + READ_NODE_FIELD(mergeSourceTargetList); + READ_NODE_FIELD(mergeActionList); READ_LOCATION_FIELD(stmt_location); READ_LOCATION_FIELD(stmt_len); @@ -1576,6 +1579,7 @@ _readModifyTable(void) READ_NODE_FIELD(partitioned_rels); READ_BOOL_FIELD(partColsUpdated); READ_NODE_FIELD(resultRelations); + READ_INT_FIELD(mergeTargetRelation); READ_INT_FIELD(resultRelIndex); READ_INT_FIELD(rootResultRelIndex); READ_NODE_FIELD(plans); @@ -1591,6 +1595,8 @@ _readModifyTable(void) READ_NODE_FIELD(onConflictWhere); READ_UINT_FIELD(exclRelRTI); READ_NODE_FIELD(exclRelTlist); + READ_NODE_FIELD(mergeSourceTargetList); + READ_NODE_FIELD(mergeActionList); READ_DONE(); } diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 8b4f031d96..b4d376b4ba 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -282,9 +282,13 @@ static ModifyTable *make_modifytable(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subplans, + List *resultRelations, + Index mergeTargetRelation, + List *subplans, List *withCheckOptionLists, List *returningLists, - List *rowMarks, OnConflictExpr *onconflict, int epqParam); + List *rowMarks, OnConflictExpr *onconflict, + List *mergeSourceTargetList, + List *mergeActionList, int epqParam); static GatherMerge *create_gather_merge_plan(PlannerInfo *root, GatherMergePath *best_path); @@ -2394,11 +2398,14 @@ create_modifytable_plan(PlannerInfo *root, ModifyTablePath *best_path) best_path->partitioned_rels, best_path->partColsUpdated, best_path->resultRelations, + best_path->mergeTargetRelation, subplans, best_path->withCheckOptionLists, best_path->returningLists, best_path->rowMarks, best_path->onconflict, + best_path->mergeSourceTargetList, + best_path->mergeActionList, best_path->epqParam); copy_generic_path_info(&plan->plan, &best_path->path); @@ -6465,9 +6472,13 @@ make_modifytable(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subplans, + List *resultRelations, + Index mergeTargetRelation, + List *subplans, List *withCheckOptionLists, List *returningLists, - List *rowMarks, OnConflictExpr *onconflict, int epqParam) + List *rowMarks, OnConflictExpr *onconflict, + List *mergeSourceTargetList, + List *mergeActionList, int epqParam) { ModifyTable *node = makeNode(ModifyTable); List *fdw_private_list; @@ -6493,6 +6504,7 @@ make_modifytable(PlannerInfo *root, node->partitioned_rels = partitioned_rels; node->partColsUpdated = partColsUpdated; node->resultRelations = resultRelations; + node->mergeTargetRelation = mergeTargetRelation; node->resultRelIndex = -1; /* will be set correctly in setrefs.c */ node->rootResultRelIndex = -1; /* will be set correctly in setrefs.c */ node->plans = subplans; @@ -6525,6 +6537,8 @@ make_modifytable(PlannerInfo *root, node->withCheckOptionLists = withCheckOptionLists; node->returningLists = returningLists; node->rowMarks = rowMarks; + node->mergeSourceTargetList = mergeSourceTargetList; + node->mergeActionList = mergeActionList; node->epqParam = epqParam; /* diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 50f858e420..8aea947be3 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -788,6 +788,24 @@ subquery_planner(PlannerGlobal *glob, Query *parse, /* exclRelTlist contains only Vars, so no preprocessing needed */ } + foreach(l, parse->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + action->targetList = (List *) + preprocess_expression(root, + (Node *) action->targetList, + EXPRKIND_TARGET); + action->qual = + preprocess_expression(root, + (Node *) action->qual, + EXPRKIND_QUAL); + } + + parse->mergeSourceTargetList = (List *) + preprocess_expression(root, (Node *) parse->mergeSourceTargetList, + EXPRKIND_TARGET); + root->append_rel_list = (List *) preprocess_expression(root, (Node *) root->append_rel_list, EXPRKIND_APPINFO); @@ -1529,6 +1547,7 @@ inheritance_planner(PlannerInfo *root) subroot->parse->returningList); Assert(!parse->onConflict); + Assert(parse->mergeActionList == NIL); } /* Result path must go into outer query's FINAL upperrel */ @@ -1587,12 +1606,15 @@ inheritance_planner(PlannerInfo *root) partitioned_rels, partColsUpdated, resultRelations, + 0, subpaths, subroots, withCheckOptionLists, returningLists, rowMarks, NULL, + NULL, + NULL, SS_assign_special_param(root))); } @@ -2126,8 +2148,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, } /* - * If this is an INSERT/UPDATE/DELETE, and we're not being called from - * inheritance_planner, add the ModifyTable node. + * If this is an INSERT/UPDATE/DELETE/MERGE, and we're not being + * called from inheritance_planner, add the ModifyTable node. */ if (parse->commandType != CMD_SELECT && !inheritance_update) { @@ -2167,12 +2189,15 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, NIL, false, list_make1_int(parse->resultRelation), + parse->mergeTarget_relation, list_make1(path), list_make1(root), withCheckOptionLists, returningLists, rowMarks, parse->onConflict, + parse->mergeSourceTargetList, + parse->mergeActionList, SS_assign_special_param(root)); } diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index 69dd327f0c..d30889ec7c 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -851,9 +851,68 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) fix_scan_list(root, splan->exclRelTlist, rtoffset); } + /* + * The MERGE produces the target rows by performing a right + * join between the target relation and the source relation + * (which could be a plain relation or a subquery). The INSERT + * and UPDATE actions of the MERGE requires access to the + * columns from the source relation. We arrange things so that + * the source relation attributes are available as INNER_VAR + * and the target relation attributes are available from the + * scan tuple. + */ + if (splan->mergeActionList != NIL) + { + /* + * mergeSourceTargetList is already setup correctly to + * include all Vars coming from the source relation. So we + * fix the targetList of individual action nodes by + * ensuring that the source relation Vars are referenced + * as INNER_VAR. Note that for this to work correctly, + * during execution, the ecxt_innertuple must be set to + * the tuple obtained from the source relation. + * + * We leave the Vars from the result relation (i.e. the + * target relation) unchanged i.e. those Vars would be + * picked from the scan slot. So during execution, we must + * ensure that ecxt_scantuple is setup correctly to refer + * to the tuple from the target relation. + */ + + indexed_tlist *itlist; + + itlist = build_tlist_index(splan->mergeSourceTargetList); + + foreach(l, splan->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + /* Fix targetList of each action. */ + action->targetList = fix_join_expr(root, + action->targetList, + NULL, itlist, + linitial_int(splan->resultRelations), + rtoffset); + + /* Fix quals too. */ + action->qual = (Node *) fix_join_expr(root, + (List *) action->qual, + NULL, itlist, + linitial_int(splan->resultRelations), + rtoffset); + } + } + splan->nominalRelation += rtoffset; splan->exclRelRTI += rtoffset; + /* + * Don't mess-up with mergeTargetRelation if it's set to zero + * i.e. an invalid value. + */ + if (splan->mergeTargetRelation > 0) + splan->mergeTargetRelation += rtoffset; + foreach(l, splan->partitioned_rels) { lfirst_int(l) += rtoffset; diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8603feef2b..4a864b2340 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -105,9 +105,13 @@ preprocess_targetlist(PlannerInfo *root) * scribbles on parse->targetList, which is not very desirable, but we * keep it that way to avoid changing APIs used by FDWs. */ - if (command_type == CMD_UPDATE || command_type == CMD_DELETE) + if (command_type == CMD_UPDATE || + command_type == CMD_DELETE) rewriteTargetListUD(parse, target_rte, target_relation); + if (command_type == CMD_MERGE) + rewriteTargetListMerge(parse, target_relation); + /* * for heap_form_tuple to work, the targetlist must match the exact order * of the attributes. We also need to fill in any missing attributes. -ay @@ -118,6 +122,39 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE command, handle targetlist of each MergeAction separately. We + * give the same treatment to MergeAction->targetList as we would have + * given to a regular INSERT/UPDATE/DELETE. + */ + if (command_type == CMD_MERGE) + { + ListCell *l; + + foreach(l, parse->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + switch (action->commandType) + { + case CMD_INSERT: + case CMD_UPDATE: + action->targetList = expand_targetlist(action->targetList, + action->commandType, + result_relation, + target_relation); + break; + case CMD_DELETE: + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + + } + } + } + /* * Add necessary junk columns for rowmarked rels. These values are needed * for locking of rels selected FOR UPDATE/SHARE, and to do EvalPlanQual @@ -348,6 +385,7 @@ expand_targetlist(List *tlist, int command_type, true /* byval */ ); } break; + case CMD_MERGE: case CMD_UPDATE: if (!att_tup->attisdropped) { diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 22133fcf12..416b3f9578 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -3284,17 +3284,21 @@ create_lockrows_path(PlannerInfo *root, RelOptInfo *rel, * 'rowMarks' is a list of PlanRowMarks (non-locking only) * 'onconflict' is the ON CONFLICT clause, or NULL * 'epqParam' is the ID of Param for EvalPlanQual re-eval + * 'mergeActionList' is a list of MERGE actions */ ModifyTablePath * create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subpaths, + List *resultRelations, + Index mergeTargetRelation, + List *subpaths, List *subroots, List *withCheckOptionLists, List *returningLists, List *rowMarks, OnConflictExpr *onconflict, - int epqParam) + List *mergeSourceTargetList, + List *mergeActionList, int epqParam) { ModifyTablePath *pathnode = makeNode(ModifyTablePath); double total_size; @@ -3359,6 +3363,7 @@ create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, pathnode->partitioned_rels = list_copy(partitioned_rels); pathnode->partColsUpdated = partColsUpdated; pathnode->resultRelations = resultRelations; + pathnode->mergeTargetRelation = mergeTargetRelation; pathnode->subpaths = subpaths; pathnode->subroots = subroots; pathnode->withCheckOptionLists = withCheckOptionLists; @@ -3366,6 +3371,8 @@ create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, pathnode->rowMarks = rowMarks; pathnode->onconflict = onconflict; pathnode->epqParam = epqParam; + pathnode->mergeSourceTargetList = mergeSourceTargetList; + pathnode->mergeActionList = mergeActionList; return pathnode; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index bd3a0c4a0a..8626cb2904 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1835,6 +1835,10 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) trigDesc->trig_delete_before_row)) result = true; break; + /* There is no separate event for MERGE, only INSERT/UPDATE/DELETE */ + case CMD_MERGE: + result = false; + break; default: elog(ERROR, "unrecognized CmdType: %d", (int) event); break; diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index da8f0f93fc..901cf24e20 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -1237,7 +1237,6 @@ find_childrel_parents(PlannerInfo *root, RelOptInfo *rel) return result; } - /* * get_baserel_parampathinfo * Get the ParamPathInfo for a parameterized path for a base relation, diff --git a/src/backend/parser/Makefile b/src/backend/parser/Makefile index f14febdbda..95fdf0b973 100644 --- a/src/backend/parser/Makefile +++ b/src/backend/parser/Makefile @@ -14,7 +14,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS) OBJS= analyze.o gram.o scan.o parser.o \ parse_agg.o parse_clause.o parse_coerce.o parse_collate.o parse_cte.o \ - parse_enr.o parse_expr.o parse_func.o parse_node.o parse_oper.o \ + parse_enr.o parse_expr.o parse_func.o parse_merge.o parse_node.o parse_oper.o \ parse_param.o parse_relation.o parse_target.o parse_type.o \ parse_utilcmd.o scansup.o diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index a4b5aaef44..7eb9544efe 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -38,6 +38,7 @@ #include "parser/parse_cte.h" #include "parser/parse_expr.h" #include "parser/parse_func.h" +#include "parser/parse_merge.h" #include "parser/parse_oper.h" #include "parser/parse_param.h" #include "parser/parse_relation.h" @@ -53,9 +54,6 @@ post_parse_analyze_hook_type post_parse_analyze_hook = NULL; static Query *transformOptionalSelectInto(ParseState *pstate, Node *parseTree); static Query *transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt); static Query *transformInsertStmt(ParseState *pstate, InsertStmt *stmt); -static List *transformInsertRow(ParseState *pstate, List *exprlist, - List *stmtcols, List *icolumns, List *attrnos, - bool strip_indirection); static OnConflictExpr *transformOnConflictClause(ParseState *pstate, OnConflictClause *onConflictClause); static int count_rowexpr_columns(ParseState *pstate, Node *expr); @@ -68,8 +66,6 @@ static void determineRecursiveColTypes(ParseState *pstate, Node *larg, List *nrtargetlist); static Query *transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt); static List *transformReturningList(ParseState *pstate, List *returningList); -static List *transformUpdateTargetList(ParseState *pstate, - List *targetList); static Query *transformDeclareCursorStmt(ParseState *pstate, DeclareCursorStmt *stmt); static Query *transformExplainStmt(ParseState *pstate, @@ -267,6 +263,7 @@ transformStmt(ParseState *pstate, Node *parseTree) case T_InsertStmt: case T_UpdateStmt: case T_DeleteStmt: + case T_MergeStmt: (void) test_raw_expression_coverage(parseTree, NULL); break; default: @@ -291,6 +288,10 @@ transformStmt(ParseState *pstate, Node *parseTree) result = transformUpdateStmt(pstate, (UpdateStmt *) parseTree); break; + case T_MergeStmt: + result = transformMergeStmt(pstate, (MergeStmt *) parseTree); + break; + case T_SelectStmt: { SelectStmt *n = (SelectStmt *) parseTree; @@ -366,6 +367,7 @@ analyze_requires_snapshot(RawStmt *parseTree) case T_InsertStmt: case T_DeleteStmt: case T_UpdateStmt: + case T_MergeStmt: case T_SelectStmt: result = true; break; @@ -896,7 +898,7 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) * attrnos: integer column numbers (must be same length as icolumns) * strip_indirection: if true, remove any field/array assignment nodes */ -static List * +List * transformInsertRow(ParseState *pstate, List *exprlist, List *stmtcols, List *icolumns, List *attrnos, bool strip_indirection) @@ -2260,9 +2262,9 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt) /* * transformUpdateTargetList - - * handle SET clause in UPDATE/INSERT ... ON CONFLICT UPDATE + * handle SET clause in UPDATE/MERGE/INSERT ... ON CONFLICT UPDATE */ -static List * +List * transformUpdateTargetList(ParseState *pstate, List *origTlist) { List *tlist = NIL; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index cd5ba2d4d8..ebca5f3eb7 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -282,6 +282,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CreateMatViewStmt RefreshMatViewStmt CreateAmStmt CreatePublicationStmt AlterPublicationStmt CreateSubscriptionStmt AlterSubscriptionStmt DropSubscriptionStmt + MergeStmt %type <node> select_no_parens select_with_parens select_clause simple_select values_clause @@ -584,6 +585,10 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound partbound_datum_list range_datum_list %type <defelt> hash_partbound_elem +%type <node> merge_when_clause opt_and_condition +%type <list> merge_when_list +%type <node> merge_update merge_delete merge_insert + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -651,7 +656,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED - MAPPING MATCH MATERIALIZED MAXVALUE METHOD MINUTE_P MINVALUE MODE MONTH_P MOVE + MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MERGE METHOD + MINUTE_P MINVALUE MODE MONTH_P MOVE NAME_P NAMES NATIONAL NATURAL NCHAR NEW NEXT NO NONE NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF @@ -920,6 +926,7 @@ stmt : | RefreshMatViewStmt | LoadStmt | LockStmt + | MergeStmt | NotifyStmt | PrepareStmt | ReassignOwnedStmt @@ -10660,6 +10667,7 @@ ExplainableStmt: | InsertStmt | UpdateStmt | DeleteStmt + | MergeStmt | DeclareCursorStmt | CreateAsStmt | CreateMatViewStmt @@ -10722,6 +10730,7 @@ PreparableStmt: | InsertStmt | UpdateStmt | DeleteStmt /* by default all are $$=$1 */ + | MergeStmt ; /***************************************************************************** @@ -11088,6 +11097,151 @@ set_target_list: ; +/***************************************************************************** + * + * QUERY: + * MERGE STATEMENT + * + *****************************************************************************/ + +MergeStmt: + MERGE INTO relation_expr_opt_alias + USING table_ref + ON a_expr + merge_when_list + { + MergeStmt *m = makeNode(MergeStmt); + + m->relation = $3; + m->source_relation = $5; + m->join_condition = $7; + m->mergeActionList = $8; + + $$ = (Node *)m; + } + ; + + +merge_when_list: + merge_when_clause { $$ = list_make1($1); } + | merge_when_list merge_when_clause { $$ = lappend($1,$2); } + ; + +merge_when_clause: + WHEN MATCHED opt_and_condition THEN merge_update + { + MergeAction *m = makeNode(MergeAction); + + m->matched = true; + m->commandType = CMD_UPDATE; + m->condition = $3; + m->stmt = $5; + + $$ = (Node *)m; + } + | WHEN MATCHED opt_and_condition THEN merge_delete + { + MergeAction *m = makeNode(MergeAction); + + m->matched = true; + m->commandType = CMD_DELETE; + m->condition = $3; + m->stmt = $5; + + $$ = (Node *)m; + } + | WHEN NOT MATCHED opt_and_condition THEN merge_insert + { + MergeAction *m = makeNode(MergeAction); + + m->matched = false; + m->commandType = CMD_INSERT; + m->condition = $4; + m->stmt = $6; + + $$ = (Node *)m; + } + | WHEN NOT MATCHED opt_and_condition THEN DO NOTHING + { + MergeAction *m = makeNode(MergeAction); + + m->matched = false; + m->commandType = CMD_NOTHING; + m->condition = $4; + m->stmt = NULL; + + $$ = (Node *)m; + } + ; + +opt_and_condition: + AND a_expr { $$ = $2; } + | { $$ = NULL; } + ; + +merge_delete: + DELETE_P + { + DeleteStmt *n = makeNode(DeleteStmt); + $$ = (Node *)n; + } + ; + +merge_update: + UPDATE SET set_clause_list + { + UpdateStmt *n = makeNode(UpdateStmt); + n->targetList = $3; + + $$ = (Node *)n; + } + ; + +merge_insert: + INSERT values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->selectStmt = $2; + + $$ = (Node *)n; + } + | INSERT OVERRIDING override_kind VALUE_P values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->override = $3; + n->selectStmt = $5; + + $$ = (Node *)n; + } + | INSERT '(' insert_column_list ')' values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = $3; + n->selectStmt = $5; + + $$ = (Node *)n; + } + | INSERT '(' insert_column_list ')' OVERRIDING override_kind VALUE_P values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = $3; + n->override = $6; + n->selectStmt = $8; + + $$ = (Node *)n; + } + | INSERT DEFAULT VALUES + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->selectStmt = NULL; + + $$ = (Node *)n; + } + ; + /***************************************************************************** * * QUERY: @@ -15088,8 +15242,10 @@ unreserved_keyword: | LOGGED | MAPPING | MATCH + | MATCHED | MATERIALIZED | MAXVALUE + | MERGE | METHOD | MINUTE_P | MINVALUE diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 377a7ed6d0..544e7300b8 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -455,6 +455,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) case EXPR_KIND_VALUES_SINGLE: errkind = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + if (isAgg) + err = _("aggregate functions are not allowed in WHEN AND conditions"); + else + err = _("grouping operations are not allowed in WHEN AND conditions"); + + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: if (isAgg) @@ -873,6 +880,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_VALUES_SINGLE: errkind = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + err = _("window functions are not allowed in WHEN AND conditions"); + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: err = _("window functions are not allowed in check constraints"); diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 3a02307bd9..9df9828df8 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -76,9 +76,6 @@ static RangeTblEntry *transformRangeTableFunc(ParseState *pstate, RangeTableFunc *t); static TableSampleClause *transformRangeTableSample(ParseState *pstate, RangeTableSample *rts); -static Node *transformFromClauseItem(ParseState *pstate, Node *n, - RangeTblEntry **top_rte, int *top_rti, - List **namespace); static Node *buildMergedJoinVar(ParseState *pstate, JoinType jointype, Var *l_colvar, Var *r_colvar); static ParseNamespaceItem *makeNamespaceItem(RangeTblEntry *rte, @@ -139,6 +136,7 @@ transformFromClause(ParseState *pstate, List *frmList) n = transformFromClauseItem(pstate, n, &rte, &rtindex, + NULL, NULL, &namespace); checkNameSpaceConflicts(pstate, pstate->p_namespace, namespace); @@ -1100,9 +1098,10 @@ getRTEForSpecialRelationTypes(ParseState *pstate, RangeVar *rv) * as table/column names by this item. (The lateral_only flags in these items * are indeterminate and should be explicitly set by the caller before use.) */ -static Node * +Node * transformFromClauseItem(ParseState *pstate, Node *n, RangeTblEntry **top_rte, int *top_rti, + RangeTblEntry **right_rte, int *right_rti, List **namespace) { if (IsA(n, RangeVar)) @@ -1194,7 +1193,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, /* Recursively transform the contained relation */ rel = transformFromClauseItem(pstate, rts->relation, - top_rte, top_rti, namespace); + top_rte, top_rti, NULL, NULL, namespace); /* Currently, grammar could only return a RangeVar as contained rel */ rtr = castNode(RangeTblRef, rel); rte = rt_fetch(rtr->rtindex, pstate->p_rtable); @@ -1222,6 +1221,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, List *l_namespace, *r_namespace, *my_namespace, + *save_namespace, *l_colnames, *r_colnames, *res_colnames, @@ -1240,6 +1240,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, j->larg = transformFromClauseItem(pstate, j->larg, &l_rte, &l_rtindex, + NULL, NULL, &l_namespace); /* @@ -1263,12 +1264,34 @@ transformFromClauseItem(ParseState *pstate, Node *n, sv_namespace_length = list_length(pstate->p_namespace); pstate->p_namespace = list_concat(pstate->p_namespace, l_namespace); + /* + * If we are running MERGE, don't make the other RTEs visible while + * parsing the source relation. It mustn't see them. + * + * XXX Currently, only MERGE passes non-NULL value for right_rte, so we + * can safely deduce if we're running MERGE or not by just looking at + * the right_rte. If that ever changes, we should look at other means + * to find that. + */ + if (right_rte) + { + save_namespace = pstate->p_namespace; + pstate->p_namespace = NIL; + } + /* And now we can process the RHS */ j->rarg = transformFromClauseItem(pstate, j->rarg, &r_rte, &r_rtindex, + NULL, NULL, &r_namespace); + /* + * And now restore the namespace again so that join-quals can see it. + */ + if (right_rte) + pstate->p_namespace = save_namespace; + /* Remove the left-side RTEs from the namespace list again */ pstate->p_namespace = list_truncate(pstate->p_namespace, sv_namespace_length); @@ -1295,6 +1318,12 @@ transformFromClauseItem(ParseState *pstate, Node *n, expandRTE(r_rte, r_rtindex, 0, -1, false, &r_colnames, &r_colvars); + if (right_rte) + *right_rte = r_rte; + + if (right_rti) + *right_rti = r_rtindex; + /* * Natural join does not explicitly specify columns; must generate * columns to join. Need to run through the list of columns from each diff --git a/src/backend/parser/parse_collate.c b/src/backend/parser/parse_collate.c index 6d34245083..51c73c4018 100644 --- a/src/backend/parser/parse_collate.c +++ b/src/backend/parser/parse_collate.c @@ -485,6 +485,7 @@ assign_collations_walker(Node *node, assign_collations_context *context) case T_FromExpr: case T_OnConflictExpr: case T_SortGroupClause: + case T_MergeAction: (void) expression_tree_walker(node, assign_collations_walker, (void *) &loccontext); diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 385e54a9b6..38fbe3366f 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -1818,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_RETURNING: case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: + case EXPR_KIND_MERGE_WHEN_AND: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3475,6 +3476,8 @@ ParseExprKindName(ParseExprKind exprKind) return "PARTITION BY"; case EXPR_KIND_CALL_ARGUMENT: return "CALL"; + case EXPR_KIND_MERGE_WHEN_AND: + return "MERGE WHEN AND"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index ea5d5212b4..615aee6d15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2277,6 +2277,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) /* okay, since we process this like a SELECT tlist */ pstate->p_hasTargetSRFs = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + err = _("set-returning functions are not allowed in WHEN AND conditions"); + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: err = _("set-returning functions are not allowed in check constraints"); diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c new file mode 100644 index 0000000000..b6e0c46656 --- /dev/null +++ b/src/backend/parser/parse_merge.c @@ -0,0 +1,670 @@ +/*------------------------------------------------------------------------- + * + * parse_merge.c + * handle merge-statement in parser + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/parser/parse_merge.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "miscadmin.h" + +#include "access/sysattr.h" +#include "nodes/makefuncs.h" +#include "parser/analyze.h" +#include "parser/parse_collate.h" +#include "parser/parsetree.h" +#include "parser/parser.h" +#include "parser/parse_clause.h" +#include "parser/parse_merge.h" +#include "parser/parse_relation.h" +#include "parser/parse_target.h" +#include "utils/rel.h" +#include "utils/relcache.h" + +static int transformMergeJoinClause(ParseState *pstate, Node *merge, + List **mergeSourceTargetList); +static void setNamespaceForMergeAction(ParseState *pstate, + MergeAction *action); +static void setNamespaceVisibilityForRTE(List *namespace, RangeTblEntry *rte, + bool rel_visible, + bool cols_visible); +static List *expandSourceTL(ParseState *pstate, RangeTblEntry *rte, + int rtindex); + +/* + * Special handling for MERGE statement is required because we assemble + * the query manually. This is similar to setTargetTable() followed + * by transformFromClause() but with a few less steps. + * + * Process the FROM clause and add items to the query's range table, + * joinlist, and namespace. + * + * A special targetlist comprising of the columns from the right-subtree of + * the join is populated and returned. Note that when the JoinExpr is + * setup by transformMergeStmt, the left subtree has the target result + * relation and the right subtree has the source relation. + * + * Returns the rangetable index of the target relation. + */ +static int +transformMergeJoinClause(ParseState *pstate, Node *merge, + List **mergeSourceTargetList) +{ + RangeTblEntry *rte, + *rt_rte; + List *namespace; + int rtindex, + rt_rtindex; + Node *n; + int mergeTarget_relation = list_length(pstate->p_rtable) + 1; + Var *var; + TargetEntry *te; + + n = transformFromClauseItem(pstate, merge, + &rte, + &rtindex, + &rt_rte, + &rt_rtindex, + &namespace); + + pstate->p_joinlist = list_make1(n); + + /* + * We created an internal join between the target and the source relation + * to carry out the MERGE actions. Normally such an unaliased join hides + * the joining relations, unless the column references are qualified. + * Also, any unqualified column refernces are resolved to the Join RTE, if + * there is a matching entry in the targetlist. But the way MERGE + * execution is later setup, we expect all column references to resolve to + * either the source or the target relation. Hence we must not add the + * Join RTE to the namespace. + * + * The last entry must be for the top-level Join RTE. We don't want to + * resolve any references to the Join RTE. So discard that. + * + * We also do not want to resolve any references from the leftside of the + * Join since that corresponds to the target relation. References to the + * columns of the target relation must be resolved from the result + * relation and not the one that is used in the join. So the + * mergeTarget_relation is marked invisible to both qualified as well as + * unqualified references. + */ + Assert(list_length(namespace) > 1); + namespace = list_truncate(namespace, list_length(namespace) - 1); + pstate->p_namespace = list_concat(pstate->p_namespace, namespace); + + setNamespaceVisibilityForRTE(pstate->p_namespace, + rt_fetch(mergeTarget_relation, pstate->p_rtable), false, false); + + /* + * Expand the right relation and add its columns to the + * mergeSourceTargetList. Note that the right relation can either be a + * plain relation or a subquery or anything that can have a + * RangeTableEntry. + */ + *mergeSourceTargetList = expandSourceTL(pstate, rt_rte, rt_rtindex); + + /* + * Add a whole-row-Var entry to support references to "source.*". + */ + var = makeWholeRowVar(rt_rte, rt_rtindex, 0, false); + te = makeTargetEntry((Expr *) var, list_length(*mergeSourceTargetList) + 1, + NULL, true); + *mergeSourceTargetList = lappend(*mergeSourceTargetList, te); + + return mergeTarget_relation; +} + +/* + * Make appropriate changes to the namespace visibility while transforming + * individual action's quals and targetlist expressions. In particular, for + * INSERT actions we must only see the source relation (since INSERT action is + * invoked for NOT MATCHED tuples and hence there is no target tuple to deal + * with). On the other hand, UPDATE and DELETE actions can see both source and + * target relations. + * + * Also, since the internal Join node can hide the source and target + * relations, we must explicitly make the respective relation as visible so + * that columns can be referenced unqualified from these relations. + */ +static void +setNamespaceForMergeAction(ParseState *pstate, MergeAction *action) +{ + RangeTblEntry *targetRelRTE, + *sourceRelRTE; + + /* Assume target relation is at index 1 */ + targetRelRTE = rt_fetch(1, pstate->p_rtable); + + /* + * Assume that the top-level join RTE is at the end. The source relation + * is just before that. + */ + sourceRelRTE = rt_fetch(list_length(pstate->p_rtable) - 1, pstate->p_rtable); + + switch (action->commandType) + { + case CMD_INSERT: + + /* + * Inserts can't see target relation, but they can see source + * relation. + */ + setNamespaceVisibilityForRTE(pstate->p_namespace, + targetRelRTE, false, false); + setNamespaceVisibilityForRTE(pstate->p_namespace, + sourceRelRTE, true, true); + break; + + case CMD_UPDATE: + case CMD_DELETE: + + /* + * Updates and deletes can see both target and source relations. + */ + setNamespaceVisibilityForRTE(pstate->p_namespace, + targetRelRTE, true, true); + setNamespaceVisibilityForRTE(pstate->p_namespace, + sourceRelRTE, true, true); + break; + + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } +} + +/* + * transformMergeStmt - + * transforms a MERGE statement + */ +Query * +transformMergeStmt(ParseState *pstate, MergeStmt *stmt) +{ + Query *qry = makeNode(Query); + ListCell *l; + AclMode targetPerms = ACL_NO_RIGHTS; + bool is_terminal[2]; + JoinExpr *joinexpr; + RangeTblEntry *resultRelRTE, *mergeRelRTE; + + /* There can't be any outer WITH to worry about */ + Assert(pstate->p_ctenamespace == NIL); + + qry->commandType = CMD_MERGE; + + /* + * Check WHEN clauses for permissions and sanity + */ + is_terminal[0] = false; + is_terminal[1] = false; + foreach(l, stmt->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + uint when_type = (action->matched ? 0 : 1); + + /* + * Collect action types so we can check Target permissions + */ + switch (action->commandType) + { + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + SelectStmt *selectStmt = (SelectStmt *) istmt->selectStmt; + + /* + * The grammar allows attaching ORDER BY, LIMIT, FOR + * UPDATE, or WITH to a VALUES clause and also multiple + * VALUES clauses. If we have any of those, ERROR. + */ + if (selectStmt && (selectStmt->valuesLists == NIL || + selectStmt->sortClause != NIL || + selectStmt->limitOffset != NULL || + selectStmt->limitCount != NULL || + selectStmt->lockingClause != NIL || + selectStmt->withClause != NULL)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("SELECT not allowed in MERGE INSERT statement"))); + + if (selectStmt && list_length(selectStmt->valuesLists) > 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("Multiple VALUES clauses not allowed in MERGE INSERT statement"))); + + targetPerms |= ACL_INSERT; + } + break; + case CMD_UPDATE: + targetPerms |= ACL_UPDATE; + break; + case CMD_DELETE: + targetPerms |= ACL_DELETE; + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } + + /* + * Check for unreachable WHEN clauses + */ + if (action->condition == NULL) + is_terminal[when_type] = true; + else if (is_terminal[when_type]) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unreachable WHEN clause specified after unconditional WHEN clause"))); + } + + /* + * Construct a query of the form SELECT relation.ctid --junk attribute + * ,relation.tableoid --junk attribute ,source_relation.<somecols> + * ,relation.<somecols> FROM relation RIGHT JOIN source_relation ON + * join_condition -- no WHERE clause - all conditions are applied in + * executor + * + * stmt->relation is the target relation, given as a RangeVar + * stmt->source_relation is a RangeVar or subquery + * + * We specify the join as a RIGHT JOIN as a simple way of forcing the + * first (larg) RTE to refer to the target table. + * + * The MERGE query's join can be tuned in some cases, see below for these + * special case tweaks. + * + * We set QSRC_PARSER to show query constructed in parse analysis + * + * Note that we have only one Query for a MERGE statement and the planner + * is called only once. That query is executed once to produce our stream + * of candidate change rows, so the query must contain all of the columns + * required by each of the targetlist or conditions for each action. + * + * As top-level statements INSERT, UPDATE and DELETE have a Query, whereas + * with MERGE the individual actions do not require separate planning, + * only different handling in the executor. See nodeModifyTable handling + * of commandType CMD_MERGE. + * + * A sub-query can include the Target, but otherwise the sub-query cannot + * reference the outermost Target table at all. + */ + qry->querySource = QSRC_PARSER; + + /* + * Setup the target table. Unlike regular UPDATE/DELETE, we don't expand + * inheritance for the target relation in case of MERGE. + * + * This special arrangement is required for handling partitioned tables + * because we perform an JOIN between the target and the source relation to + * identify the matching and not-matching rows. If we take the usual path + * of expanding the target table's inheritance and create one subplan per + * partition, then we we won't be able to correctly identify the matching + * and not-matching rows since for a given source row, there may not be a + * matching row in one partition, but it may exists in some other + * partition. So we must first append all the qualifying rows from all the + * partitions and then do the matching. + * + * Once a target row is returned by the underlying join, we find the + * correct partition and setup required state to carry out UPDATE/DELETE. + * All of this happens during execution. + */ + qry->resultRelation = setTargetTable(pstate, stmt->relation, + false, /* do not expand inheritance */ + true, targetPerms); + + /* + * Create a JOIN between the target and the source relation. + */ + joinexpr = makeNode(JoinExpr); + joinexpr->isNatural = false; + joinexpr->alias = NULL; + joinexpr->usingClause = NIL; + joinexpr->quals = stmt->join_condition; + joinexpr->larg = (Node *) stmt->relation; + joinexpr->rarg = (Node *) stmt->source_relation; + + /* + * Simplify the MERGE query as much as possible + * + * These seem like things that could go into Optimizer, but they are + * semantic simplications rather than optimizations, per se. + * + * If there are no INSERT actions we won't be using the non-matching + * candidate rows for anything, so no need for an outer join. We do still + * need an inner join for UPDATE and DELETE actions. + * + * Possible additional simplifications... + * + * XXX if we have a constant ON clause, we can skip join altogether + * + * XXX if we have a constant subquery, we can also skip join + * + * XXX if we were really keen we could look through the actionList and + * pull out common conditions, if there were no terminal clauses and put + * them into the main query as an early row filter but that seems like an + * atypical case and so checking for it would be likely to just be wasted + * effort. + */ + if (targetPerms & ACL_INSERT) + joinexpr->jointype = JOIN_RIGHT; + else + joinexpr->jointype = JOIN_INNER; + + /* + * We use a special purpose transformation here because the normal + * routines don't quite work right for the MERGE case. + * + * A special mergeSourceTargetList is setup by transformMergeJoinClause(). + * It refers to all the attributes provided by the source relation. This + * is later used by set_plan_refs() to fix the UPDATE/INSERT target lists + * to so that they can correctly fetch the attributes from the source + * relation. + * + * The target relation when used in the underlying join, gets a new RTE + * with rte->inh set to true. We remember this RTE (and later pass on to + * the planner and executor) for two main reasons: + * + * 1. If we ever need to run EvalPlanQual while performing MERGE, we must + * make the modified tuple available to the underlying join query, which is + * using a different RTE from the resultRelation RTE. + * + * 2. rewriteTargetListMerge() requires the RTE of the underlying join in + * order to add junk CTID and TABLEOID attributes. + */ + qry->mergeTarget_relation = transformMergeJoinClause(pstate, (Node *) joinexpr, + &qry->mergeSourceTargetList); + + /* + * The target table referenced in the MERGE is looked up twice; once while + * setting it up as the result relation and again when it's used in the + * underlying the join query. In some rare situations, it may happen that + * these lookups return different results, for example, if a new relation + * with the same name gets created in a schema which is ahead in the + * search_path, in between the two lookups. + * + * It's a very narrow case, but nevertheless we guard against it by simply + * checking if the OIDs returned by the two lookups is the same. If not, we + * just throw an error. + */ + Assert(qry->resultRelation > 0); + Assert(qry->mergeTarget_relation > 0); + + /* Fetch both the RTEs */ + resultRelRTE = rt_fetch(qry->resultRelation, pstate->p_rtable); + mergeRelRTE = rt_fetch(qry->mergeTarget_relation, pstate->p_rtable); + + if (resultRelRTE->relid != mergeRelRTE->relid) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("relation referenced by MERGE statement has changed"))); + + /* + * This query should just provide the source relation columns. Later, in + * preprocess_targetlist(), we shall also add "ctid" attribute of the + * target relation to ensure that the target tuple can be fetched + * correctly. + */ + qry->targetList = qry->mergeSourceTargetList; + + /* qry has no WHERE clause so absent quals are shown as NULL */ + qry->jointree = makeFromExpr(pstate->p_joinlist, NULL); + qry->rtable = pstate->p_rtable; + + /* + * XXX MERGE is unsupported in various cases + */ + if (!(pstate->p_target_relation->rd_rel->relkind == RELKIND_RELATION || + pstate->p_target_relation->rd_rel->relkind == RELKIND_MATVIEW || + pstate->p_target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for this relation type"))); + + if (pstate->p_target_relation->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && + pstate->p_target_relation->rd_rel->relhassubclass) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for relations with inheritance"))); + + if (pstate->p_target_relation->rd_rel->relhasrules) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for relations with rules"))); + + /* + * We now have a good query shape, so now look at the when conditions and + * action targetlists. + * + * Overall, the MERGE Query's targetlist is NIL. + * + * Each individual action has its own targetlist that needs separate + * transformation. These transforms don't do anything to the overall + * targetlist, since that is only used for resjunk columns. + * + * We can reference any column in Target or Source, which is OK because + * both of those already have RTEs. There is nothing like the EXCLUDED + * pseudo-relation for INSERT ON CONFLICT. + */ + foreach(l, stmt->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + /* + * Set namespace for the specific action. This must be done before + * analysing the WHEN quals and the action targetlisst. + */ + setNamespaceForMergeAction(pstate, action); + + /* + * Transform the when condition. + * + * Note that these quals are NOT added to the join quals; instead they + * are evaluated sepaartely during execution to decide which of the + * WHEN MATCHED or WHEN NOT MATCHED actions to execute. + */ + action->qual = transformWhereClause(pstate, action->condition, + EXPR_KIND_MERGE_WHEN_AND, "WHEN"); + + /* + * Transform target lists for each INSERT and UPDATE action stmt + */ + switch (action->commandType) + { + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + SelectStmt *selectStmt = (SelectStmt *) istmt->selectStmt; + List *exprList = NIL; + ListCell *lc; + RangeTblEntry *rte; + ListCell *icols; + ListCell *attnos; + List *icolumns; + List *attrnos; + + pstate->p_is_insert = true; + + icolumns = checkInsertTargets(pstate, istmt->cols, &attrnos); + Assert(list_length(icolumns) == list_length(attrnos)); + + /* + * Handle INSERT much like in transformInsertStmt + */ + if (selectStmt == NULL) + { + /* + * We have INSERT ... DEFAULT VALUES. We can handle + * this case by emitting an empty targetlist --- all + * columns will be defaulted when the planner expands + * the targetlist. + */ + exprList = NIL; + } + else + { + /* + * Process INSERT ... VALUES with a single VALUES + * sublist. We treat this case separately for + * efficiency. The sublist is just computed directly + * as the Query's targetlist, with no VALUES RTE. So + * it works just like a SELECT without any FROM. + */ + List *valuesLists = selectStmt->valuesLists; + + Assert(list_length(valuesLists) == 1); + Assert(selectStmt->intoClause == NULL); + + /* + * Do basic expression transformation (same as a ROW() + * expr, but allow SetToDefault at top level) + */ + exprList = transformExpressionList(pstate, + (List *) linitial(valuesLists), + EXPR_KIND_VALUES_SINGLE, + true); + + /* Prepare row for assignment to target table */ + exprList = transformInsertRow(pstate, exprList, + istmt->cols, + icolumns, attrnos, + false); + } + + /* + * Generate action's target list using the computed list + * of expressions. Also, mark all the target columns as + * needing insert permissions. + */ + rte = pstate->p_target_rangetblentry; + icols = list_head(icolumns); + attnos = list_head(attrnos); + foreach(lc, exprList) + { + Expr *expr = (Expr *) lfirst(lc); + ResTarget *col; + AttrNumber attr_num; + TargetEntry *tle; + + col = lfirst_node(ResTarget, icols); + attr_num = (AttrNumber) lfirst_int(attnos); + + tle = makeTargetEntry(expr, + attr_num, + col->name, + false); + action->targetList = lappend(action->targetList, tle); + + rte->insertedCols = bms_add_member(rte->insertedCols, + attr_num - FirstLowInvalidHeapAttributeNumber); + + icols = lnext(icols); + attnos = lnext(attnos); + } + } + break; + case CMD_UPDATE: + { + UpdateStmt *ustmt = (UpdateStmt *) action->stmt; + + pstate->p_is_insert = false; + action->targetList = transformUpdateTargetList(pstate, ustmt->targetList); + } + break; + case CMD_DELETE: + break; + + case CMD_NOTHING: + action->targetList = NIL; + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } + } + + qry->mergeActionList = stmt->mergeActionList; + + /* XXX maybe later */ + qry->returningList = NULL; + + qry->hasTargetSRFs = false; + qry->hasSubLinks = pstate->p_hasSubLinks; + + assign_query_collations(pstate, qry); + + return qry; +} + +static void +setNamespaceVisibilityForRTE(List *namespace, RangeTblEntry *rte, + bool rel_visible, + bool cols_visible) +{ + ListCell *lc; + + foreach(lc, namespace) + { + ParseNamespaceItem *nsitem = (ParseNamespaceItem *) lfirst(lc); + + if (nsitem->p_rte == rte) + { + nsitem->p_rel_visible = rel_visible; + nsitem->p_cols_visible = cols_visible; + break; + } + } + +} + +/* + * Expand the source relation to include all attributes of this RTE. + * + * This function is very similar to expandRelAttrs except that we don't mark + * columns for SELECT privileges. That will be decided later when we transform + * the action targetlists and the WHEN quals for actual references to the + * source relation. + */ +static List * +expandSourceTL(ParseState *pstate, RangeTblEntry *rte, int rtindex) +{ + List *names, + *vars; + ListCell *name, + *var; + List *te_list = NIL; + + expandRTE(rte, rtindex, 0, -1, false, &names, &vars); + + /* + * Require read access to the table. + */ + rte->requiredPerms |= ACL_SELECT; + + forboth(name, names, var, vars) + { + char *label = strVal(lfirst(name)); + Var *varnode = (Var *) lfirst(var); + TargetEntry *te; + + te = makeTargetEntry((Expr *) varnode, + (AttrNumber) pstate->p_next_resno++, + label, + false); + te_list = lappend(te_list, te); + } + + Assert(name == NULL && var == NULL); /* lists not the same length? */ + + return te_list; +} diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c index 053ae02c9f..5583404e1b 100644 --- a/src/backend/parser/parse_relation.c +++ b/src/backend/parser/parse_relation.c @@ -728,6 +728,16 @@ scanRTEForColumn(ParseState *pstate, RangeTblEntry *rte, const char *colname, colname), parser_errposition(pstate, location))); + /* In MERGE when and condition, no system column is allowed */ + if (pstate->p_expr_kind == EXPR_KIND_MERGE_WHEN_AND && + attnum < InvalidAttrNumber && + !(attnum == TableOidAttributeNumber || attnum == ObjectIdAttributeNumber)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_COLUMN_REFERENCE), + errmsg("system column \"%s\" reference in WHEN AND condition is invalid", + colname), + parser_errposition(pstate, location))); + if (attnum != InvalidAttrNumber) { /* now check to see if column actually is defined */ diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index 66253fc3d3..45875ec289 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1376,6 +1376,53 @@ rewriteTargetListUD(Query *parsetree, RangeTblEntry *target_rte, } } +void +rewriteTargetListMerge(Query *parsetree, Relation target_relation) +{ + Var *var = NULL; + const char *attrname; + TargetEntry *tle; + + Assert(target_relation->rd_rel->relkind == RELKIND_RELATION || + target_relation->rd_rel->relkind == RELKIND_MATVIEW || + target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE); + + /* + * Emit CTID so that executor can find the row to update or delete. + */ + var = makeVar(parsetree->mergeTarget_relation, + SelfItemPointerAttributeNumber, + TIDOID, + -1, + InvalidOid, + 0); + + attrname = "ctid"; + tle = makeTargetEntry((Expr *) var, + list_length(parsetree->targetList) + 1, + pstrdup(attrname), + true); + + parsetree->targetList = lappend(parsetree->targetList, tle); + + /* + * Emit TABLEOID so that executor can find the row to update or delete. + */ + var = makeVar(parsetree->mergeTarget_relation, + TableOidAttributeNumber, + OIDOID, + -1, + InvalidOid, + 0); + + attrname = "tableoid"; + tle = makeTargetEntry((Expr *) var, + list_length(parsetree->targetList) + 1, + pstrdup(attrname), + true); + + parsetree->targetList = lappend(parsetree->targetList, tle); +} /* * matchLocks - @@ -3330,6 +3377,7 @@ RewriteQuery(Query *parsetree, List *rewrite_events) } else if (event == CMD_UPDATE) { + Assert(parsetree->override == OVERRIDING_NOT_SET); parsetree->targetList = rewriteTargetListIU(parsetree->targetList, parsetree->commandType, @@ -3337,6 +3385,50 @@ RewriteQuery(Query *parsetree, List *rewrite_events) rt_entry_relation, parsetree->resultRelation, NULL); } + else if (event == CMD_MERGE) + { + Assert(parsetree->override == OVERRIDING_NOT_SET); + + /* + * Rewrite each action targetlist separately + */ + foreach(lc1, parsetree->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(lc1); + + switch (action->commandType) + { + case CMD_NOTHING: + case CMD_DELETE: /* Nothing to do here */ + break; + case CMD_UPDATE: + action->targetList = + rewriteTargetListIU(action->targetList, + action->commandType, + parsetree->override, + rt_entry_relation, + parsetree->resultRelation, + NULL); + break; + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + + action->targetList = + rewriteTargetListIU(action->targetList, + action->commandType, + istmt->override, + rt_entry_relation, + parsetree->resultRelation, + NULL); + } + break; + default: + elog(ERROR, "unrecognized commandType: %d", action->commandType); + break; + } + } + } else if (event == CMD_DELETE) { /* Nothing to do here */ @@ -3350,13 +3442,19 @@ RewriteQuery(Query *parsetree, List *rewrite_events) locks = matchLocks(event, rt_entry_relation->rd_rules, result_relation, parsetree, &hasUpdate); - product_queries = fireRules(parsetree, - result_relation, - event, - locks, - &instead, - &returning, - &qual_product); + /* + * MERGE doesn't support rules as it's unclear how that could work. + */ + if (event == CMD_MERGE) + product_queries = NIL; + else + product_queries = fireRules(parsetree, + result_relation, + event, + locks, + &instead, + &returning, + &qual_product); /* * If there were no INSTEAD rules, and the target relation is a view diff --git a/src/backend/rewrite/rowsecurity.c b/src/backend/rewrite/rowsecurity.c index ce77a18bc9..6e85886e64 100644 --- a/src/backend/rewrite/rowsecurity.c +++ b/src/backend/rewrite/rowsecurity.c @@ -379,6 +379,95 @@ get_row_security_policies(Query *root, RangeTblEntry *rte, int rt_index, } } + /* + * FOR MERGE, we fetch policies for UPDATE, DELETE and INSERT (and ALL) + * and set them up so that we can enforce the appropriate policy depending + * on the final action we take. + * + * We don't fetch the SELECT policies since they are correctly applied to + * the root->mergeTarget_relation. The target rows are selected after + * joining the mergeTarget_relation and the source relation and hence it's + * enough to apply SELECT policies to the mergeTarget_relation. + * + * We don't push the UPDATE/DELETE USING quals to the RTE because we don't + * really want to apply them while scanning the relation since we don't + * know whether we will be doing a UPDATE or a DELETE at the end. We apply + * the respective policy once we decide the final action on the target + * tuple. + * + * XXX We are setting up USING quals as WITH CHECK. If RLS prohibits + * UPDATE/DELETE on the target row, we shall throw an error instead of + * silently ignoring the row. This is different than how normal + * UPDATE/DELETE works and more in line with INSERT ON CONFLICT DO UPDATE + * handling. + */ + if (commandType == CMD_MERGE) + { + List *merge_permissive_policies; + List *merge_restrictive_policies; + + /* + * Fetch the UPDATE policies and set them up to execute on the + * existing target row before doing UPDATE. + */ + get_policies_for_relation(rel, CMD_UPDATE, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + /* + * WCO_RLS_MERGE_UPDATE_CHECK is used to check UPDATE USING quals on + * the existing target row. + */ + add_with_check_options(rel, rt_index, + WCO_RLS_MERGE_UPDATE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + true); + + /* + * Same with DELETE policies. + */ + get_policies_for_relation(rel, CMD_DELETE, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + add_with_check_options(rel, rt_index, + WCO_RLS_MERGE_DELETE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + true); + + /* + * No special handling is required for INSERT policies. They will be + * checked and enforced during ExecInsert(). But we must add them to + * withCheckOptions. + */ + get_policies_for_relation(rel, CMD_INSERT, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + add_with_check_options(rel, rt_index, + WCO_RLS_INSERT_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + false); + + /* Enforce the WITH CHECK clauses of the UPDATE policies */ + add_with_check_options(rel, rt_index, + WCO_RLS_UPDATE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + false); + } + heap_close(rel, NoLock); /* @@ -438,6 +527,14 @@ get_policies_for_relation(Relation relation, CmdType cmd, Oid user_id, if (policy->polcmd == ACL_DELETE_CHR) cmd_matches = true; break; + case CMD_MERGE: + + /* + * We do not support a separate policy for MERGE command. + * Instead it derives from the policies defined for other + * commands. + */ + break; default: elog(ERROR, "unrecognized policy command type %d", (int) cmd); diff --git a/src/backend/tcop/pquery.c b/src/backend/tcop/pquery.c index 66cc5c35c6..50f852a4aa 100644 --- a/src/backend/tcop/pquery.c +++ b/src/backend/tcop/pquery.c @@ -193,6 +193,11 @@ ProcessQuery(PlannedStmt *plan, "DELETE " UINT64_FORMAT, queryDesc->estate->es_processed); break; + case CMD_MERGE: + snprintf(completionTag, COMPLETION_TAG_BUFSIZE, + "MERGE " UINT64_FORMAT, + queryDesc->estate->es_processed); + break; default: strcpy(completionTag, "???"); break; diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index e144583bd1..2521944b9d 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -110,6 +110,7 @@ CommandIsReadOnly(PlannedStmt *pstmt) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: return false; case CMD_UTILITY: /* For now, treat all utility commands as read/write */ @@ -1832,6 +1833,8 @@ QueryReturnsTuples(Query *parsetree) case CMD_SELECT: /* returns tuples */ return true; + case CMD_MERGE: + return false; case CMD_INSERT: case CMD_UPDATE: case CMD_DELETE: @@ -2076,6 +2079,10 @@ CreateCommandTag(Node *parsetree) tag = "UPDATE"; break; + case T_MergeStmt: + tag = "MERGE"; + break; + case T_SelectStmt: tag = "SELECT"; break; @@ -2819,6 +2826,9 @@ CreateCommandTag(Node *parsetree) case CMD_DELETE: tag = "DELETE"; break; + case CMD_MERGE: + tag = "MERGE"; + break; case CMD_UTILITY: tag = CreateCommandTag(stmt->utilityStmt); break; @@ -2879,6 +2889,9 @@ CreateCommandTag(Node *parsetree) case CMD_DELETE: tag = "DELETE"; break; + case CMD_MERGE: + tag = "MERGE"; + break; case CMD_UTILITY: tag = CreateCommandTag(stmt->utilityStmt); break; @@ -2927,6 +2940,7 @@ GetCommandLogLevel(Node *parsetree) case T_InsertStmt: case T_DeleteStmt: case T_UpdateStmt: + case T_MergeStmt: lev = LOGSTMT_MOD; break; @@ -3366,6 +3380,7 @@ GetCommandLogLevel(Node *parsetree) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: lev = LOGSTMT_MOD; break; @@ -3396,6 +3411,7 @@ GetCommandLogLevel(Node *parsetree) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: lev = LOGSTMT_MOD; break; diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 4c0256b18a..100174138d 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -67,6 +67,7 @@ typedef enum LockTupleMode */ typedef struct HeapUpdateFailureData { + HTSU_Result result; ItemPointerData ctid; TransactionId xmax; CommandId cmax; diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h index a5b8610fa2..1b79a80310 100644 --- a/src/include/commands/trigger.h +++ b/src/include/commands/trigger.h @@ -206,7 +206,8 @@ extern bool ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, - HeapTuple fdw_trigtuple); + HeapTuple fdw_trigtuple, + HeapUpdateFailureData *hufdp); extern void ExecARDeleteTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, @@ -225,7 +226,8 @@ extern TupleTableSlot *ExecBRUpdateTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, HeapTuple fdw_trigtuple, - TupleTableSlot *slot); + TupleTableSlot *slot, + HeapUpdateFailureData *hufdp); extern void ExecARUpdateTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, diff --git a/src/include/executor/execPartition.h b/src/include/executor/execPartition.h index 03a599ad57..9f55f6409e 100644 --- a/src/include/executor/execPartition.h +++ b/src/include/executor/execPartition.h @@ -114,6 +114,7 @@ extern int ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, TupleTableSlot *slot, EState *estate); +extern int ExecFindPartitionByOid(PartitionTupleRouting *proute, Oid partoid); extern ResultRelInfo *ExecInitPartitionInfo(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, PartitionTupleRouting *proute, diff --git a/src/include/executor/nodeMerge.h b/src/include/executor/nodeMerge.h new file mode 100644 index 0000000000..c222e9ee65 --- /dev/null +++ b/src/include/executor/nodeMerge.h @@ -0,0 +1,22 @@ +/*------------------------------------------------------------------------- + * + * nodeMerge.h + * + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/executor/nodeMerge.h + * + *------------------------------------------------------------------------- + */ +#ifndef NODEMERGE_H +#define NODEMERGE_H + +#include "nodes/execnodes.h" + +extern void +ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, + JunkFilter *junkfilter, ResultRelInfo *resultRelInfo); + +#endif /* NODEMERGE_H */ diff --git a/src/include/executor/nodeModifyTable.h b/src/include/executor/nodeModifyTable.h index 0d7e579e1c..686cfa6171 100644 --- a/src/include/executor/nodeModifyTable.h +++ b/src/include/executor/nodeModifyTable.h @@ -18,5 +18,26 @@ extern ModifyTableState *ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags); extern void ExecEndModifyTable(ModifyTableState *node); extern void ExecReScanModifyTable(ModifyTableState *node); +extern TupleTableSlot *ExecPrepareTupleRouting(ModifyTableState *mtstate, + EState *estate, + struct PartitionTupleRouting *proute, + ResultRelInfo *targetRelInfo, + TupleTableSlot *slot); +extern TupleTableSlot *ExecDelete(ModifyTableState *mtstate, + ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *planSlot, + EPQState *epqstate, EState *estate, bool *tupleDeleted, + bool processReturning, HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag); +extern TupleTableSlot *ExecUpdate(ModifyTableState *mtstate, + ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *slot, + TupleTableSlot *planSlot, EPQState *epqstate, EState *estate, + bool *tuple_updated, HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag); +extern TupleTableSlot *ExecInsert(ModifyTableState *mtstate, + TupleTableSlot *slot, + TupleTableSlot *planSlot, + EState *estate, + MergeActionState *actionState, + bool canSetTag); #endif /* NODEMODIFYTABLE_H */ diff --git a/src/include/executor/spi.h b/src/include/executor/spi.h index e5bdaecc4e..78410b9f77 100644 --- a/src/include/executor/spi.h +++ b/src/include/executor/spi.h @@ -64,6 +64,7 @@ typedef struct _SPI_plan *SPIPlanPtr; #define SPI_OK_REL_REGISTER 15 #define SPI_OK_REL_UNREGISTER 16 #define SPI_OK_TD_REGISTER 17 +#define SPI_OK_MERGE 18 #define SPI_OPT_NONATOMIC (1 << 0) diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 2c2d2823c0..b4ee132bb4 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -360,8 +360,17 @@ typedef struct JunkFilter AttrNumber *jf_cleanMap; TupleTableSlot *jf_resultSlot; AttrNumber jf_junkAttNo; + AttrNumber jf_otherJunkAttNo; } JunkFilter; +typedef struct MergeState +{ + /* List of MERGE MATCHED action states */ + List *matchedActionStates; + /* List of MERGE NOT MATCHED action states */ + List *notMatchedActionStates; +} MergeState; + /* * OnConflictSetState * @@ -452,8 +461,38 @@ typedef struct ResultRelInfo /* relation descriptor for root partitioned table */ Relation ri_PartitionRoot; + + int ri_PartitionLeafIndex; + /* for running MERGE on this result relation */ + MergeState *ri_mergeState; + + /* + * While executing MERGE, the target relation is processed twice; once to + * as a result relation and to run a join between the target and the + * source. We generate two different RTEs for these two purposes, one with + * rte->inh set to false and other with rte->inh set to true. + * + * Since the plan re-evaluated by EvalPlanQual uses the second RTE, we must + * install the updated tuple in the scan corresponding to that RTE. The + * following member tracks the index of the second RTE for EvalPlanQual + * purposes. ri_mergeTargetRTI is non-zero only when MERGE is in-progress. + * We use ri_mergeTargetRTI to run EvalPlanQual for MERGE and + * ri_RangeTableIndex elsewhere. + */ + Index ri_mergeTargetRTI; } ResultRelInfo; +/* + * Get the Range table index for EvalPlanQual. + * + * We use the ri_mergeTargetRTI if set, otherwise use ri_RangeTableIndex. + * ri_mergeTargetRTI should really be ever set iff we're running MERGE. + */ +#define GetEPQRangeTableIndex(r) \ + (((r)->ri_mergeTargetRTI > 0) \ + ? (r)->ri_mergeTargetRTI \ + : (r)->ri_RangeTableIndex) + /* ---------------- * EState information * @@ -1004,6 +1043,24 @@ typedef struct ProjectSetState MemoryContext argcontext; /* context for SRF arguments */ } ProjectSetState; +/* ---------------- + * MergeActionState information + * ---------------- + */ +typedef struct MergeActionState +{ + NodeTag type; + bool matched; /* true if a WHEN MATCHED action, + * false if a NOT MATCHED action. */ + ExprState *whenqual; /* additional conditions attached to + * WHEN [NOT] MATCHED clause */ + CmdType commandType; /* INSERT/UPDATE/DELETE/DO NOTHING */ + ProjectionInfo *proj; /* projection information for the tuple + * produced by this action */ + TupleDesc tupDesc; /* tuple descriptor associated with the + * projection */ +} MergeActionState; + /* ---------------- * ModifyTableState information * ---------------- @@ -1011,7 +1068,7 @@ typedef struct ProjectSetState typedef struct ModifyTableState { PlanState ps; /* its first field is NodeTag */ - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ bool mt_done; /* are we done? */ PlanState **mt_plans; /* subplans (one per target rel) */ @@ -1027,6 +1084,8 @@ typedef struct ModifyTableState List *mt_excludedtlist; /* the excluded pseudo relation's tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection target */ + TupleTableSlot *mt_mergeproj; /* MERGE action projection target */ + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; @@ -1038,6 +1097,9 @@ typedef struct ModifyTableState /* Per plan map for tuple conversion from child to root */ TupleConversionMap **mt_per_subplan_tupconv_maps; + + int mt_merge_subcommands; /* Flags show which cmd types are + * present */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 443de22704..fce48026b6 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -97,6 +97,7 @@ typedef enum NodeTag T_PlanState, T_ResultState, T_ProjectSetState, + T_MergeActionState, T_ModifyTableState, T_AppendState, T_MergeAppendState, @@ -308,6 +309,8 @@ typedef enum NodeTag T_InsertStmt, T_DeleteStmt, T_UpdateStmt, + T_MergeStmt, + T_MergeAction, T_SelectStmt, T_AlterTableStmt, T_AlterTableCmd, @@ -657,7 +660,8 @@ typedef enum CmdType CMD_SELECT, /* select stmt */ CMD_UPDATE, /* update stmt */ CMD_INSERT, /* insert stmt */ - CMD_DELETE, + CMD_DELETE, /* delete stmt */ + CMD_MERGE, /* merge stmt */ CMD_UTILITY, /* cmds like create, destroy, copy, vacuum, * etc. */ CMD_NOTHING /* dummy command for instead nothing rules diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 92082b3a7a..0c904f4d7f 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -38,7 +38,7 @@ typedef enum OverridingKind typedef enum QuerySource { QSRC_ORIGINAL, /* original parsetree (explicit query) */ - QSRC_PARSER, /* added by parse analysis (now unused) */ + QSRC_PARSER, /* added by parse analysis in MERGE */ QSRC_INSTEAD_RULE, /* added by unconditional INSTEAD rule */ QSRC_QUAL_INSTEAD_RULE, /* added by conditional INSTEAD rule */ QSRC_NON_INSTEAD_RULE /* added by non-INSTEAD rule */ @@ -107,7 +107,7 @@ typedef struct Query { NodeTag type; - CmdType commandType; /* select|insert|update|delete|utility */ + CmdType commandType; /* select|insert|update|delete|merge|utility */ QuerySource querySource; /* where did I come from? */ @@ -118,7 +118,7 @@ typedef struct Query Node *utilityStmt; /* non-null if commandType == CMD_UTILITY */ int resultRelation; /* rtable index of target relation for - * INSERT/UPDATE/DELETE; 0 for SELECT */ + * INSERT/UPDATE/DELETE/MERGE; 0 for SELECT */ bool hasAggs; /* has aggregates in tlist or havingQual */ bool hasWindowFuncs; /* has window functions in tlist */ @@ -169,6 +169,9 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's, which are * only added during rewrite and therefore * are not written out as part of Query. */ + int mergeTarget_relation; + List *mergeSourceTargetList; + List *mergeActionList; /* list of actions for MERGE (only) */ /* * The following two fields identify the portion of the source text string @@ -1128,7 +1131,9 @@ typedef enum WCOKind WCO_VIEW_CHECK, /* WCO on an auto-updatable view */ WCO_RLS_INSERT_CHECK, /* RLS INSERT WITH CHECK policy */ WCO_RLS_UPDATE_CHECK, /* RLS UPDATE WITH CHECK policy */ - WCO_RLS_CONFLICT_CHECK /* RLS ON CONFLICT DO UPDATE USING policy */ + WCO_RLS_CONFLICT_CHECK, /* RLS ON CONFLICT DO UPDATE USING policy */ + WCO_RLS_MERGE_UPDATE_CHECK, /* RLS MERGE UPDATE USING policy */ + WCO_RLS_MERGE_DELETE_CHECK /* RLS MERGE DELETE USING policy */ } WCOKind; typedef struct WithCheckOption @@ -1503,6 +1508,32 @@ typedef struct UpdateStmt WithClause *withClause; /* WITH clause */ } UpdateStmt; +/* ---------------------- + * Merge Statement + * ---------------------- + */ +typedef struct MergeStmt +{ + NodeTag type; + RangeVar *relation; /* target relation to merge */ + Node *source_relation; /* source relation */ + Node *join_condition; /* join condition between source and target */ + List *mergeActionList; /* list of MergeAction(s) */ +} MergeStmt; + +typedef struct MergeAction +{ + NodeTag type; + bool matched; /* true if a WHEN MATCHED action, + * false if a WHEN NOT MATCHED action */ + Node *condition; /* WHEN AND conditions (raw parser) */ + Node *qual; /* transformed WHEN AND conditions */ + CmdType commandType; /* type of action - INSERT/UPDATE/DELETE/DO + * NOTHING */ + Node *stmt; /* T_UpdateStmt etc */ + List *targetList; /* the target list (of ResTarget) */ +} MergeAction; + /* ---------------------- * Select Statement * diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index c922216b7d..0a797f0a05 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -18,6 +18,7 @@ #include "lib/stringinfo.h" #include "nodes/bitmapset.h" #include "nodes/lockoptions.h" +#include "nodes/parsenodes.h" #include "nodes/primnodes.h" @@ -42,7 +43,7 @@ typedef struct PlannedStmt { NodeTag type; - CmdType commandType; /* select|insert|update|delete|utility */ + CmdType commandType; /* select|insert|update|delete|merge|utility */ uint64 queryId; /* query identifier (copied from Query) */ @@ -216,13 +217,14 @@ typedef struct ProjectSet typedef struct ModifyTable { Plan plan; - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ Index nominalRelation; /* Parent RT index for use of EXPLAIN */ /* RT indexes of non-leaf tables in a partition tree */ List *partitioned_rels; bool partColsUpdated; /* some part key in hierarchy updated */ List *resultRelations; /* integer list of RT indexes */ + Index mergeTargetRelation; /* RT index of the merge target */ int resultRelIndex; /* index of first resultRel in plan's list */ int rootResultRelIndex; /* index of the partitioned table root */ List *plans; /* plan(s) producing source data */ @@ -238,6 +240,8 @@ typedef struct ModifyTable Node *onConflictWhere; /* WHERE for ON CONFLICT UPDATE */ Index exclRelRTI; /* RTI of the EXCLUDED pseudo relation */ List *exclRelTlist; /* tlist of the EXCLUDED pseudo relation */ + List *mergeSourceTargetList; + List *mergeActionList; /* actions for MERGE */ } ModifyTable; /* ---------------- diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h index abbbda9e91..91dfff4cb5 100644 --- a/src/include/nodes/relation.h +++ b/src/include/nodes/relation.h @@ -1670,7 +1670,7 @@ typedef struct LockRowsPath } LockRowsPath; /* - * ModifyTablePath represents performing INSERT/UPDATE/DELETE modifications + * ModifyTablePath represents performing INSERT/UPDATE/DELETE/MERGE * * We represent most things that will be in the ModifyTable plan node * literally, except we have child Path(s) not Plan(s). But analysis of the @@ -1679,13 +1679,14 @@ typedef struct LockRowsPath typedef struct ModifyTablePath { Path path; - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ Index nominalRelation; /* Parent RT index for use of EXPLAIN */ /* RT indexes of non-leaf tables in a partition tree */ List *partitioned_rels; bool partColsUpdated; /* some part key in hierarchy updated */ List *resultRelations; /* integer list of RT indexes */ + Index mergeTargetRelation;/* RT index of merge target relation */ List *subpaths; /* Path(s) producing source data */ List *subroots; /* per-target-table PlannerInfos */ List *withCheckOptionLists; /* per-target-table WCO lists */ @@ -1693,6 +1694,8 @@ typedef struct ModifyTablePath List *rowMarks; /* PlanRowMarks (non-locking only) */ OnConflictExpr *onconflict; /* ON CONFLICT clause, or NULL */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ + List *mergeSourceTargetList; + List *mergeActionList; /* actions for MERGE */ } ModifyTablePath; /* diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index 381bc30813..895bf6959d 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -241,11 +241,14 @@ extern ModifyTablePath *create_modifytable_path(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subpaths, + List *resultRelations, + Index mergeTargetRelation, + List *subpaths, List *subroots, List *withCheckOptionLists, List *returningLists, List *rowMarks, OnConflictExpr *onconflict, - int epqParam); + List *mergeSourceTargetList, + List *mergeActionList, int epqParam); extern LimitPath *create_limit_path(PlannerInfo *root, RelOptInfo *rel, Path *subpath, Node *limitOffset, Node *limitCount, diff --git a/src/include/parser/analyze.h b/src/include/parser/analyze.h index 687ae1b5b7..41fb10666e 100644 --- a/src/include/parser/analyze.h +++ b/src/include/parser/analyze.h @@ -32,6 +32,11 @@ extern Query *parse_sub_analyze(Node *parseTree, ParseState *parentParseState, bool locked_from_parent, bool resolve_unknowns); +extern List *transformInsertRow(ParseState *pstate, List *exprlist, + List *stmtcols, List *icolumns, List *attrnos, + bool strip_indirection); +extern List *transformUpdateTargetList(ParseState *pstate, + List *targetList); extern Query *transformTopLevelStmt(ParseState *pstate, RawStmt *parseTree); extern Query *transformStmt(ParseState *pstate, Node *parseTree); diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index cf32197bc3..4dff55a8e9 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -244,8 +244,10 @@ PG_KEYWORD("locked", LOCKED, UNRESERVED_KEYWORD) PG_KEYWORD("logged", LOGGED, UNRESERVED_KEYWORD) PG_KEYWORD("mapping", MAPPING, UNRESERVED_KEYWORD) PG_KEYWORD("match", MATCH, UNRESERVED_KEYWORD) +PG_KEYWORD("matched", MATCHED, UNRESERVED_KEYWORD) PG_KEYWORD("materialized", MATERIALIZED, UNRESERVED_KEYWORD) PG_KEYWORD("maxvalue", MAXVALUE, UNRESERVED_KEYWORD) +PG_KEYWORD("merge", MERGE, UNRESERVED_KEYWORD) PG_KEYWORD("method", METHOD, UNRESERVED_KEYWORD) PG_KEYWORD("minute", MINUTE_P, UNRESERVED_KEYWORD) PG_KEYWORD("minvalue", MINVALUE, UNRESERVED_KEYWORD) diff --git a/src/include/parser/parse_clause.h b/src/include/parser/parse_clause.h index 2c0e092862..30121c98ed 100644 --- a/src/include/parser/parse_clause.h +++ b/src/include/parser/parse_clause.h @@ -20,7 +20,10 @@ extern void transformFromClause(ParseState *pstate, List *frmList); extern int setTargetTable(ParseState *pstate, RangeVar *relation, bool inh, bool alsoSource, AclMode requiredPerms); extern bool interpretOidsOption(List *defList, bool allowOids); - +extern Node *transformFromClauseItem(ParseState *pstate, Node *n, + RangeTblEntry **top_rte, int *top_rti, + RangeTblEntry **right_rte, int *right_rti, + List **namespace); extern Node *transformWhereClause(ParseState *pstate, Node *clause, ParseExprKind exprKind, const char *constructName); extern Node *transformLimitClause(ParseState *pstate, Node *clause, diff --git a/src/include/parser/parse_merge.h b/src/include/parser/parse_merge.h new file mode 100644 index 0000000000..0151809e09 --- /dev/null +++ b/src/include/parser/parse_merge.h @@ -0,0 +1,19 @@ +/*------------------------------------------------------------------------- + * + * parse_merge.h + * handle merge-stmt in parser + * + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/parser/parse_merge.h + * + *------------------------------------------------------------------------- + */ +#ifndef PARSE_MERGE_H +#define PARSE_MERGE_H + +#include "parser/parse_node.h" +extern Query *transformMergeStmt(ParseState *pstate, MergeStmt *stmt); +#endif diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index 0230543810..8b80e79fd2 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -50,6 +50,7 @@ typedef enum ParseExprKind EXPR_KIND_INSERT_TARGET, /* INSERT target list item */ EXPR_KIND_UPDATE_SOURCE, /* UPDATE assignment source item */ EXPR_KIND_UPDATE_TARGET, /* UPDATE assignment target item */ + EXPR_KIND_MERGE_WHEN_AND, /* MERGE WHEN ... AND condition */ EXPR_KIND_GROUP_BY, /* GROUP BY */ EXPR_KIND_ORDER_BY, /* ORDER BY */ EXPR_KIND_DISTINCT_ON, /* DISTINCT ON */ @@ -127,7 +128,8 @@ typedef Node *(*CoerceParamHook) (ParseState *pstate, Param *param, * p_parent_cte: CommonTableExpr that immediately contains the current query, * if any. * - * p_target_relation: target relation, if query is INSERT, UPDATE, or DELETE. + * p_target_relation: target relation, if query is INSERT, UPDATE, DELETE + * or MERGE. * * p_target_rangetblentry: target relation's entry in the rtable list. * @@ -181,7 +183,7 @@ struct ParseState List *p_ctenamespace; /* current namespace for common table exprs */ List *p_future_ctes; /* common table exprs not yet in namespace */ CommonTableExpr *p_parent_cte; /* this query's containing CTE */ - Relation p_target_relation; /* INSERT/UPDATE/DELETE target rel */ + Relation p_target_relation; /* INSERT/UPDATE/DELETE/MERGE target rel */ RangeTblEntry *p_target_rangetblentry; /* target rel's RTE */ bool p_is_insert; /* process assignment like INSERT not UPDATE */ List *p_windowdefs; /* raw representations of window clauses */ diff --git a/src/include/rewrite/rewriteHandler.h b/src/include/rewrite/rewriteHandler.h index 8128199fc3..1ab5de3942 100644 --- a/src/include/rewrite/rewriteHandler.h +++ b/src/include/rewrite/rewriteHandler.h @@ -25,6 +25,7 @@ extern void AcquireRewriteLocks(Query *parsetree, extern Node *build_column_default(Relation rel, int attrno); extern void rewriteTargetListUD(Query *parsetree, RangeTblEntry *target_rte, Relation target_relation); +extern void rewriteTargetListMerge(Query *parsetree, Relation target_relation); extern Query *get_view_query(Relation view); extern const char *view_query_is_auto_updatable(Query *viewquery, diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c index 4c0114c514..a432636322 100644 --- a/src/interfaces/libpq/fe-exec.c +++ b/src/interfaces/libpq/fe-exec.c @@ -3055,9 +3055,9 @@ PQoidValue(const PGresult *res) /* * PQcmdTuples - - * If the last command was INSERT/UPDATE/DELETE/MOVE/FETCH/COPY, return - * a string containing the number of inserted/affected tuples. If not, - * return "". + * If the last command was INSERT/UPDATE/DELETE/MERGE/MOVE/FETCH/COPY, + * return a string containing the number of inserted/affected tuples. + * If not, return "". * * XXX: this should probably return an int */ @@ -3084,7 +3084,8 @@ PQcmdTuples(PGresult *res) strncmp(res->cmdStatus, "DELETE ", 7) == 0 || strncmp(res->cmdStatus, "UPDATE ", 7) == 0) p = res->cmdStatus + 7; - else if (strncmp(res->cmdStatus, "FETCH ", 6) == 0) + else if (strncmp(res->cmdStatus, "FETCH ", 6) == 0 || + strncmp(res->cmdStatus, "MERGE ", 6) == 0) p = res->cmdStatus + 6; else if (strncmp(res->cmdStatus, "MOVE ", 5) == 0 || strncmp(res->cmdStatus, "COPY ", 5) == 0) diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c index 38ea7a091f..8a1d32a95c 100644 --- a/src/pl/plpgsql/src/pl_exec.c +++ b/src/pl/plpgsql/src/pl_exec.c @@ -3932,7 +3932,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, /* * On the first call for this statement generate the plan, and detect - * whether the statement is INSERT/UPDATE/DELETE + * whether the statement is INSERT/UPDATE/DELETE/MERGE */ if (expr->plan == NULL) { @@ -3953,6 +3953,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, { if (q->commandType == CMD_INSERT || q->commandType == CMD_UPDATE || + q->commandType == CMD_MERGE || q->commandType == CMD_DELETE) stmt->mod_stmt = true; } @@ -4010,6 +4011,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, case SPI_OK_INSERT_RETURNING: case SPI_OK_UPDATE_RETURNING: case SPI_OK_DELETE_RETURNING: + case SPI_OK_MERGE: Assert(stmt->mod_stmt); exec_set_found(estate, (SPI_processed != 0)); break; @@ -4187,6 +4189,7 @@ exec_stmt_dynexecute(PLpgSQL_execstate *estate, case SPI_OK_INSERT_RETURNING: case SPI_OK_UPDATE_RETURNING: case SPI_OK_DELETE_RETURNING: + case SPI_OK_MERGE: case SPI_OK_UTILITY: case SPI_OK_REWRITTEN: break; diff --git a/src/pl/plpgsql/src/pl_gram.y b/src/pl/plpgsql/src/pl_gram.y index 4c80936678..7d9fb2f039 100644 --- a/src/pl/plpgsql/src/pl_gram.y +++ b/src/pl/plpgsql/src/pl_gram.y @@ -303,6 +303,7 @@ static void check_raise_parameters(PLpgSQL_stmt_raise *stmt); %token <keyword> K_LAST %token <keyword> K_LOG %token <keyword> K_LOOP +%token <keyword> K_MERGE %token <keyword> K_MESSAGE %token <keyword> K_MESSAGE_TEXT %token <keyword> K_MOVE @@ -1930,6 +1931,10 @@ stmt_execsql : K_IMPORT { $$ = make_execsql_stmt(K_INSERT, @1); } + | K_MERGE + { + $$ = make_execsql_stmt(K_MERGE, @1); + } | T_WORD { int tok; @@ -2451,6 +2456,7 @@ unreserved_keyword : | K_IS | K_LAST | K_LOG + | K_MERGE | K_MESSAGE | K_MESSAGE_TEXT | K_MOVE @@ -2912,6 +2918,8 @@ make_execsql_stmt(int firsttoken, int location) { if (prev_tok == K_INSERT) continue; /* INSERT INTO is not an INTO-target */ + if (prev_tok == K_MERGE) + continue; /* MERGE INTO is not an INTO-target */ if (firsttoken == K_IMPORT) continue; /* IMPORT ... INTO is not an INTO-target */ if (have_into) diff --git a/src/pl/plpgsql/src/pl_scanner.c b/src/pl/plpgsql/src/pl_scanner.c index 65774f9902..85078ac15b 100644 --- a/src/pl/plpgsql/src/pl_scanner.c +++ b/src/pl/plpgsql/src/pl_scanner.c @@ -137,6 +137,7 @@ static const ScanKeyword unreserved_keywords[] = { PG_KEYWORD("is", K_IS, UNRESERVED_KEYWORD) PG_KEYWORD("last", K_LAST, UNRESERVED_KEYWORD) PG_KEYWORD("log", K_LOG, UNRESERVED_KEYWORD) + PG_KEYWORD("merge", K_MERGE, UNRESERVED_KEYWORD) PG_KEYWORD("message", K_MESSAGE, UNRESERVED_KEYWORD) PG_KEYWORD("message_text", K_MESSAGE_TEXT, UNRESERVED_KEYWORD) PG_KEYWORD("move", K_MOVE, UNRESERVED_KEYWORD) diff --git a/src/pl/plpgsql/src/plpgsql.h b/src/pl/plpgsql/src/plpgsql.h index f7619a63f9..8d6ef3326f 100644 --- a/src/pl/plpgsql/src/plpgsql.h +++ b/src/pl/plpgsql/src/plpgsql.h @@ -845,8 +845,8 @@ typedef struct PLpgSQL_stmt_execsql PLpgSQL_stmt_type cmd_type; int lineno; PLpgSQL_expr *sqlstmt; - bool mod_stmt; /* is the stmt INSERT/UPDATE/DELETE? Note: - * mod_stmt is set when we plan the query */ + bool mod_stmt; /* is the stmt INSERT/UPDATE/DELETE/MERGE? + * Note mod_stmt is set when we plan the query */ bool into; /* INTO supplied? */ bool strict; /* INTO STRICT flag */ PLpgSQL_variable *target; /* INTO target (record or row) */ diff --git a/src/test/isolation/expected/merge-delete.out b/src/test/isolation/expected/merge-delete.out new file mode 100644 index 0000000000..40e62901b7 --- /dev/null +++ b/src/test/isolation/expected/merge-delete.out @@ -0,0 +1,97 @@ +Parsed test spec with 2 sessions + +starting permutation: delete c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete c1 update1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete c1 update1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete c1 merge2 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: merge_delete c1 merge2 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: delete update1 c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; <waiting ...> +step c1: COMMIT; +step update1: <... completed> +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete update1 c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; <waiting ...> +step c1: COMMIT; +step update1: <... completed> +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete merge2 c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: merge_delete merge2 c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; diff --git a/src/test/isolation/expected/merge-insert-update.out b/src/test/isolation/expected/merge-insert-update.out new file mode 100644 index 0000000000..317fa16a3d --- /dev/null +++ b/src/test/isolation/expected/merge-insert-update.out @@ -0,0 +1,84 @@ +Parsed test spec with 2 sessions + +starting permutation: merge1 c1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +1 merge1 +step c2: COMMIT; + +starting permutation: merge1 c1 merge2 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step select2: SELECT * FROM target; +key val + +1 merge1 updated by merge2 +step c2: COMMIT; + +starting permutation: insert1 merge2 c1 select2 c2 +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: merge1 merge2 c1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: merge1 merge2 a1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step a1: ABORT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2 +step c2: COMMIT; + +starting permutation: delete1 insert1 c1 merge2 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step select2: SELECT * FROM target; +key val + +1 insert1 updated by merge2 +step c2: COMMIT; + +starting permutation: delete1 insert1 merge2 c1 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: delete1 insert1 merge2i c1 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2i: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +1 insert1 +step c2: COMMIT; diff --git a/src/test/isolation/expected/merge-match-recheck.out b/src/test/isolation/expected/merge-match-recheck.out new file mode 100644 index 0000000000..96a9f45ac8 --- /dev/null +++ b/src/test/isolation/expected/merge-match-recheck.out @@ -0,0 +1,106 @@ +Parsed test spec with 2 sessions + +starting permutation: update1 merge_status c2 select1 c1 +step update1: UPDATE target t SET balance = balance + 10, val = t.val || ' updated by update1' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 170 s2 setup updated by update1 when1 +step c1: COMMIT; + +starting permutation: update2 merge_status c2 select1 c1 +step update2: UPDATE target t SET status = 's2', val = t.val || ' updated by update2' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s3 setup updated by update2 when2 +step c1: COMMIT; + +starting permutation: update3 merge_status c2 select1 c1 +step update3: UPDATE target t SET status = 's3', val = t.val || ' updated by update3' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s4 setup updated by update3 when3 +step c1: COMMIT; + +starting permutation: update5 merge_status c2 select1 c1 +step update5: UPDATE target t SET status = 's5', val = t.val || ' updated by update5' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s5 setup updated by update5 +step c1: COMMIT; + +starting permutation: update_bal1 merge_bal c2 select1 c1 +step update_bal1: UPDATE target t SET balance = 50, val = t.val || ' updated by update_bal1' WHERE t.key = 1; +step merge_bal: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND balance < 100 THEN + UPDATE SET balance = balance * 2, val = t.val || ' when1' + WHEN MATCHED AND balance < 200 THEN + UPDATE SET balance = balance * 4, val = t.val || ' when2' + WHEN MATCHED AND balance < 300 THEN + UPDATE SET balance = balance * 8, val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_bal: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 100 s1 setup updated by update_bal1 when1 +step c1: COMMIT; diff --git a/src/test/isolation/expected/merge-update.out b/src/test/isolation/expected/merge-update.out new file mode 100644 index 0000000000..60ae42ebd0 --- /dev/null +++ b/src/test/isolation/expected/merge-update.out @@ -0,0 +1,213 @@ +Parsed test spec with 2 sessions + +starting permutation: merge1 c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +step c2: COMMIT; + +starting permutation: merge1 c1 merge2a select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step c1: COMMIT; +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2a c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2a: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2a a1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step a1: ABORT; +step merge2a: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2b c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2b: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2b' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED AND t.key < 2 THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2b: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2b +step c2: COMMIT; + +starting permutation: merge1 merge2c c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2c: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2c' as val) s + ON s.key = t.key AND t.key < 2 + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2c: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2c +step c2: COMMIT; + +starting permutation: pa_merge1 pa_merge2a c1 pa_select2 c2 +step pa_merge1: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set val = t.val || ' updated by ' || s.val; + +step pa_merge2a: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step pa_merge2a: <... completed> +step pa_select2: SELECT * FROM pa_target; +key val + +2 initial +2 initial updated by pa_merge2a +step c2: COMMIT; + +starting permutation: pa_merge2 pa_merge2a c1 pa_select2 c2 +step pa_merge2: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step pa_merge2a: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step pa_merge2a: <... completed> +step pa_select2: SELECT * FROM pa_target; +key val + +1 pa_merge2a +2 initial +2 initial updated by pa_merge1 +step c2: COMMIT; diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index 74d7d59546..644e071112 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -33,6 +33,10 @@ test: insert-conflict-do-update test: insert-conflict-do-update-2 test: insert-conflict-do-update-3 test: insert-conflict-toast +test: merge-insert-update +test: merge-delete +test: merge-update +test: merge-match-recheck test: delete-abort-savept test: delete-abort-savept-2 test: aborted-keyrevoke diff --git a/src/test/isolation/specs/merge-delete.spec b/src/test/isolation/specs/merge-delete.spec new file mode 100644 index 0000000000..656954f847 --- /dev/null +++ b/src/test/isolation/specs/merge-delete.spec @@ -0,0 +1,51 @@ +# MERGE DELETE +# +# This test looks at the interactions involving concurrent deletes +# comparing the behavior of MERGE, DELETE and UPDATE + +setup +{ + CREATE TABLE target (key int primary key, val text); + INSERT INTO target VALUES (1, 'setup1'); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "delete" { DELETE FROM target t WHERE t.key = 1; } +step "merge_delete" { MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "update1" { UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; } +step "merge2" { MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; } +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } + +# Basic effects +permutation "delete" "c1" "select2" "c2" +permutation "merge_delete" "c1" "select2" "c2" + +# One after the other, no concurrency +permutation "delete" "c1" "update1" "select2" "c2" +permutation "merge_delete" "c1" "update1" "select2" "c2" +permutation "delete" "c1" "merge2" "select2" "c2" +permutation "merge_delete" "c1" "merge2" "select2" "c2" + +# Now with concurrency +permutation "delete" "update1" "c1" "select2" "c2" +permutation "merge_delete" "update1" "c1" "select2" "c2" +permutation "delete" "merge2" "c1" "select2" "c2" +permutation "merge_delete" "merge2" "c1" "select2" "c2" diff --git a/src/test/isolation/specs/merge-insert-update.spec b/src/test/isolation/specs/merge-insert-update.spec new file mode 100644 index 0000000000..704492be1f --- /dev/null +++ b/src/test/isolation/specs/merge-insert-update.spec @@ -0,0 +1,52 @@ +# MERGE INSERT UPDATE +# +# This looks at how we handle concurrent INSERTs, illustrating how the +# behavior differs from INSERT ... ON CONFLICT + +setup +{ + CREATE TABLE target (key int primary key, val text); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge1" { MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; } +step "delete1" { DELETE FROM target WHERE key = 1; } +step "insert1" { INSERT INTO target VALUES (1, 'insert1'); } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge2" { MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; } + +step "merge2i" { MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; } + +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } +step "a2" { ABORT; } + +# Basic effects +permutation "merge1" "c1" "select2" "c2" +permutation "merge1" "c1" "merge2" "select2" "c2" + +# check concurrent inserts +permutation "insert1" "merge2" "c1" "select2" "c2" +permutation "merge1" "merge2" "c1" "select2" "c2" +permutation "merge1" "merge2" "a1" "select2" "c2" + +# check how we handle when visible row has been concurrently deleted, then same key re-inserted +permutation "delete1" "insert1" "c1" "merge2" "select2" "c2" +permutation "delete1" "insert1" "merge2" "c1" "select2" "c2" +permutation "delete1" "insert1" "merge2i" "c1" "select2" "c2" diff --git a/src/test/isolation/specs/merge-match-recheck.spec b/src/test/isolation/specs/merge-match-recheck.spec new file mode 100644 index 0000000000..193033da17 --- /dev/null +++ b/src/test/isolation/specs/merge-match-recheck.spec @@ -0,0 +1,79 @@ +# MERGE MATCHED RECHECK +# +# This test looks at what happens when we have complex +# WHEN MATCHED AND conditions and a concurrent UPDATE causes a +# recheck of the AND condition on the new row + +setup +{ + CREATE TABLE target (key int primary key, balance integer, status text, val text); + INSERT INTO target VALUES (1, 160, 's1', 'setup'); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge_status" +{ + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; +} + +step "merge_bal" +{ + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND balance < 100 THEN + UPDATE SET balance = balance * 2, val = t.val || ' when1' + WHEN MATCHED AND balance < 200 THEN + UPDATE SET balance = balance * 4, val = t.val || ' when2' + WHEN MATCHED AND balance < 300 THEN + UPDATE SET balance = balance * 8, val = t.val || ' when3'; +} + +step "select1" { SELECT * FROM target; } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "update1" { UPDATE target t SET balance = balance + 10, val = t.val || ' updated by update1' WHERE t.key = 1; } +step "update2" { UPDATE target t SET status = 's2', val = t.val || ' updated by update2' WHERE t.key = 1; } +step "update3" { UPDATE target t SET status = 's3', val = t.val || ' updated by update3' WHERE t.key = 1; } +step "update5" { UPDATE target t SET status = 's5', val = t.val || ' updated by update5' WHERE t.key = 1; } +step "update_bal1" { UPDATE target t SET balance = 50, val = t.val || ' updated by update_bal1' WHERE t.key = 1; } +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } + +# merge_status sees concurrently updated row and rechecks WHEN conditions, but recheck passes and final status = 's2' +permutation "update1" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final status = 's3' not 's2' +permutation "update2" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final status = 's4' not 's2' +permutation "update3" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, but we skip update and MERGE does nothing +permutation "update5" "merge_status" "c2" "select1" "c1" + +# merge_bal sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final balance = 100 not 640 +permutation "update_bal1" "merge_bal" "c2" "select1" "c1" diff --git a/src/test/isolation/specs/merge-update.spec b/src/test/isolation/specs/merge-update.spec new file mode 100644 index 0000000000..64e849966e --- /dev/null +++ b/src/test/isolation/specs/merge-update.spec @@ -0,0 +1,132 @@ +# MERGE UPDATE +# +# This test exercises atypical cases +# 1. UPDATEs of PKs that change the join in the ON clause +# 2. UPDATEs with WHEN AND conditions that would fail after concurrent update +# 3. UPDATEs with extra ON conditions that would fail after concurrent update + +setup +{ + CREATE TABLE target (key int primary key, val text); + INSERT INTO target VALUES (1, 'setup1'); + + CREATE TABLE pa_target (key integer, val text) + PARTITION BY LIST (key); + CREATE TABLE part1 (key integer, val text); + CREATE TABLE part2 (val text, key integer); + CREATE TABLE part3 (key integer, val text); + + ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); + ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); + ALTER TABLE pa_target ATTACH PARTITION part3 DEFAULT; + + INSERT INTO pa_target VALUES (1, 'initial'); + INSERT INTO pa_target VALUES (2, 'initial'); +} + +teardown +{ + DROP TABLE target; + DROP TABLE pa_target CASCADE; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge1" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "pa_merge1" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set val = t.val || ' updated by ' || s.val; +} +step "pa_merge2" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge2a" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "merge2b" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2b' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED AND t.key < 2 THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "merge2c" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2c' as val) s + ON s.key = t.key AND t.key < 2 + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "pa_merge2a" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "select2" { SELECT * FROM target; } +step "pa_select2" { SELECT * FROM pa_target; } +step "c2" { COMMIT; } + +# Basic effects +permutation "merge1" "c1" "select2" "c2" + +# One after the other, no concurrency +permutation "merge1" "c1" "merge2a" "select2" "c2" + +# Now with concurrency +permutation "merge1" "merge2a" "c1" "select2" "c2" +permutation "merge1" "merge2a" "a1" "select2" "c2" +permutation "merge1" "merge2b" "c1" "select2" "c2" +permutation "merge1" "merge2c" "c1" "select2" "c2" +permutation "pa_merge1" "pa_merge2a" "c1" "pa_select2" "c2" +permutation "pa_merge2" "pa_merge2a" "c1" "pa_select2" "c2" diff --git a/src/test/regress/expected/identity.out b/src/test/regress/expected/identity.out index d7d5178f5d..3a6016c80a 100644 --- a/src/test/regress/expected/identity.out +++ b/src/test/regress/expected/identity.out @@ -386,3 +386,58 @@ CREATE TABLE itest_child PARTITION OF itest_parent ( ) FOR VALUES FROM ('2016-07-01') TO ('2016-08-01'); -- error ERROR: identity columns are not supported on partitions DROP TABLE itest_parent; +-- MERGE tests +CREATE TABLE itest14 (a int GENERATED ALWAYS AS IDENTITY, b text); +CREATE TABLE itest15 (a int GENERATED BY DEFAULT AS IDENTITY, b text); +MERGE INTO itest14 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); +ERROR: cannot insert into column "a" +DETAIL: Column "a" is an identity column defined as GENERATED ALWAYS. +HINT: Use OVERRIDING SYSTEM VALUE to override. +MERGE INTO itest14 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); +ERROR: cannot insert into column "a" +DETAIL: Column "a" is an identity column defined as GENERATED ALWAYS. +HINT: Use OVERRIDING SYSTEM VALUE to override. +MERGE INTO itest14 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); +SELECT * FROM itest14; + a | b +----+------------------- + 30 | inserted by merge +(1 row) + +SELECT * FROM itest15; + a | b +----+------------------- + 10 | inserted by merge + 1 | inserted by merge + 30 | inserted by merge +(3 rows) + +DROP TABLE itest14; +DROP TABLE itest15; diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out new file mode 100644 index 0000000000..05c4287078 --- /dev/null +++ b/src/test/regress/expected/merge.out @@ -0,0 +1,1599 @@ +-- +-- MERGE +-- +--\set VERBOSITY verbose +--set debug_print_rewritten = true; +--set debug_print_parse = true; +--set debug_print_pretty = true; +CREATE USER merge_privs; +CREATE USER merge_no_privs; +DROP TABLE IF EXISTS target; +NOTICE: table "target" does not exist, skipping +DROP TABLE IF EXISTS source; +NOTICE: table "source" does not exist, skipping +CREATE TABLE target (tid integer, balance integer); +CREATE TABLE source (sid integer, delta integer); --no index +INSERT INTO target VALUES (1, 10); +INSERT INTO target VALUES (2, 20); +INSERT INTO target VALUES (3, 30); +SELECT t.ctid is not null as matched, t.*, s.* FROM source s FULL OUTER JOIN target t ON s.sid = t.tid ORDER BY t.tid, s.sid; + matched | tid | balance | sid | delta +---------+-----+---------+-----+------- + t | 1 | 10 | | + t | 2 | 20 | | + t | 3 | 30 | | +(3 rows) + +ALTER TABLE target OWNER TO merge_privs; +ALTER TABLE source OWNER TO merge_privs; +CREATE TABLE target2 (tid integer, balance integer); +CREATE TABLE source2 (sid integer, delta integer); +ALTER TABLE target2 OWNER TO merge_no_privs; +ALTER TABLE source2 OWNER TO merge_no_privs; +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_privs; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Merge Join + Merge Cond: (t_1.tid = s.sid) + -> Sort + Sort Key: t_1.tid + -> Seq Scan on target t_1 + -> Sort + Sort Key: s.sid + -> Seq Scan on source s +(9 rows) + +-- +-- Errors +-- +MERGE INTO target t RANDOMWORD +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: syntax error at or near "RANDOMWORD" +LINE 1: MERGE INTO target t RANDOMWORD + ^ +-- MATCHED/INSERT error +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: syntax error at or near "INSERT" +LINE 5: INSERT DEFAULT VALUES + ^ +-- incorrectly specifying INTO target +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT INTO target DEFAULT VALUES +; +ERROR: syntax error at or near "INTO" +LINE 5: INSERT INTO target DEFAULT VALUES + ^ +-- NOT MATCHED/UPDATE +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: syntax error at or near "UPDATE" +LINE 5: UPDATE SET balance = 0 + ^ +-- UPDATE tablename +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE target SET balance = 0 +; +ERROR: syntax error at or near "target" +LINE 5: UPDATE target SET balance = 0 + ^ +-- permissions +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: permission denied for table source2 +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_no_privs; +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: permission denied for table target +GRANT UPDATE ON target2 TO merge_privs; +SET SESSION AUTHORIZATION merge_privs; +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN MATCHED THEN + DELETE +; +ERROR: permission denied for table target2 +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: permission denied for table target2 +-- check if the target can be accessed from source relation subquery; we should +-- not be able to do so +MERGE INTO target t +USING (SELECT * FROM source WHERE t.tid > sid) s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 2: USING (SELECT * FROM source WHERE t.tid > sid) s + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +-- +-- initial tests +-- +-- zero rows in source has no effect +MERGE INTO target +USING source +ON target.tid = source.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ROLLBACK; +-- insert some non-matching source rows to work from +INSERT INTO source VALUES (4, 40); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 4 | 40 +(1 row) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + DO NOTHING +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + | +(4 rows) + +ROLLBACK; +-- index plans +INSERT INTO target SELECT generate_series(1000,2500), 0; +ALTER TABLE target ADD PRIMARY KEY (tid); +ANALYZE target; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL); + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Left Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +; +DELETE FROM target WHERE tid > 100; +ANALYZE target; +-- insert some matching source rows to work from +INSERT INTO source VALUES (2, 5); +INSERT INTO source VALUES (3, 20); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 3 | 20 + 4 | 40 +(3 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +-- equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 +(3 rows) + +ROLLBACK; +-- equivalent of a DELETE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 +(1 row) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | +(4 rows) + +ROLLBACK; +-- duplicate source row causes multiple target row update ERROR +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 2 | 5 + 3 | 20 + 4 | 40 +(4 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: MERGE command cannot affect row a second time +HINT: Ensure that not more than one source rows match any one target row +ROLLBACK; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +ERROR: MERGE command cannot affect row a second time +HINT: Ensure that not more than one source rows match any one target row +ROLLBACK; +-- correct source data +DELETE FROM source WHERE sid = 2; +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 3 | 20 + 4 | 40 +(3 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +-- remove constraints +alter table target drop CONSTRAINT target_pkey; +alter table target alter column tid drop not null; +-- multiple actions +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4) +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 + 4 | 4 +(4 rows) + +ROLLBACK; +-- should be equivalent +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4); +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 + 4 | 4 +(4 rows) + +ROLLBACK; +-- column references +-- do a simple equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 25 + 3 | 50 +(3 rows) + +ROLLBACK; +-- do a simple equivalent of an INSERT SELECT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- and again with explicitly identified column list +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- and again with a subtle error: referring to non-existent target row for NOT MATCHED +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 5: INSERT (tid, balance) VALUES (t.tid, s.delta) + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +-- and again with a constant ON clause +BEGIN; +MERGE INTO target t +USING source AS s +ON (SELECT true) +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 5: INSERT (tid, balance) VALUES (t.tid, s.delta) + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +SELECT * FROM target ORDER BY tid; +ERROR: current transaction is aborted, commands ignored until end of transaction block +ROLLBACK; +-- now the classic UPSERT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 25 + 3 | 50 + 4 | 40 +(4 rows) + +ROLLBACK; +-- unreachable WHEN clause should ERROR +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN /* Terminal WHEN clause for MATCHED */ + DELETE +WHEN MATCHED AND s.delta > 0 THEN + UPDATE SET balance = t.balance - s.delta +; +ERROR: unreachable WHEN clause specified after unconditional WHEN clause +ROLLBACK; +-- conditional WHEN clause +CREATE TABLE wq_target (tid integer not null, balance integer DEFAULT -1); +CREATE TABLE wq_source (balance integer, sid integer); +INSERT INTO wq_source (sid, balance) VALUES (1, 100); +BEGIN; +-- try a simple INSERT with default values first +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +ROLLBACK; +-- this time with a FALSE condition +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND FALSE THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- +(0 rows) + +-- this time with an actual condition which returns false +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance <> 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- +(0 rows) + +BEGIN; +-- and now with a condition which returns true +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +ROLLBACK; +-- conditions in the NOT MATCHED clause can only refer to source columns +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND t.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 3: WHEN NOT MATCHED AND t.balance = 100 THEN + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +SELECT * FROM wq_target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +ROLLBACK; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +-- conditions in MATCHED clause can refer to both source and target +SELECT * FROM wq_source; + balance | sid +---------+----- + 100 | 1 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +-- check if AND works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 199 +(1 row) + +-- check if OR works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 199 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 199 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 299 +(1 row) + +-- check if subqueries work in the conditions? +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance > (SELECT max(balance) FROM target) THEN + UPDATE SET balance = t.balance + s.balance; +-- check if we can access system columns in the conditions +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.xmin = t.xmax THEN + UPDATE SET balance = t.balance + s.balance; +ERROR: system column "xmin" reference in WHEN AND condition is invalid +LINE 3: WHEN MATCHED AND t.xmin = t.xmax THEN + ^ +ALTER TABLE wq_target SET WITH OIDS; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 399 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.oid >= 0 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 499 +(1 row) + +-- test preventing WHEN AND conditions from writing to the database +create or replace function merge_when_and_write() returns boolean +language plpgsql as +$$ +BEGIN + INSERT INTO target VALUES (100, 100); + RETURN TRUE; +END; +$$; +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND (merge_when_and_write()) THEN + UPDATE SET balance = t.balance + s.balance; +ROLLBACK; +drop function merge_when_and_write(); +DROP TABLE wq_target, wq_source; +-- test triggers +create or replace function merge_trigfunc () returns trigger +language plpgsql as +$$ +BEGIN + RAISE NOTICE '% % % trigger', TG_WHEN, TG_OP, TG_LEVEL; + IF (TG_WHEN = 'BEFORE' AND TG_LEVEL = 'ROW') THEN + IF (TG_OP = 'DELETE') THEN + RETURN OLD; + ELSE + RETURN NEW; + END IF; + ELSE + RETURN NULL; + END IF; +END; +$$; +CREATE TRIGGER merge_bsi BEFORE INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsu BEFORE UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsd BEFORE DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asi AFTER INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asu AFTER UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asd AFTER DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bri BEFORE INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bru BEFORE UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_brd BEFORE DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ari AFTER INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_aru AFTER UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ard AFTER DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +-- now the classic UPSERT, with a DELETE +BEGIN; +UPDATE target SET balance = 0 WHERE tid = 3; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE DELETE STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: BEFORE DELETE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER DELETE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER DELETE STATEMENT trigger +NOTICE: AFTER UPDATE STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger + QUERY PLAN +------------------------------------------------------------------ + Merge on target t (actual rows=0 loops=1) + Tuples Inserted: 1 + Tuples Updated: 1 + Tuples Deleted: 1 + -> Hash Left Join (actual rows=3 loops=1) + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s (actual rows=3 loops=1) + -> Hash (actual rows=3 loops=1) + Buckets: 1024 Batches: 1 Memory Usage: 9kB + -> Seq Scan on target t_1 (actual rows=3 loops=1) + Trigger merge_ard: calls=1 + Trigger merge_ari: calls=1 + Trigger merge_aru: calls=1 + Trigger merge_asd: calls=1 + Trigger merge_asi: calls=1 + Trigger merge_asu: calls=1 + Trigger merge_brd: calls=1 + Trigger merge_bri: calls=1 + Trigger merge_bru: calls=1 + Trigger merge_bsd: calls=1 + Trigger merge_bsi: calls=1 + Trigger merge_bsu: calls=1 +(22 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 15 + 4 | 40 +(3 rows) + +ROLLBACK; +-- test from PL/pgSQL +-- make sure MERGE INTO isn't interpreted to mean returning variables like SELECT INTO +BEGIN; +DO LANGUAGE plpgsql $$ +BEGIN +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +; +END; +$$; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +ROLLBACK; +--source constants +BEGIN; +MERGE INTO target t +USING (SELECT 9 AS sid, 57 AS delta) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 9 | 57 +(4 rows) + +ROLLBACK; +--source query +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta as newname FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.newname) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +--self-merge +BEGIN; +MERGE INTO target t1 +USING target t2 +ON t1.tid = t2.tid +WHEN MATCHED THEN + UPDATE SET balance = t1.balance + t2.balance +WHEN NOT MATCHED THEN + INSERT VALUES (t2.tid, t2.balance) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 20 + 2 | 40 + 3 | 60 +(3 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING (SELECT tid as sid, balance as delta FROM target WHERE balance > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING +(SELECT sid, max(delta) AS delta + FROM source + GROUP BY sid + HAVING count(*) = 1 + ORDER BY sid ASC) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- plpgsql parameters and results +BEGIN; +CREATE FUNCTION merge_func (p_id integer, p_bal integer) +RETURNS INTEGER +LANGUAGE plpgsql +AS $$ +DECLARE + result integer; +BEGIN +MERGE INTO target t +USING (SELECT p_id AS sid) AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance - p_bal +; +IF FOUND THEN + GET DIAGNOSTICS result := ROW_COUNT; +END IF; +RETURN result; +END; +$$; +SELECT merge_func(3, 4); +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger + merge_func +------------ + 1 +(1 row) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 26 +(3 rows) + +ROLLBACK; +-- PREPARE +BEGIN; +prepare foom as merge into target t using (select 1 as sid) s on (t.tid = s.sid) when matched then update set balance = 1; +execute foom; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 1 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +BEGIN; +PREPARE foom2 (integer, integer) AS +MERGE INTO target t +USING (SELECT 1) s +ON t.tid = $1 +WHEN MATCHED THEN +UPDATE SET balance = $2; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +execute foom2 (1, 1); +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger + QUERY PLAN +------------------------------------------------------ + Merge on target t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 1 + Tuples Deleted: 0 + -> Seq Scan on target t_1 (actual rows=1 loops=1) + Filter: (tid = 1) + Rows Removed by Filter: 2 + Trigger merge_aru: calls=1 + Trigger merge_asu: calls=1 + Trigger merge_bru: calls=1 + Trigger merge_bsu: calls=1 +(11 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 1 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +-- subqueries in source relation +CREATE TABLE sq_target (tid integer NOT NULL, balance integer); +CREATE TABLE sq_source (delta integer, sid integer, balance integer DEFAULT 0); +INSERT INTO sq_target(tid, balance) VALUES (1,100), (2,200), (3,300); +INSERT INTO sq_source(sid, delta) VALUES (1,10), (2,20), (4,40); +BEGIN; +MERGE INTO sq_target t +USING (SELECT * FROM sq_source) s +ON tid = sid +WHEN MATCHED AND t.balance > delta THEN + UPDATE SET balance = t.balance + delta; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 3 | 300 + 1 | 110 + 2 | 220 +(3 rows) + +ROLLBACK; +-- try a view +CREATE VIEW v AS SELECT * FROM sq_source WHERE sid < 2; +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = v.balance + delta; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 2 | 200 + 3 | 300 + 1 | 10 +(3 rows) + +ROLLBACK; +-- ambiguous reference to a column +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +ERROR: column reference "balance" is ambiguous +LINE 5: UPDATE SET balance = balance + delta + ^ +ROLLBACK; +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 2 | 200 + 3 | 300 + -1 | -11 +(3 rows) + +ROLLBACK; +-- CTEs +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +WITH targq AS ( + SELECT * FROM v +) +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +; +ERROR: syntax error at or near "MERGE" +LINE 4: MERGE INTO sq_target t + ^ +ROLLBACK; +-- RETURNING +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +RETURNING * +; +ERROR: syntax error at or near "RETURNING" +LINE 10: RETURNING * + ^ +ROLLBACK; +-- Subqueries +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = (SELECT count(*) FROM sq_target) +; +ROLLBACK; +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND (SELECT count(*) > 0 FROM sq_target) THEN + UPDATE SET balance = 42 +; +ROLLBACK; +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid AND (SELECT count(*) > 0 FROM sq_target) +WHEN MATCHED THEN + UPDATE SET balance = 42 +; +ROLLBACK; +DROP TABLE sq_target, sq_source CASCADE; +NOTICE: drop cascades to view v +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 PARTITION OF pa_target FOR VALUES IN (1,4); +CREATE TABLE part2 PARTITION OF pa_target FOR VALUES IN (2,5,6); +CREATE TABLE part3 PARTITION OF pa_target FOR VALUES IN (3,8,9); +CREATE TABLE part4 PARTITION OF pa_target DEFAULT; +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 330 | initial updated by merge + 4 | 40 | inserted by merge + 5 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 7 | 770 | initial updated by merge + 8 | 80 | inserted by merge + 9 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 11 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 13 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 30 | inserted by merge + 3 | 300 | initial + 4 | 40 | inserted by merge + 5 | 500 | initial + 5 | 50 | inserted by merge + 6 | 60 | inserted by merge + 7 | 700 | initial + 7 | 70 | inserted by merge + 8 | 80 | inserted by merge + 9 | 90 | inserted by merge + 9 | 900 | initial + 10 | 100 | inserted by merge + 11 | 1100 | initial + 11 | 110 | inserted by merge + 12 | 120 | inserted by merge + 13 | 1300 | initial + 13 | 130 | inserted by merge + 14 | 140 | inserted by merge +(20 rows) + +ROLLBACK; +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 2 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 4 | 40 | inserted by merge + 4 | 330 | initial updated by merge + 6 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 8 | 80 | inserted by merge + 8 | 770 | initial updated by merge + 10 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 12 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 14 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +DROP TABLE pa_target CASCADE; +-- The target table is partitioned in the same way, but this time by attaching +-- partitions which have columns in different order, dropped columns etc. +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 (tid integer, balance float, val text); +CREATE TABLE part2 (balance float, tid integer, val text); +CREATE TABLE part3 (tid integer, balance float, val text); +CREATE TABLE part4 (extraid text, tid integer, balance float, val text); +ALTER TABLE part4 DROP COLUMN extraid; +ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); +ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); +ALTER TABLE pa_target ATTACH PARTITION part3 FOR VALUES IN (3,8,9); +ALTER TABLE pa_target ATTACH PARTITION part4 DEFAULT; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 330 | initial updated by merge + 4 | 40 | inserted by merge + 5 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 7 | 770 | initial updated by merge + 8 | 80 | inserted by merge + 9 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 11 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 13 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 30 | inserted by merge + 3 | 300 | initial + 4 | 40 | inserted by merge + 5 | 500 | initial + 5 | 50 | inserted by merge + 6 | 60 | inserted by merge + 7 | 700 | initial + 7 | 70 | inserted by merge + 8 | 80 | inserted by merge + 9 | 90 | inserted by merge + 9 | 900 | initial + 10 | 100 | inserted by merge + 11 | 1100 | initial + 11 | 110 | inserted by merge + 12 | 120 | inserted by merge + 13 | 1300 | initial + 13 | 130 | inserted by merge + 14 | 140 | inserted by merge +(20 rows) + +ROLLBACK; +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 2 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 4 | 40 | inserted by merge + 4 | 330 | initial updated by merge + 6 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 8 | 80 | inserted by merge + 8 | 770 | initial updated by merge + 10 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 12 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 14 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; +-- Sub-partitionin +CREATE TABLE pa_target (logts timestamp, tid integer, balance float, val text) + PARTITION BY RANGE (logts); +CREATE TABLE part_m01 PARTITION OF pa_target + FOR VALUES FROM ('2017-01-01') TO ('2017-02-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m01_odd PARTITION OF part_m01 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m01_even PARTITION OF part_m01 + FOR VALUES IN (2,4,6,8); +CREATE TABLE part_m02 PARTITION OF pa_target + FOR VALUES FROM ('2017-02-01') TO ('2017-03-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m02_odd PARTITION OF part_m02 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m02_even PARTITION OF part_m02 + FOR VALUES IN (2,4,6,8); +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT '2017-01-31', id, id * 100, 'initial' FROM generate_series(1,9,3) AS id; +INSERT INTO pa_target SELECT '2017-02-28', id, id * 100, 'initial' FROM generate_series(2,9,3) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING (SELECT '2017-01-15' AS slogts, * FROM pa_source WHERE sid < 10) s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + logts | tid | balance | val +--------------------------+-----+---------+-------------------------- + Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge + Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge + Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge +(9 rows) + +ROLLBACK; +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; +-- some complex joins on the source side +CREATE TABLE cj_target (tid integer, balance float, val text); +CREATE TABLE cj_source1 (sid1 integer, scat integer, delta integer); +CREATE TABLE cj_source2 (sid2 integer, sval text); +INSERT INTO cj_source1 VALUES (1, 10, 100); +INSERT INTO cj_source1 VALUES (1, 20, 200); +INSERT INTO cj_source1 VALUES (2, 20, 300); +INSERT INTO cj_source1 VALUES (3, 10, 400); +INSERT INTO cj_source2 VALUES (1, 'initial source2'); +INSERT INTO cj_source2 VALUES (2, 'initial source2'); +INSERT INTO cj_source2 VALUES (3, 'initial source2'); +-- source relation is an unalised join +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid1, delta, sval); +-- try accessing columns from either side of the source join +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta, sval) +WHEN MATCHED THEN + DELETE; +-- some simple expressions in INSERT targetlist +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta + scat, sval) +WHEN MATCHED THEN + UPDATE SET val = val || ' updated by merge'; +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN MATCHED THEN + UPDATE SET val = val || ' ' || delta::text; +SELECT * FROM cj_target; + tid | balance | val +-----+---------+---------------------------------- + 3 | 400 | initial source2 updated by merge + 1 | 220 | initial source2 200 + 1 | 110 | initial source2 200 + 2 | 320 | initial source2 300 +(4 rows) + +ALTER TABLE cj_source1 RENAME COLUMN sid1 TO sid; +ALTER TABLE cj_source2 RENAME COLUMN sid2 TO sid; +TRUNCATE cj_target; +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON s1.sid = s2.sid +ON t.tid = s1.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s2.sid, delta, sval); +DROP TABLE cj_source2, cj_source1, cj_target; +-- Function scans +CREATE TABLE fs_target (a int, b int, c text); +MERGE INTO fs_target t +USING generate_series(1,100,1) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1); +MERGE INTO fs_target t +USING generate_series(1,100,2) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id, c = 'updated '|| id.*::text +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1, 'inserted ' || id.*::text); +SELECT count(*) FROM fs_target; + count +------- + 100 +(1 row) + +DROP TABLE fs_target; +-- SERIALIZABLE test +-- handled in isolation tests +-- prepare +RESET SESSION AUTHORIZATION; +DROP TABLE target, target2; +DROP TABLE source, source2; +DROP FUNCTION merge_trigfunc(); +DROP USER merge_privs; +DROP USER merge_no_privs; diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index ac8968d24f..864f2c1345 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -517,6 +517,104 @@ SELECT atest6 FROM atest6; -- ok (0 rows) COPY atest6 TO stdout; -- ok +-- test column privileges with MERGE +SET SESSION AUTHORIZATION regress_priv_user1; +CREATE TABLE mtarget (a int, b text); +CREATE TABLE msource (a int, b text); +INSERT INTO mtarget VALUES (1, 'init1'), (2, 'init2'); +INSERT INTO msource VALUES (1, 'source1'), (2, 'source2'), (3, 'source3'); +GRANT SELECT (a) ON msource TO regress_priv_user4; +GRANT SELECT (a) ON mtarget TO regress_priv_user4; +GRANT INSERT (a,b) ON mtarget TO regress_priv_user4; +GRANT UPDATE (b) ON mtarget TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; +-- +-- test source privileges +-- +-- fail (no SELECT priv on s.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table msource +-- fail (s.b used in the INSERTed values) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table msource +-- fail (s.b used in the WHEN quals) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND s.b = 'x' THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table msource +-- this should be ok since only s.a is accessed +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'ok' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ROLLBACK; +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT SELECT (b) ON msource TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; +-- should now be ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ROLLBACK; +-- +-- test target privileges +-- +-- fail (no SELECT priv on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table mtarget +-- fail (no UPDATE on t.a) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b, a = t.a + 1 +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table mtarget +-- fail (no SELECT on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table mtarget +-- ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b; +ROLLBACK; +-- fail (no DELETE) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ERROR: permission denied for table mtarget +-- grant delete privileges +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT DELETE ON mtarget TO regress_priv_user4; +-- should be ok now +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ROLLBACK; -- check error reporting with column privs SET SESSION AUTHORIZATION regress_priv_user1; CREATE TABLE t1 (c1 int, c2 int, c3 int check (c3 < 5), primary key (c1, c2)); diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index f1ae40df61..bf7af3ba82 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -2138,6 +2138,188 @@ ERROR: new row violates row-level security policy (USING expression) for table INSERT INTO document VALUES (1, (SELECT cid from category WHERE cname = 'novel'), 1, 'regress_rls_bob', 'my first novel') ON CONFLICT (did) DO UPDATE SET dauthor = 'regress_rls_carol'; ERROR: new row violates row-level security policy for table "document" +-- +-- MERGE +-- +RESET SESSION AUTHORIZATION; +DROP POLICY p3_with_all ON document; +ALTER TABLE document ADD COLUMN dnotes text DEFAULT ''; +-- all documents are readable +CREATE POLICY p1 ON document FOR SELECT USING (true); +-- one may insert documents only authored by them +CREATE POLICY p2 ON document FOR INSERT WITH CHECK (dauthor = current_user); +-- one may only update documents in 'novel' category +CREATE POLICY p3 ON document FOR UPDATE + USING (cid = (SELECT cid from category WHERE cname = 'novel')) + WITH CHECK (dauthor = current_user); +-- one may only delete documents in 'manga' category +CREATE POLICY p4 ON document FOR DELETE + USING (cid = (SELECT cid from category WHERE cname = 'manga')); +SELECT * FROM document; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-------------------+----------------------------------+-------- + 1 | 11 | 1 | regress_rls_bob | my first novel | + 3 | 22 | 2 | regress_rls_bob | my science fiction | + 4 | 44 | 1 | regress_rls_bob | my first manga | + 5 | 44 | 2 | regress_rls_bob | my second manga | + 6 | 22 | 1 | regress_rls_carol | great science fiction | + 7 | 33 | 2 | regress_rls_carol | great technology book | + 8 | 44 | 1 | regress_rls_carol | great manga | + 9 | 22 | 1 | regress_rls_dave | awesome science fiction | + 10 | 33 | 2 | regress_rls_dave | awesome technology book | + 11 | 33 | 1 | regress_rls_carol | hoge | + 33 | 22 | 1 | regress_rls_bob | okay science fiction | + 2 | 11 | 2 | regress_rls_bob | my first novel | + 78 | 33 | 1 | regress_rls_bob | some technology novel | + 79 | 33 | 1 | regress_rls_bob | technology book, can only insert | +(14 rows) + +SET SESSION AUTHORIZATION regress_rls_bob; +-- Fails, since update violates WITH CHECK qual on dauthor +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge1 ', dauthor = 'regress_rls_alice'; +ERROR: new row violates row-level security policy for table "document" +-- Should be OK since USING and WITH CHECK quals pass +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge2 '; +-- Even when dauthor is updated explicitly, but to the existing value +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge3 ', dauthor = 'regress_rls_bob'; +-- There is a MATCH for did = 3, but UPDATE's USING qual does not allow +-- updating an item in category 'science fiction' +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge '; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- The same thing with DELETE action, but fails again because no permissions +-- to delete items in 'science fiction' category that did 3 belongs to. +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- Document with did 4 belongs to 'manga' category which is allowed for +-- deletion. But this fails because the UPDATE action is matched first and +-- UPDATE policy does not allow updation in the category. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes = '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- UPDATE action is not matched this time because of the WHEN AND qual. +-- DELETE still fails because role regress_rls_bob does not have SELECT +-- privileges on 'manga' category row in the category table. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +SELECT * FROM document WHERE did = 4; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-----------------+----------------+-------- + 4 | 44 | 1 | regress_rls_bob | my first manga | +(1 row) + +-- Switch to regress_rls_carol role and try the DELETE again. It should succeed +-- this time +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_carol; +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +-- Switch back to regress_rls_bob role +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_bob; +-- Try INSERT action. This fails because we are trying to insert +-- dauthor = regress_rls_dave and INSERT's WITH CHECK does not allow +-- that +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_dave', 'another novel'); +ERROR: new row violates row-level security policy for table "document" +-- This should be fine +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +-- ok +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge4 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +-- drop and create a new SELECT policy which prevents us from reading +-- any document except with category 'magna' +RESET SESSION AUTHORIZATION; +DROP POLICY p1 ON document; +CREATE POLICY p1 ON document FOR SELECT + USING (cid = (SELECT cid from category WHERE cname = 'manga')); +SET SESSION AUTHORIZATION regress_rls_bob; +-- MERGE can no longer see the matching row and hence attempts the +-- NOT MATCHED action, which results in unique key violation +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge5 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +ERROR: duplicate key value violates unique constraint "document_pkey" +RESET SESSION AUTHORIZATION; +-- drop the restrictive SELECT policy so that we can look at the +-- final state of the table +DROP POLICY p1 ON document; +-- Just check everything went per plan +SELECT * FROM document; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-------------------+----------------------------------+----------------------------------------------------------------------- + 3 | 22 | 2 | regress_rls_bob | my science fiction | + 5 | 44 | 2 | regress_rls_bob | my second manga | + 6 | 22 | 1 | regress_rls_carol | great science fiction | + 7 | 33 | 2 | regress_rls_carol | great technology book | + 8 | 44 | 1 | regress_rls_carol | great manga | + 9 | 22 | 1 | regress_rls_dave | awesome science fiction | + 10 | 33 | 2 | regress_rls_dave | awesome technology book | + 11 | 33 | 1 | regress_rls_carol | hoge | + 33 | 22 | 1 | regress_rls_bob | okay science fiction | + 2 | 11 | 2 | regress_rls_bob | my first novel | + 78 | 33 | 1 | regress_rls_bob | some technology novel | + 79 | 33 | 1 | regress_rls_bob | technology book, can only insert | + 12 | 11 | 1 | regress_rls_bob | another novel | + 1 | 11 | 1 | regress_rls_bob | my first novel | notes added by merge2 notes added by merge3 notes added by merge4 +(14 rows) + -- -- ROLE/GROUP -- diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 5149b72fe9..d369a73173 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -3263,6 +3263,37 @@ CREATE RULE rules_parted_table_insert AS ON INSERT to rules_parted_table ALTER RULE rules_parted_table_insert ON rules_parted_table RENAME TO rules_parted_table_insert_redirect; DROP TABLE rules_parted_table; -- +-- test MERGE +-- +CREATE TABLE rule_merge1 (a int, b text); +CREATE TABLE rule_merge2 (a int, b text); +CREATE RULE rule1 AS ON INSERT TO rule_merge1 + DO INSTEAD INSERT INTO rule_merge2 VALUES (NEW.*); +CREATE RULE rule2 AS ON UPDATE TO rule_merge1 + DO INSTEAD UPDATE rule_merge2 SET a = NEW.a, b = NEW.b + WHERE a = OLD.a; +CREATE RULE rule3 AS ON DELETE TO rule_merge1 + DO INSTEAD DELETE FROM rule_merge2 WHERE a = OLD.a; +-- MERGE not supported for table with rules +MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); +ERROR: MERGE is not supported for relations with rules +-- should be ok with the other table though +MERGE INTO rule_merge2 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); +-- -- Test enabling/disabling -- CREATE TABLE ruletest1 (a int); diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out index f534d0db18..a336a1626c 100644 --- a/src/test/regress/expected/triggers.out +++ b/src/test/regress/expected/triggers.out @@ -2760,6 +2760,54 @@ delete from self_ref where a = 1; NOTICE: trigger_func(self_ref) called: action = DELETE, when = BEFORE, level = STATEMENT NOTICE: trigger = self_ref_s_trig, old table = (1,), (2,1), (3,2), (4,3) drop table self_ref; +-- +-- test transition tables with MERGE +-- +create table merge_target_table (a int primary key, b text); +create trigger merge_target_table_insert_trig + after insert on merge_target_table referencing new table as new_table + for each statement execute procedure dump_insert(); +create trigger merge_target_table_update_trig + after update on merge_target_table referencing old table as old_table new table as new_table + for each statement execute procedure dump_update(); +create trigger merge_target_table_delete_trig + after delete on merge_target_table referencing old table as old_table + for each statement execute procedure dump_delete(); +create table merge_source_table (a int, b text); +insert into merge_source_table + values (1, 'initial1'), (2, 'initial2'), + (3, 'initial3'), (4, 'initial4'); +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_insert_trig, new table = (1,initial1), (2,initial2), (3,initial3), (4,initial4) +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_delete_trig, old table = (3,initial3), (4,initial4) +NOTICE: trigger = merge_target_table_update_trig, old table = (1,initial1), (2,initial2), new table = (1,"initial1 updated by merge"), (2,"initial2 updated by merge") +NOTICE: trigger = merge_target_table_insert_trig, new table = <NULL> +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated again by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_delete_trig, old table = <NULL> +NOTICE: trigger = merge_target_table_update_trig, old table = (1,"initial1 updated by merge"), (2,"initial2 updated by merge"), new table = (1,"initial1 updated by merge updated again by merge"), (2,"initial2 updated by merge updated again by merge") +NOTICE: trigger = merge_target_table_insert_trig, new table = (3,initial3), (4,initial4) +drop table merge_source_table, merge_target_table; -- cleanup drop function dump_insert(); drop function dump_update(); diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index d308a05117..bfb9f11156 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -84,7 +84,7 @@ test: select_into select_distinct select_distinct_on select_implicit select_havi # ---------- # Another group of parallel tests # ---------- -test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password +test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password merge # ---------- # Another group of parallel tests diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 45147e9328..900814d33c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -122,6 +122,7 @@ test: tablesample test: groupingsets test: drop_operator test: password +test: merge test: alter_generic test: alter_operator test: misc diff --git a/src/test/regress/sql/identity.sql b/src/test/regress/sql/identity.sql index a35f331f4e..f8f34eaf18 100644 --- a/src/test/regress/sql/identity.sql +++ b/src/test/regress/sql/identity.sql @@ -246,3 +246,48 @@ CREATE TABLE itest_child PARTITION OF itest_parent ( f3 WITH OPTIONS GENERATED ALWAYS AS IDENTITY ) FOR VALUES FROM ('2016-07-01') TO ('2016-08-01'); -- error DROP TABLE itest_parent; + +-- MERGE tests +CREATE TABLE itest14 (a int GENERATED ALWAYS AS IDENTITY, b text); +CREATE TABLE itest15 (a int GENERATED BY DEFAULT AS IDENTITY, b text); + +MERGE INTO itest14 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); + +MERGE INTO itest14 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest14 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); + +SELECT * FROM itest14; +SELECT * FROM itest15; +DROP TABLE itest14; +DROP TABLE itest15; diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql new file mode 100644 index 0000000000..8b5244fc63 --- /dev/null +++ b/src/test/regress/sql/merge.sql @@ -0,0 +1,1068 @@ +-- +-- MERGE +-- +--\set VERBOSITY verbose + +--set debug_print_rewritten = true; +--set debug_print_parse = true; +--set debug_print_pretty = true; + + +CREATE USER merge_privs; +CREATE USER merge_no_privs; +DROP TABLE IF EXISTS target; +DROP TABLE IF EXISTS source; +CREATE TABLE target (tid integer, balance integer); +CREATE TABLE source (sid integer, delta integer); --no index +INSERT INTO target VALUES (1, 10); +INSERT INTO target VALUES (2, 20); +INSERT INTO target VALUES (3, 30); +SELECT t.ctid is not null as matched, t.*, s.* FROM source s FULL OUTER JOIN target t ON s.sid = t.tid ORDER BY t.tid, s.sid; + +ALTER TABLE target OWNER TO merge_privs; +ALTER TABLE source OWNER TO merge_privs; + +CREATE TABLE target2 (tid integer, balance integer); +CREATE TABLE source2 (sid integer, delta integer); + +ALTER TABLE target2 OWNER TO merge_no_privs; +ALTER TABLE source2 OWNER TO merge_no_privs; + +GRANT INSERT ON target TO merge_no_privs; + +SET SESSION AUTHORIZATION merge_privs; + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + +-- +-- Errors +-- +MERGE INTO target t RANDOMWORD +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +-- MATCHED/INSERT error +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + INSERT DEFAULT VALUES +; +-- incorrectly specifying INTO target +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT INTO target DEFAULT VALUES +; +-- NOT MATCHED/UPDATE +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + UPDATE SET balance = 0 +; +-- UPDATE tablename +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE target SET balance = 0 +; + +-- permissions + +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_no_privs; + +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +GRANT UPDATE ON target2 TO merge_privs; +SET SESSION AUTHORIZATION merge_privs; + +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN MATCHED THEN + DELETE +; + +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; + +-- check if the target can be accessed from source relation subquery; we should +-- not be able to do so +MERGE INTO target t +USING (SELECT * FROM source WHERE t.tid > sid) s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; + +-- +-- initial tests +-- +-- zero rows in source has no effect +MERGE INTO target +USING source +ON target.tid = source.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ROLLBACK; + +-- insert some non-matching source rows to work from +INSERT INTO source VALUES (4, 40); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + DO NOTHING +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- index plans +INSERT INTO target SELECT generate_series(1000,2500), 0; +ALTER TABLE target ADD PRIMARY KEY (tid); +ANALYZE target; + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL); +; +DELETE FROM target WHERE tid > 100; +ANALYZE target; + +-- insert some matching source rows to work from +INSERT INTO source VALUES (2, 5); +INSERT INTO source VALUES (3, 20); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +-- equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- equivalent of a DELETE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- duplicate source row causes multiple target row update ERROR +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +ROLLBACK; + +-- correct source data +DELETE FROM source WHERE sid = 2; +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +-- remove constraints +alter table target drop CONSTRAINT target_pkey; +alter table target alter column tid drop not null; + +-- multiple actions +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4) +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- should be equivalent +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4); +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- column references +-- do a simple equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- do a simple equivalent of an INSERT SELECT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- and again with explicitly identified column list +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- and again with a subtle error: referring to non-existent target row for NOT MATCHED +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; + +-- and again with a constant ON clause +BEGIN; +MERGE INTO target t +USING source AS s +ON (SELECT true) +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- now the classic UPSERT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- unreachable WHEN clause should ERROR +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN /* Terminal WHEN clause for MATCHED */ + DELETE +WHEN MATCHED AND s.delta > 0 THEN + UPDATE SET balance = t.balance - s.delta +; +ROLLBACK; + +-- conditional WHEN clause +CREATE TABLE wq_target (tid integer not null, balance integer DEFAULT -1); +CREATE TABLE wq_source (balance integer, sid integer); + +INSERT INTO wq_source (sid, balance) VALUES (1, 100); + +BEGIN; +-- try a simple INSERT with default values first +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +-- this time with a FALSE condition +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND FALSE THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +-- this time with an actual condition which returns false +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance <> 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +BEGIN; +-- and now with a condition which returns true +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +-- conditions in the NOT MATCHED clause can only refer to source columns +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND t.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +-- conditions in MATCHED clause can refer to both source and target +SELECT * FROM wq_source; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if AND works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if OR works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 199 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if subqueries work in the conditions? +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance > (SELECT max(balance) FROM target) THEN + UPDATE SET balance = t.balance + s.balance; + +-- check if we can access system columns in the conditions +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.xmin = t.xmax THEN + UPDATE SET balance = t.balance + s.balance; + +ALTER TABLE wq_target SET WITH OIDS; +SELECT * FROM wq_target; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.oid >= 0 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- test preventing WHEN AND conditions from writing to the database +create or replace function merge_when_and_write() returns boolean +language plpgsql as +$$ +BEGIN + INSERT INTO target VALUES (100, 100); + RETURN TRUE; +END; +$$; + +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND (merge_when_and_write()) THEN + UPDATE SET balance = t.balance + s.balance; +ROLLBACK; +drop function merge_when_and_write(); + +DROP TABLE wq_target, wq_source; + +-- test triggers +create or replace function merge_trigfunc () returns trigger +language plpgsql as +$$ +BEGIN + RAISE NOTICE '% % % trigger', TG_WHEN, TG_OP, TG_LEVEL; + IF (TG_WHEN = 'BEFORE' AND TG_LEVEL = 'ROW') THEN + IF (TG_OP = 'DELETE') THEN + RETURN OLD; + ELSE + RETURN NEW; + END IF; + ELSE + RETURN NULL; + END IF; +END; +$$; +CREATE TRIGGER merge_bsi BEFORE INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsu BEFORE UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsd BEFORE DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asi AFTER INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asu AFTER UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asd AFTER DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bri BEFORE INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bru BEFORE UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_brd BEFORE DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ari AFTER INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_aru AFTER UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ard AFTER DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); + +-- now the classic UPSERT, with a DELETE +BEGIN; +UPDATE target SET balance = 0 WHERE tid = 3; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- test from PL/pgSQL +-- make sure MERGE INTO isn't interpreted to mean returning variables like SELECT INTO +BEGIN; +DO LANGUAGE plpgsql $$ +BEGIN +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +; +END; +$$; +ROLLBACK; + +--source constants +BEGIN; +MERGE INTO target t +USING (SELECT 9 AS sid, 57 AS delta) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +--source query +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta as newname FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.newname) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +--self-merge +BEGIN; +MERGE INTO target t1 +USING target t2 +ON t1.tid = t2.tid +WHEN MATCHED THEN + UPDATE SET balance = t1.balance + t2.balance +WHEN NOT MATCHED THEN + INSERT VALUES (t2.tid, t2.balance) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING (SELECT tid as sid, balance as delta FROM target WHERE balance > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING +(SELECT sid, max(delta) AS delta + FROM source + GROUP BY sid + HAVING count(*) = 1 + ORDER BY sid ASC) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- plpgsql parameters and results +BEGIN; +CREATE FUNCTION merge_func (p_id integer, p_bal integer) +RETURNS INTEGER +LANGUAGE plpgsql +AS $$ +DECLARE + result integer; +BEGIN +MERGE INTO target t +USING (SELECT p_id AS sid) AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance - p_bal +; +IF FOUND THEN + GET DIAGNOSTICS result := ROW_COUNT; +END IF; +RETURN result; +END; +$$; +SELECT merge_func(3, 4); +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- PREPARE +BEGIN; +prepare foom as merge into target t using (select 1 as sid) s on (t.tid = s.sid) when matched then update set balance = 1; +execute foom; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +PREPARE foom2 (integer, integer) AS +MERGE INTO target t +USING (SELECT 1) s +ON t.tid = $1 +WHEN MATCHED THEN +UPDATE SET balance = $2; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +execute foom2 (1, 1); +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- subqueries in source relation + +CREATE TABLE sq_target (tid integer NOT NULL, balance integer); +CREATE TABLE sq_source (delta integer, sid integer, balance integer DEFAULT 0); + +INSERT INTO sq_target(tid, balance) VALUES (1,100), (2,200), (3,300); +INSERT INTO sq_source(sid, delta) VALUES (1,10), (2,20), (4,40); + +BEGIN; +MERGE INTO sq_target t +USING (SELECT * FROM sq_source) s +ON tid = sid +WHEN MATCHED AND t.balance > delta THEN + UPDATE SET balance = t.balance + delta; +SELECT * FROM sq_target; +ROLLBACK; + +-- try a view +CREATE VIEW v AS SELECT * FROM sq_source WHERE sid < 2; + +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = v.balance + delta; +SELECT * FROM sq_target; +ROLLBACK; + +-- ambiguous reference to a column +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +ROLLBACK; + +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +SELECT * FROM sq_target; +ROLLBACK; + +-- CTEs +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +WITH targq AS ( + SELECT * FROM v +) +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +; +ROLLBACK; + +-- RETURNING +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +RETURNING * +; +ROLLBACK; + +-- Subqueries +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = (SELECT count(*) FROM sq_target) +; +ROLLBACK; + +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND (SELECT count(*) > 0 FROM sq_target) THEN + UPDATE SET balance = 42 +; +ROLLBACK; + +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid AND (SELECT count(*) > 0 FROM sq_target) +WHEN MATCHED THEN + UPDATE SET balance = 42 +; +ROLLBACK; + +DROP TABLE sq_target, sq_source CASCADE; + +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); + +CREATE TABLE part1 PARTITION OF pa_target FOR VALUES IN (1,4); +CREATE TABLE part2 PARTITION OF pa_target FOR VALUES IN (2,5,6); +CREATE TABLE part3 PARTITION OF pa_target FOR VALUES IN (3,8,9); +CREATE TABLE part4 PARTITION OF pa_target DEFAULT; + +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_target CASCADE; + +-- The target table is partitioned in the same way, but this time by attaching +-- partitions which have columns in different order, dropped columns etc. +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 (tid integer, balance float, val text); +CREATE TABLE part2 (balance float, tid integer, val text); +CREATE TABLE part3 (tid integer, balance float, val text); +CREATE TABLE part4 (extraid text, tid integer, balance float, val text); +ALTER TABLE part4 DROP COLUMN extraid; + +ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); +ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); +ALTER TABLE pa_target ATTACH PARTITION part3 FOR VALUES IN (3,8,9); +ALTER TABLE pa_target ATTACH PARTITION part4 DEFAULT; + +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; + +-- Sub-partitionin +CREATE TABLE pa_target (logts timestamp, tid integer, balance float, val text) + PARTITION BY RANGE (logts); + +CREATE TABLE part_m01 PARTITION OF pa_target + FOR VALUES FROM ('2017-01-01') TO ('2017-02-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m01_odd PARTITION OF part_m01 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m01_even PARTITION OF part_m01 + FOR VALUES IN (2,4,6,8); +CREATE TABLE part_m02 PARTITION OF pa_target + FOR VALUES FROM ('2017-02-01') TO ('2017-03-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m02_odd PARTITION OF part_m02 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m02_even PARTITION OF part_m02 + FOR VALUES IN (2,4,6,8); + +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT '2017-01-31', id, id * 100, 'initial' FROM generate_series(1,9,3) AS id; +INSERT INTO pa_target SELECT '2017-02-28', id, id * 100, 'initial' FROM generate_series(2,9,3) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING (SELECT '2017-01-15' AS slogts, * FROM pa_source WHERE sid < 10) s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; + +-- some complex joins on the source side + +CREATE TABLE cj_target (tid integer, balance float, val text); +CREATE TABLE cj_source1 (sid1 integer, scat integer, delta integer); +CREATE TABLE cj_source2 (sid2 integer, sval text); +INSERT INTO cj_source1 VALUES (1, 10, 100); +INSERT INTO cj_source1 VALUES (1, 20, 200); +INSERT INTO cj_source1 VALUES (2, 20, 300); +INSERT INTO cj_source1 VALUES (3, 10, 400); +INSERT INTO cj_source2 VALUES (1, 'initial source2'); +INSERT INTO cj_source2 VALUES (2, 'initial source2'); +INSERT INTO cj_source2 VALUES (3, 'initial source2'); + +-- source relation is an unalised join +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid1, delta, sval); + +-- try accessing columns from either side of the source join +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta, sval) +WHEN MATCHED THEN + DELETE; + +-- some simple expressions in INSERT targetlist +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta + scat, sval) +WHEN MATCHED THEN + UPDATE SET val = val || ' updated by merge'; + +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN MATCHED THEN + UPDATE SET val = val || ' ' || delta::text; + +SELECT * FROM cj_target; + +ALTER TABLE cj_source1 RENAME COLUMN sid1 TO sid; +ALTER TABLE cj_source2 RENAME COLUMN sid2 TO sid; + +TRUNCATE cj_target; + +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON s1.sid = s2.sid +ON t.tid = s1.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s2.sid, delta, sval); + +DROP TABLE cj_source2, cj_source1, cj_target; + +-- Function scans +CREATE TABLE fs_target (a int, b int, c text); +MERGE INTO fs_target t +USING generate_series(1,100,1) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1); + +MERGE INTO fs_target t +USING generate_series(1,100,2) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id, c = 'updated '|| id.*::text +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1, 'inserted ' || id.*::text); + +SELECT count(*) FROM fs_target; +DROP TABLE fs_target; + +-- SERIALIZABLE test +-- handled in isolation tests + +-- prepare + +RESET SESSION AUTHORIZATION; +DROP TABLE target, target2; +DROP TABLE source, source2; +DROP FUNCTION merge_trigfunc(); +DROP USER merge_privs; +DROP USER merge_no_privs; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index f7f3bbbeeb..0a8abf2076 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -349,6 +349,114 @@ UPDATE atest5 SET one = 1; -- fail SELECT atest6 FROM atest6; -- ok COPY atest6 TO stdout; -- ok +-- test column privileges with MERGE +SET SESSION AUTHORIZATION regress_priv_user1; +CREATE TABLE mtarget (a int, b text); +CREATE TABLE msource (a int, b text); +INSERT INTO mtarget VALUES (1, 'init1'), (2, 'init2'); +INSERT INTO msource VALUES (1, 'source1'), (2, 'source2'), (3, 'source3'); + +GRANT SELECT (a) ON msource TO regress_priv_user4; +GRANT SELECT (a) ON mtarget TO regress_priv_user4; +GRANT INSERT (a,b) ON mtarget TO regress_priv_user4; +GRANT UPDATE (b) ON mtarget TO regress_priv_user4; + +SET SESSION AUTHORIZATION regress_priv_user4; + +-- +-- test source privileges +-- + +-- fail (no SELECT priv on s.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- fail (s.b used in the INSERTed values) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- fail (s.b used in the WHEN quals) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND s.b = 'x' THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- this should be ok since only s.a is accessed +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'ok' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ROLLBACK; + +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT SELECT (b) ON msource TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; + +-- should now be ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ROLLBACK; + +-- +-- test target privileges +-- + +-- fail (no SELECT priv on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- fail (no UPDATE on t.a) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b, a = t.a + 1 +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- fail (no SELECT on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b; +ROLLBACK; + +-- fail (no DELETE) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; + +-- grant delete privileges +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT DELETE ON mtarget TO regress_priv_user4; +-- should be ok now +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ROLLBACK; + -- check error reporting with column privs SET SESSION AUTHORIZATION regress_priv_user1; CREATE TABLE t1 (c1 int, c2 int, c3 int check (c3 < 5), primary key (c1, c2)); diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql index f3a31dbee0..6c75208998 100644 --- a/src/test/regress/sql/rowsecurity.sql +++ b/src/test/regress/sql/rowsecurity.sql @@ -812,6 +812,162 @@ INSERT INTO document VALUES (4, (SELECT cid from category WHERE cname = 'novel') INSERT INTO document VALUES (1, (SELECT cid from category WHERE cname = 'novel'), 1, 'regress_rls_bob', 'my first novel') ON CONFLICT (did) DO UPDATE SET dauthor = 'regress_rls_carol'; +-- +-- MERGE +-- +RESET SESSION AUTHORIZATION; +DROP POLICY p3_with_all ON document; + +ALTER TABLE document ADD COLUMN dnotes text DEFAULT ''; +-- all documents are readable +CREATE POLICY p1 ON document FOR SELECT USING (true); +-- one may insert documents only authored by them +CREATE POLICY p2 ON document FOR INSERT WITH CHECK (dauthor = current_user); +-- one may only update documents in 'novel' category +CREATE POLICY p3 ON document FOR UPDATE + USING (cid = (SELECT cid from category WHERE cname = 'novel')) + WITH CHECK (dauthor = current_user); +-- one may only delete documents in 'manga' category +CREATE POLICY p4 ON document FOR DELETE + USING (cid = (SELECT cid from category WHERE cname = 'manga')); + +SELECT * FROM document; + +SET SESSION AUTHORIZATION regress_rls_bob; + +-- Fails, since update violates WITH CHECK qual on dauthor +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge1 ', dauthor = 'regress_rls_alice'; + +-- Should be OK since USING and WITH CHECK quals pass +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge2 '; + +-- Even when dauthor is updated explicitly, but to the existing value +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge3 ', dauthor = 'regress_rls_bob'; + +-- There is a MATCH for did = 3, but UPDATE's USING qual does not allow +-- updating an item in category 'science fiction' +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge '; + +-- The same thing with DELETE action, but fails again because no permissions +-- to delete items in 'science fiction' category that did 3 belongs to. +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE; + +-- Document with did 4 belongs to 'manga' category which is allowed for +-- deletion. But this fails because the UPDATE action is matched first and +-- UPDATE policy does not allow updation in the category. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes = '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +-- UPDATE action is not matched this time because of the WHEN AND qual. +-- DELETE still fails because role regress_rls_bob does not have SELECT +-- privileges on 'manga' category row in the category table. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +SELECT * FROM document WHERE did = 4; + +-- Switch to regress_rls_carol role and try the DELETE again. It should succeed +-- this time +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_carol; + +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +-- Switch back to regress_rls_bob role +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_bob; + +-- Try INSERT action. This fails because we are trying to insert +-- dauthor = regress_rls_dave and INSERT's WITH CHECK does not allow +-- that +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_dave', 'another novel'); + +-- This should be fine +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +-- ok +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge4 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +-- drop and create a new SELECT policy which prevents us from reading +-- any document except with category 'magna' +RESET SESSION AUTHORIZATION; +DROP POLICY p1 ON document; +CREATE POLICY p1 ON document FOR SELECT + USING (cid = (SELECT cid from category WHERE cname = 'manga')); + +SET SESSION AUTHORIZATION regress_rls_bob; + +-- MERGE can no longer see the matching row and hence attempts the +-- NOT MATCHED action, which results in unique key violation +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge5 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +RESET SESSION AUTHORIZATION; +-- drop the restrictive SELECT policy so that we can look at the +-- final state of the table +DROP POLICY p1 ON document; +-- Just check everything went per plan +SELECT * FROM document; + -- -- ROLE/GROUP -- diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql index a82f52d154..b866268892 100644 --- a/src/test/regress/sql/rules.sql +++ b/src/test/regress/sql/rules.sql @@ -1191,6 +1191,39 @@ CREATE RULE rules_parted_table_insert AS ON INSERT to rules_parted_table ALTER RULE rules_parted_table_insert ON rules_parted_table RENAME TO rules_parted_table_insert_redirect; DROP TABLE rules_parted_table; +-- +-- test MERGE +-- +CREATE TABLE rule_merge1 (a int, b text); +CREATE TABLE rule_merge2 (a int, b text); +CREATE RULE rule1 AS ON INSERT TO rule_merge1 + DO INSTEAD INSERT INTO rule_merge2 VALUES (NEW.*); +CREATE RULE rule2 AS ON UPDATE TO rule_merge1 + DO INSTEAD UPDATE rule_merge2 SET a = NEW.a, b = NEW.b + WHERE a = OLD.a; +CREATE RULE rule3 AS ON DELETE TO rule_merge1 + DO INSTEAD DELETE FROM rule_merge2 WHERE a = OLD.a; + +-- MERGE not supported for table with rules +MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); + +-- should be ok with the other table though +MERGE INTO rule_merge2 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); + -- -- Test enabling/disabling -- diff --git a/src/test/regress/sql/triggers.sql b/src/test/regress/sql/triggers.sql index 9d3e0ef707..8145519104 100644 --- a/src/test/regress/sql/triggers.sql +++ b/src/test/regress/sql/triggers.sql @@ -2109,6 +2109,53 @@ delete from self_ref where a = 1; drop table self_ref; +-- +-- test transition tables with MERGE +-- +create table merge_target_table (a int primary key, b text); +create trigger merge_target_table_insert_trig + after insert on merge_target_table referencing new table as new_table + for each statement execute procedure dump_insert(); +create trigger merge_target_table_update_trig + after update on merge_target_table referencing old table as old_table new table as new_table + for each statement execute procedure dump_update(); +create trigger merge_target_table_delete_trig + after delete on merge_target_table referencing old table as old_table + for each statement execute procedure dump_delete(); + +create table merge_source_table (a int, b text); +insert into merge_source_table + values (1, 'initial1'), (2, 'initial2'), + (3, 'initial3'), (4, 'initial4'); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when not matched then + insert values (a, b); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated again by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); + +drop table merge_source_table, merge_target_table; + -- cleanup drop function dump_insert(); drop function dump_update(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 17bf55c1f5..a7eb3524cf 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1228,6 +1228,8 @@ MemoryContextCallbackFunction MemoryContextCounters MemoryContextData MemoryContextMethods +MergeAction +MergeActionState MergeAppend MergeAppendPath MergeAppendState @@ -1235,6 +1237,7 @@ MergeJoin MergeJoinClause MergeJoinState MergePath +MergeStmt MergeScanSelCache MetaCommand MinMaxAggInfo -- 2.14.3 (Apple Git-98) ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> @ 2018-03-26 15:05 ` Simon Riggs <[email protected]> 1 sibling, 0 replies; 81+ messages in thread From: Simon Riggs @ 2018-03-26 15:05 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 26 March 2018 at 15:39, Pavan Deolasee <[email protected]> wrote: > Now that ON CONFLICT patch is in, here are rebased patches. The second patch > is to add support for CTE (thanks Peter). > > Apart from rebase, the following things are fixed/improved: > > - Added test cases for column level privileges as suggested by Peter. One > problem got discovered during the process. Since we expand and track source > relation targetlist, the exiting code was demanding SELECT privileges on all > attributes, even though MERGE is only referencing a few attributes on which > the user has privilege. Fixed that by disassociating expansion from the > actual referencing. Good catch Peter. > - Added a test case for RLS where SELECT policy actually hides some rows, as > suggested by Stephen in the past > > - Added check to compare result relation's and merge target relation's OIDs, > as suggested by Robert. Simon thinks it's not necessary given that we now > scan catalog using MVCC snapshot. So will leave it to his discretion when he > takes it up for commit No problem with adding the test, its quick to compare two Oids. > - Improved explanation regarding why we need a second RTE for merge target > relation and general cleanup/improvements in that area > I think it will be a good idea to send any further patches as add-on patches > for reviewer/committer's sake. I will do that unless someone disagrees. +1 So v25 is the "commit candidate" and we can add other patches to it. Given recent bugfix/changes I don't plan to commit this tomorrow anymore. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> @ 2018-03-26 16:06 ` Simon Riggs <[email protected]> 2018-03-27 08:24 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:28 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 1 sibling, 2 replies; 81+ messages in thread From: Simon Riggs @ 2018-03-26 16:06 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 26 March 2018 at 15:39, Pavan Deolasee <[email protected]> wrote: > reviewer 1. In ExecMergeMatched() we have a line of code that does this... if (TransactionIdIsCurrentTransactionId(hufd.xmax)) then errcode(ERRCODE_CARDINALITY_VIOLATION) I notice this is correct, but too strong. It should be possible to run a sequence of MERGE commands inside a transaction, repeatedly updating the same set of rows, as is possible with UPDATE. We need to check whether the xid is the current subxid and the cid is the current commandid, rather than using TransactionIdIsCurrentTransactionId() On further analysis, I note that ON CONFLICT suffers from this problem as well, looks like I just refactored it from there. 2. EXPLAIN ANALYZE looks unchanged from some time back. The math is only correct as long as there are zero rows that do not cause an INS/UPD/DEL. We don't test for that. I think this is a regression from an earlier report of the same bug, or perhaps I didn't fix it at the time. 3. sp. depedning trigger.sgml 4. trigger.sgml replace "specific actions specified" with "events specified in actions" to avoid the double use of "specific" 5. I take it the special code for TransformMerge target relations is replaced by "right_rte"? Seems fragile to leave it like that. Can we add an Assert()? Do we care? 6. I didn't understand "Assume that the top-level join RTE is at the end. The source relation + * is just before that." What is there is no source relation? 7. The formatting of the SQL statement in transformMergeStmt that begins "Construct a query of the form" is borked, so the SQL layout is unclear, just needs pretty print 8. I didn't document why I thought this was true "XXX if we have a constant subquery, we can also skip join", but one of the explain analyze outputs shows this is already true - where we provide a constant query and it skips the join. So perhaps we can remove the comment. (Search for "Seq Scan on target t_1") 9. I think we need to mention that MERGE won't work with rules or inheritance (only partitioning) in the main doc page. The current text says that rules are ignored, which would be true if we didn't specifically throw ERROR feature not supported. 10. Comment needs updating for changes in code below it - "In MERGE when and condition, no system column is allowed" 11. In comment "Since the plan re-evaluated by EvalPlanQual uses the second RTE", suggest using "join RTE" to make it more explicit which RTE we are discussing 12. Missed out merge.sgml from v25 patch. 13. For triggers we say "No separate triggers are defined for <command>MERGE</command>" we should also state the same caveat for POLICY events That's all I can see so far. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-27 08:24 ` Simon Riggs <[email protected]> 2018-03-27 09:31 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 1 sibling, 1 reply; 81+ messages in thread From: Simon Riggs @ 2018-03-27 08:24 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 26 March 2018 at 17:06, Simon Riggs <[email protected]> wrote: > On 26 March 2018 at 15:39, Pavan Deolasee <[email protected]> wrote: > > That's all I can see so far. * change comment “once to” to “once” in src/include/nodes/execnodes.h * change comment “and to run” to “and once to run” * change “result relation” to “target relation” * XXX we probably need to check plan output for CMD_MERGE also * Spurious line feed in src/backend/optimizer/prep/preptlist.c * No need to remove whitespace in src/backend/optimizer/util/relnode.c * README should note that the TABLEOID junk column is not strictly needed when joining to a non-partitioned table but we don't try to optimize that away. Is that an XXX to fix in future or do we just think the extra 4 bytes won't make much difference so we leave it? * Comment in rewriteTargetListMerge() should mention TABLEOID exists to allow us to find the correct relation, not the correct row, comment just copied from CTID above it. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 08:24 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-27 09:31 ` Pavan Deolasee <[email protected]> 2018-03-27 10:46 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Pavan Deolasee @ 2018-03-27 09:31 UTC (permalink / raw) To: Simon Riggs <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Tue, Mar 27, 2018 at 1:54 PM, Simon Riggs <[email protected]> wrote: > On 26 March 2018 at 17:06, Simon Riggs <[email protected]> wrote: > > On 26 March 2018 at 15:39, Pavan Deolasee <[email protected]> > wrote: > > > > > That's all I can see so far. > > * change comment “once to” to “once” in src/include/nodes/execnodes.h > * change comment “and to run” to “and once to run” > * change “result relation” to “target relation” > Fixed all of that in the patch v26 set I just sent. > > * XXX we probably need to check plan output for CMD_MERGE also > Yeah. Added those checks for MERGE action's target lists in v26. > > * Spurious line feed in src/backend/optimizer/prep/preptlist.c > Couldn't spot it. Will look closer, but any hint will be appreciated. > > * No need to remove whitespace in src/backend/optimizer/util/relnode.c > Fixed in v26. > > * README should note that the TABLEOID junk column is not strictly > needed when joining to a non-partitioned table but we don't try to > optimize that away. Is that an XXX to fix in future or do we just > think the extra 4 bytes won't make much difference so we leave it? > I actually took the opportunity to conditionally fetch tableoid only if we are dealing with partitioned table. > > * Comment in rewriteTargetListMerge() should mention TABLEOID exists > to allow us to find the correct relation, not the correct row, comment > just copied from CTID above it. > > Fixed in v26. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 08:24 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:31 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> @ 2018-03-27 10:46 ` Simon Riggs <[email protected]> 2018-03-27 11:30 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Simon Riggs @ 2018-03-27 10:46 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 27 March 2018 at 10:31, Pavan Deolasee <[email protected]> wrote: > Fixed in v26. More comments on v26 * Change errmsg “Ensure that not more than one source rows match any one target row” should be “Ensure that not more than one source row matches any one target row” * I think we need an example in the docs showing a constant source query, so people can understand how to use MERGE for OLTP as well as large ELT * Long comment in ExecMerge() needs rewording, formatting and spell check I suggest not referring to an "order" since that concept doesn't exist anywhere else * Need tests for coverage of these ERROR messages Named security policy violation SELECT not allowed in MERGE INSERT... Multiple VALUES clauses not... MERGE is not supported for this... MERGE is not supported for relations with inheritance MERGE is not supported for relations with rules -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 08:24 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:31 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-27 10:46 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-27 11:30 ` Simon Riggs <[email protected]> 2018-03-29 06:37 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Simon Riggs @ 2018-03-27 11:30 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 27 March 2018 at 11:46, Simon Riggs <[email protected]> wrote: > On 27 March 2018 at 10:31, Pavan Deolasee <[email protected]> wrote: > >> Fixed in v26. > > More comments on v26 In terms of further performance optimization, if there is just one WHEN AND condition and no unconditional WHEN clauses then we can add the WHEN AND easily to the join query. That seems like an easy thing to do for PG11 -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 08:24 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:31 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-27 10:46 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 11:30 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-29 06:37 ` Pavan Deolasee <[email protected]> 2018-03-29 06:56 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Pavan Deolasee @ 2018-03-29 06:37 UTC (permalink / raw) To: Simon Riggs <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Tue, Mar 27, 2018 at 5:00 PM, Simon Riggs <[email protected]> wrote: > > In terms of further performance optimization, if there is just one > WHEN AND condition and no unconditional WHEN clauses then we can add > the WHEN AND easily to the join query. > > That seems like an easy thing to do for PG11 > > I think we need to be careful in terms of what can be pushed down to the join, in presence of WHEN NOT MATCHED actions. If we push the WHEN AND qual to the join then I am worried that some rows which should have been reported "matched" and later filtered out as part of the WHEN quals, will get reported as "not-matched", thus triggering WHEN NOT MATCHED action. For example, postgres=# select * from target ; a | b ---+---- 1 | 10 2 | 20 (2 rows) postgres=# select * from source ; a | b ---+----- 2 | 200 3 | 300 (2 rows) postgres=# BEGIN; BEGIN postgres=# EXPLAIN ANALYZE MERGE INTO target t USING source s ON t.a = s.a WHEN MATCHED AND t.a < 2 THEN UPDATE SET b = s.b WHEN NOT MATCHED THEN INSERT VALUES (s.a, -1); QUERY PLAN --------------------------------------------------------------------------------------------------------------------------- Merge on target t (cost=317.01..711.38 rows=25538 width=46) (actual time=0.104..0.104 rows=0 loops=1) * Tuples Inserted: 1* Tuples Updated: 0 Tuples Deleted: 0 * Tuples Skipped: 1* -> Merge Left Join (cost=317.01..711.38 rows=25538 width=46) (actual time=0.071..0.074 rows=2 loops=1) Merge Cond: (s.a = t_1.a) -> Sort (cost=158.51..164.16 rows=2260 width=40) (actual time=0.042..0.043 rows=2 loops=1) Sort Key: s.a Sort Method: quicksort Memory: 25kB -> Seq Scan on source s (cost=0.00..32.60 rows=2260 width=40) (actual time=0.027..0.031 rows=2 loops=1) -> Sort (cost=158.51..164.16 rows=2260 width=10) (actual time=0.019..0.020 rows=2 loops=1) Sort Key: t_1.a Sort Method: quicksort Memory: 25kB -> Seq Scan on target t_1 (cost=0.00..32.60 rows=2260 width=10) (actual time=0.012..0.014 rows=2 loops=1) Planning Time: 0.207 ms Execution Time: 0.199 ms (17 rows) postgres=# abort; ROLLBACK postgres=# BEGIN; BEGIN postgres=# EXPLAIN ANALYZE MERGE INTO target t USING source s ON t.a = s.a AND t.a < 2 WHEN MATCHED THEN UPDATE SET b = s.b WHEN NOT MATCHED THEN INSERT VALUES (s.a, -1); QUERY PLAN -------------------------------------------------------------------------------------------------------------------------- Merge on target t (cost=232.74..364.14 rows=8509 width=46) (actual time=0.128..0.128 rows=0 loops=1) * Tuples Inserted: 2* Tuples Updated: 0 Tuples Deleted: 0 Tuples Skipped: 0 -> Merge Right Join (cost=232.74..364.14 rows=8509 width=46) (actual time=0.070..0.072 rows=2 loops=1) Merge Cond: (t_1.a = s.a) -> Sort (cost=74.23..76.11 rows=753 width=10) (actual time=0.038..0.039 rows=1 loops=1) Sort Key: t_1.a Sort Method: quicksort Memory: 25kB -> Seq Scan on target t_1 (cost=0.00..38.25 rows=753 width=10) (actual time=0.026..0.028 rows=1 loops=1) Filter: (a < 2) Rows Removed by Filter: 1 -> Sort (cost=158.51..164.16 rows=2260 width=40) (actual time=0.024..0.025 rows=2 loops=1) Sort Key: s.a Sort Method: quicksort Memory: 25kB -> Seq Scan on source s (cost=0.00..32.60 rows=2260 width=40) (actual time=0.014..0.017 rows=2 loops=1) Planning Time: 0.218 ms Execution Time: 0.234 ms (19 rows) postgres=# abort; ROLLBACK If you look at the first MERGE statement, we filter one matched source row (2,200) using (t.a < 2) and do not take any action for that row. This filtering happens after the RIGHT JOIN has reported it as "matched". But if we push down the qual to the join, then the join will see that the source row has no match and hence send that row for NOT MATCHED processing, thus inserting it into the table again. I am not saying there is no scope for improvement. But we need to be careful about what can be pushed down to the join and what must be applied after the join. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 08:24 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:31 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-27 10:46 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 11:30 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-29 06:37 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> @ 2018-03-29 06:56 ` Simon Riggs <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Simon Riggs @ 2018-03-29 06:56 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 29 March 2018 at 07:37, Pavan Deolasee <[email protected]> wrote: > > > On Tue, Mar 27, 2018 at 5:00 PM, Simon Riggs <[email protected]> wrote: >> >> >> In terms of further performance optimization, if there is just one >> WHEN AND condition and no unconditional WHEN clauses then we can add >> the WHEN AND easily to the join query. >> >> That seems like an easy thing to do for PG11 >> > > I think we need to be careful in terms of what can be pushed down to the > join, in presence of WHEN NOT MATCHED actions. If we push the WHEN AND qual > to the join then I am worried that some rows which should have been reported > "matched" and later filtered out as part of the WHEN quals, will get > reported as "not-matched", thus triggering WHEN NOT MATCHED action. > postgres=# EXPLAIN ANALYZE MERGE INTO target t USING source s ON t.a = s.a > WHEN MATCHED AND t.a < 2 THEN UPDATE SET b = s.b WHEN NOT MATCHED THEN > INSERT VALUES (s.a, -1); That has an unconditional WHEN clause, so would block the push down using my stated rule above. With something like this MERGE INTO target t USING source s ON t.a = s.a WHEN MATCHED AND t.a < 2 THEN UPDATE SET b = s.b; or this MERGE INTO target t USING source s ON t.a = s.a WHEN MATCHED AND t.a < 2 THEN UPDATE SET b = s.b WHEN NOT MATCHED DO NOTHING; or this MERGE INTO target t USING source s ON t.a = s.a WHEN MATCHED AND t.a < 2 THEN UPDATE SET b = s.b WHEN MATCHED DO NOTHING WHEN NOT MATCHED DO NOTHING; then we can push down "t.a < 2" into the WHERE clause of the join query. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-27 09:28 ` Pavan Deolasee <[email protected]> 2018-03-27 09:40 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-28 02:58 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 1 sibling, 2 replies; 81+ messages in thread From: Pavan Deolasee @ 2018-03-27 09:28 UTC (permalink / raw) To: Simon Riggs <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Mon, Mar 26, 2018 at 9:36 PM, Simon Riggs <[email protected]> wrote: > On 26 March 2018 at 15:39, Pavan Deolasee <[email protected]> > wrote: > > > reviewer > > 1. In ExecMergeMatched() we have a line of code that does this... > > if (TransactionIdIsCurrentTransactionId(hufd.xmax)) > then errcode(ERRCODE_CARDINALITY_VIOLATION) > > I notice this is correct, but too strong. It should be possible to run > a sequence of MERGE commands inside a transaction, repeatedly updating > the same set of rows, as is possible with UPDATE. > > We need to check whether the xid is the current subxid and the cid is > the current commandid, rather than using > TransactionIdIsCurrentTransactionId() > > AFAICS this is fine because we invoke that code only when HeapTupleSatisfiesUpdate returns HeapTupleSelfUpdated i.e. for the case when the tuple is updated by our transaction after the scan is started. HeapTupleSatisfiesUpdate already checks for command id before returning HeapTupleSelfUpdated. > > 2. EXPLAIN ANALYZE looks unchanged from some time back. The math is > only correct as long as there are zero rows that do not cause an > INS/UPD/DEL. > We don't test for that. I think this is a regression from an earlier > report of the same bug, or perhaps I didn't fix it at the time. > I've now added a separate counter to count all three actions and we also report "Tuples skipped" which could be either because there was no action to handle that source tuple or quals did not match. Added regression tests specific to EXPLAIN ANALYZE. > > 3. sp. depedning trigger.sgml > Fixed. > > 4. trigger.sgml replace "specific actions specified" with "events > specified in actions" > to avoid the double use of "specific" > Fixed. > > 5. I take it the special code for TransformMerge target relations is > replaced by "right_rte"? Seems fragile to leave it like that. Can we > add an Assert()? Do we care? > I didn't get this point. Can you please explain? > > 6. I didn't understand "Assume that the top-level join RTE is at the > end. The source relation > + * is just before that." > What is there is no source relation? > Can that happen? I mean shouldn't there always be a source relation? It could be a subquery or a function scan or just a plain relation, but something, right? > > 7. The formatting of the SQL statement in transformMergeStmt that > begins "Construct a query of the form" is borked, so the SQL layout is > unclear, just needs pretty print > Fixed. > > 8. I didn't document why I thought this was true "XXX if we have a > constant subquery, we can also skip join", but one of the explain > analyze outputs shows this is already true - where we provide a > constant query and it skips the join. So perhaps we can remove the > comment. (Search for "Seq Scan on target t_1") > Agree, removed. > > 9. I think we need to mention that MERGE won't work with rules or > inheritance (only partitioning) in the main doc page. The current > text says that rules are ignored, which would be true if we didn't > specifically throw ERROR feature not supported. > > Added a short para to merge.sgml > 10. Comment needs updating for changes in code below it - "In MERGE > when and condition, no system column is allowed" > > Yeah, that's kinda half-true since the code below supports TABLEOID and OID system columns. I am thinking about this in a larger context though. Peter has expressed desire to support system columns in WHEN targetlist and quals. I gave it a try and it seems if we remove that error block, all system columns are supported readily. But only from the target side. There is a problem if we try to refer a system column from the source side since the mergrSourceTargetList only includes user columns and so set_plan_refs() complains about a system column. I am not sure what's the best way to handle this. May be we can add system columns to the mergrSourceTargetList. I haven't yet found a neat way to do that. > 11. In comment "Since the plan re-evaluated by EvalPlanQual uses the > second RTE", suggest using "join RTE" to make it more explicit which > RTE we are discussing > Ok, fixed. > > 12. Missed out merge.sgml from v25 patch. > Ouch, added. Also generating a new patch which includes merge.sgml and sending other improvements as add-ons. > > 13. For triggers we say "No separate triggers are defined for > <command>MERGE</command>" > we should also state the same caveat for POLICY events > Ok. Added a short para in create_policy.sgml Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services Attachments: [application/octet-stream] v26-0004-Basic-tab-completion-for-MERGE.patch (5.4K, ../../CABOikdOwA4+CJO4MwzF250huNuBWejWugf-GZwrbX-k=NDc2rw@mail.gmail.com/3-v26-0004-Basic-tab-completion-for-MERGE.patch) download | inline diff: From 8a3ddfd59e5162596c40305d6d64c216382e7c31 Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Tue, 27 Mar 2018 14:56:34 +0530 Subject: [PATCH v26 4/4] Basic tab-completion for MERGE --- src/bin/psql/tab-complete.c | 78 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 72 insertions(+), 6 deletions(-) diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 08d8ef09a4..bc8db69874 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -659,6 +659,28 @@ static const SchemaQuery Query_for_list_of_updatables = { NULL }; +/* Relations supporting MERGE */ +static const SchemaQuery Query_for_list_of_mergetargets = { + /* min_server_version */ + 110000, + /* catname */ + "pg_catalog.pg_class c", + /* selcondition */ + "c.relkind IN (" CppAsString2(RELKIND_RELATION) ", " + CppAsString2(RELKIND_PARTITIONED_TABLE) ") AND " + "c.relhasrules = false AND " + "(c.relhassubclass = false OR " + " c.relkind = " CppAsString2(RELKIND_PARTITIONED_TABLE) ")", + /* viscondition */ + "pg_catalog.pg_table_is_visible(c.oid)", + /* namespace */ + "c.relnamespace", + /* result */ + "pg_catalog.quote_ident(c.relname)", + /* qualresult */ + NULL +}; + static const SchemaQuery Query_for_list_of_relations = { /* min_server_version */ 0, @@ -1605,7 +1627,7 @@ psql_completion(const char *text, int start, int end) "COMMENT", "COMMIT", "COPY", "CREATE", "DEALLOCATE", "DECLARE", "DELETE FROM", "DISCARD", "DO", "DROP", "END", "EXECUTE", "EXPLAIN", "FETCH", "GRANT", "IMPORT", "INSERT", "LISTEN", "LOAD", "LOCK", - "MOVE", "NOTIFY", "PREPARE", + "MERGE", "MOVE", "NOTIFY", "PREPARE", "REASSIGN", "REFRESH MATERIALIZED VIEW", "REINDEX", "RELEASE", "RESET", "REVOKE", "ROLLBACK", "SAVEPOINT", "SECURITY LABEL", "SELECT", "SET", "SHOW", "START", @@ -2999,14 +3021,15 @@ psql_completion(const char *text, int start, int end) * Complete EXPLAIN [ANALYZE] [VERBOSE] with list of EXPLAIN-able commands */ else if (Matches1("EXPLAIN")) - COMPLETE_WITH_LIST7("SELECT", "INSERT", "DELETE", "UPDATE", "DECLARE", - "ANALYZE", "VERBOSE"); + COMPLETE_WITH_LIST8("SELECT", "INSERT", "DELETE", "UPDATE", "MERGE", + "DECLARE", "ANALYZE", "VERBOSE"); else if (Matches2("EXPLAIN", "ANALYZE")) - COMPLETE_WITH_LIST6("SELECT", "INSERT", "DELETE", "UPDATE", "DECLARE", - "VERBOSE"); + COMPLETE_WITH_LIST7("SELECT", "INSERT", "DELETE", "UPDATE", "MERGE", + "DECLARE", "VERBOSE"); else if (Matches2("EXPLAIN", "VERBOSE") || Matches3("EXPLAIN", "ANALYZE", "VERBOSE")) - COMPLETE_WITH_LIST5("SELECT", "INSERT", "DELETE", "UPDATE", "DECLARE"); + COMPLETE_WITH_LIST6("SELECT", "INSERT", "DELETE", "UPDATE", "MERGE", + "DECLARE"); /* FETCH && MOVE */ /* Complete FETCH with one of FORWARD, BACKWARD, RELATIVE */ @@ -3300,6 +3323,49 @@ psql_completion(const char *text, int start, int end) Matches5("LOCK", "TABLE", MatchAny, "IN", "SHARE")) COMPLETE_WITH_LIST3("MODE", "ROW EXCLUSIVE MODE", "UPDATE EXCLUSIVE MODE"); +/* MERGE --- can be inside EXPLAIN */ + else if (TailMatches1("MERGE")) + COMPLETE_WITH_CONST("INTO"); + else if (TailMatches2("MERGE", "INTO")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_mergetargets, NULL); + else if (TailMatches3("MERGE", "INTO", MatchAny)) + COMPLETE_WITH_CONST("USING"); + else if (TailMatches4("MERGE", "INTO", MatchAny, MatchAny)) + COMPLETE_WITH_CONST("USING"); + else if (TailMatches5("MERGE", "INTO", MatchAny, "AS", MatchAny)) + COMPLETE_WITH_CONST("USING"); + else if (TailMatches4("MERGE", "INTO", MatchAny, "USING")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL); + else if (TailMatches6("MERGE", "INTO", MatchAny, "AS", MatchAny, "USING")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL); + else if (TailMatches5("MERGE", "INTO", MatchAny, MatchAny, "USING")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL); + else if (TailMatches5("MERGE", "INTO", MatchAny, "USING", MatchAny)) + COMPLETE_WITH_CONST("ON"); + else if (TailMatches7("MERGE", "INTO", MatchAny, "USING", MatchAny, "AS", MatchAny)) + COMPLETE_WITH_CONST("ON"); + else if (TailMatches9("MERGE", "INTO", MatchAny, "AS", MatchAny, "USING", MatchAny, "AS", MatchAny)) + COMPLETE_WITH_CONST("ON"); + else if (TailMatches7("MERGE", "INTO", MatchAny, MatchAny, "USING", MatchAny, MatchAny)) + COMPLETE_WITH_CONST("ON"); + else if (TailMatches5("INTO", MatchAny, "USING", MatchAny, "ON")) + COMPLETE_WITH_ATTR(prev4_wd, ""); + else if (TailMatches7("INTO", MatchAny, MatchAny, "USING", MatchAny, MatchAny, "ON")) + COMPLETE_WITH_ATTR(prev6_wd, ""); + else if (TailMatches9("INTO", MatchAny, "AS", MatchAny, "USING", MatchAny, "AS", MatchAny, "ON")) + COMPLETE_WITH_ATTR(prev8_wd, ""); + else if (TailMatches4("USING", MatchAny, "ON", MatchAny)) + COMPLETE_WITH_LIST2("WHEN MATCHED", "WHEN NOT MATCHED"); + else if (TailMatches2("WHEN", "MATCHED")) + COMPLETE_WITH_LIST2("THEN", "AND"); + else if (TailMatches3("WHEN", "NOT", "MATCHED")) + COMPLETE_WITH_LIST2("THEN", "AND"); + else if (TailMatches3("WHEN", "MATCHED", "THEN")) + COMPLETE_WITH_LIST2("UPDATE", "DELETE"); + else if (TailMatches4("WHEN", "NOT", "MATCHED", "THEN")) + COMPLETE_WITH_LIST2("INSERT", "DO"); + else if (TailMatches5("WHEN", "NOT", "MATCHED", "THEN", "DO")) + COMPLETE_WITH_CONST("NOTHING"); /* NOTIFY --- can be inside EXPLAIN, RULE, etc */ else if (TailMatches1("NOTIFY")) -- 2.14.3 (Apple Git-98) [application/octet-stream] v26-0003-Fix-EXPLAIN-ANALYZE-output-to-report-counts-corr.patch (27.4K, ../../CABOikdOwA4+CJO4MwzF250huNuBWejWugf-GZwrbX-k=NDc2rw@mail.gmail.com/4-v26-0003-Fix-EXPLAIN-ANALYZE-output-to-report-counts-corr.patch) download | inline diff: From 1fb26a555eeb82a6dfeea91bdac21369b9121422 Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Tue, 27 Mar 2018 09:20:33 +0530 Subject: [PATCH v26 3/4] Fix EXPLAIN ANALYZE output to report counts correctly. Fix some typos in passing Address Simon's review comments Run ExecCheckPlanOutput() for MERGE action's targetlists Fetch tableoid only for partitioned tables Some other misc fixes --- doc/src/sgml/ref/create_policy.sgml | 7 ++ doc/src/sgml/ref/merge.sgml | 7 ++ doc/src/sgml/trigger.sgml | 6 +- src/backend/commands/explain.c | 9 ++- src/backend/commands/trigger.c | 2 +- src/backend/executor/nodeMerge.c | 29 ++++--- src/backend/executor/nodeModifyTable.c | 11 +-- src/backend/optimizer/util/relnode.c | 1 + src/backend/parser/parse_clause.c | 6 ++ src/backend/parser/parse_merge.c | 23 ++---- src/backend/rewrite/rewriteHandler.c | 32 ++++---- src/include/executor/instrument.h | 7 +- src/include/nodes/execnodes.h | 11 ++- src/test/regress/expected/merge.out | 138 ++++++++++++++++++++++++++++++++- src/test/regress/expected/with.out | 16 ++-- src/test/regress/sql/merge.sql | 44 +++++++++++ 16 files changed, 277 insertions(+), 72 deletions(-) diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index 0e35b0ef43..32f39a48ba 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -94,6 +94,13 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable exist, a <quote>default deny</quote> policy is assumed, so that no rows will be visible or updatable. </para> + + <para> + No separate policy exists for <command>MERGE</command>. Instead policies + defined for <literal>SELECT</literal>, <literal>INSERT</literal>, + <literal>UPDATE</literal> and <literal>DELETE</literal> are applied + while executing MERGE, depending on the actions that are activated. + </para> </refsect1> <refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index 405a4cee29..539e512ced 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -115,6 +115,13 @@ DELETE of the <replaceable class="parameter">target_table_name</replaceable> referred to in a <literal>condition</literal>. </para> + + <para> + MERGE is not supported if the <replaceable + class="parameter">target_table_name</replaceable> has + <literal>RULES</literal> defined on it. + See <xref linkend="rules"/> for more information about <literal>RULES</literal>. + </para> </refsect1> <refsect1> diff --git a/doc/src/sgml/trigger.sgml b/doc/src/sgml/trigger.sgml index ac662bc64d..cce58fbf1d 100644 --- a/doc/src/sgml/trigger.sgml +++ b/doc/src/sgml/trigger.sgml @@ -186,15 +186,15 @@ No separate triggers are defined for <command>MERGE</command>. Instead, statement-level or row-level <command>UPDATE</command>, <command>DELETE</command> and <command>INSERT</command> triggers are fired - depedning on what actions are specified in the <command>MERGE</command> query + depending on what actions are specified in the <command>MERGE</command> query and what actions are activated. </para> <para> While running a <command>MERGE</command> command, statement-level <literal>BEFORE</literal> and <literal>AFTER</literal> triggers are fired for - specific actions specified in the <command>MERGE</command>, irrespective of - whether the action is finally activated or not. This is same as + events specified in the actions of the <command>MERGE</command> command, + irrespective of whether the action is finally activated or not. This is same as an <command>UPDATE</command> statement that updates no rows, yet statement-level triggers are fired. The row-level triggers are fired only when a row is actually updated, inserted or deleted. So it's perfectly legal diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index dc2f727d21..ca486872ac 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3086,18 +3086,21 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, double insert_path; double update_path; double delete_path; + double skipped_path; InstrEndLoop(mtstate->mt_plans[0]->instrument); /* count the number of source rows */ total = mtstate->mt_plans[0]->instrument->ntuples; - update_path = mtstate->ps.instrument->nfiltered1; - delete_path = mtstate->ps.instrument->nfiltered2; - insert_path = total - update_path - delete_path; + insert_path = mtstate->ps.instrument->nfiltered1; + update_path = mtstate->ps.instrument->nfiltered2; + delete_path = mtstate->ps.instrument->nfiltered3; + skipped_path = total - insert_path - update_path - delete_path; ExplainPropertyFloat("Tuples Inserted", NULL, insert_path, 0, es); ExplainPropertyFloat("Tuples Updated", NULL, update_path, 0, es); ExplainPropertyFloat("Tuples Deleted", NULL, delete_path, 0, es); + ExplainPropertyFloat("Tuples Skipped", NULL, skipped_path, 0, es); } } diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c index 331c37ad67..1617706376 100644 --- a/src/backend/commands/trigger.c +++ b/src/backend/commands/trigger.c @@ -3317,7 +3317,7 @@ ltrmark:; * If we're running MERGE then we must install the * new tuple in the slot of the underlying join query and * not the result relation itself. If the join does not - * yeild any tuple, the caller will take the necessary + * yield any tuple, the caller will take the necessary * action. */ epqslot = EvalPlanQual(estate, diff --git a/src/backend/executor/nodeMerge.c b/src/backend/executor/nodeMerge.c index e633af7704..ee41ed2eb2 100644 --- a/src/backend/executor/nodeMerge.c +++ b/src/backend/executor/nodeMerge.c @@ -59,8 +59,6 @@ ExecMergeMatched(ModifyTableState *mtstate, EState *estate, { ExprContext *econtext = mtstate->ps.ps_ExprContext; bool isNull; - Datum datum; - Oid tableoid = InvalidOid; List *mergeMatchedActionStates = NIL; HeapUpdateFailureData hufd; bool tuple_updated, @@ -73,22 +71,22 @@ ExecMergeMatched(ModifyTableState *mtstate, EState *estate, ListCell *l; TupleTableSlot *saved_slot = slot; - - /* - * We always fetch the tableoid while performing MATCHED MERGE action. - * This is strictly not required if the target table is not a partitioned - * table. But we are not yet optimising for that case. - */ - datum = ExecGetJunkAttribute(slot, junkfilter->jf_otherJunkAttNo, - &isNull); - Assert(!isNull); - tableoid = DatumGetObjectId(datum); - if (mtstate->mt_partition_tuple_routing) { + Datum datum; + Oid tableoid = InvalidOid; int leaf_part_index; PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing; + /* + * In case of partitioned table, we fetch the tableoid while performing + * MATCHED MERGE action. + */ + datum = ExecGetJunkAttribute(slot, junkfilter->jf_otherJunkAttNo, + &isNull); + Assert(!isNull); + tableoid = DatumGetObjectId(datum); + /* * If we're dealing with a MATCHED tuple, then tableoid must have been * set correctly. In case of partitioned table, we must now fetch the @@ -361,9 +359,9 @@ lmerge_matched:; } if (action->commandType == CMD_UPDATE && tuple_updated) - InstrCountFiltered1(&mtstate->ps, 1); - if (action->commandType == CMD_DELETE && tuple_deleted) InstrCountFiltered2(&mtstate->ps, 1); + if (action->commandType == CMD_DELETE && tuple_deleted) + InstrCountFiltered3(&mtstate->ps, 1); /* * We've activated one of the WHEN clauses, so we don't search @@ -466,6 +464,7 @@ ExecMergeNotMatched(ModifyTableState *mtstate, EState *estate, /* Revert ExecPrepareTupleRouting's state change. */ if (proute) estate->es_result_relation_info = resultRelInfo; + InstrCountFiltered1(&mtstate->ps, 1); break; case CMD_NOTHING: /* Do Nothing */ diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index def7eec223..08f812fc6d 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2733,9 +2733,13 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) switch (action->commandType) { case CMD_INSERT: + ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, + action->targetList); mtstate->mt_merge_subcommands |= MERGE_INSERT; break; case CMD_UPDATE: + ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, + action->targetList); mtstate->mt_merge_subcommands |= MERGE_UPDATE; break; case CMD_DELETE: @@ -2814,10 +2818,6 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) subplan = mtstate->mt_plans[i]->plan; - /* - * XXX we probably need to check plan output for CMD_MERGE - * also - */ if (operation == CMD_INSERT || operation == CMD_UPDATE) ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, subplan->targetlist); @@ -2842,7 +2842,8 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) if (!AttributeNumberIsValid(j->jf_junkAttNo)) elog(ERROR, "could not find junk ctid column"); - if (operation == CMD_MERGE) + if (operation == CMD_MERGE && + relkind == RELKIND_PARTITIONED_TABLE) { j->jf_otherJunkAttNo = ExecFindJunkAttribute(j, "tableoid"); if (!AttributeNumberIsValid(j->jf_otherJunkAttNo)) diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index 901cf24e20..da8f0f93fc 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -1237,6 +1237,7 @@ find_childrel_parents(PlannerInfo *root, RelOptInfo *rel) return result; } + /* * get_baserel_parampathinfo * Get the ParamPathInfo for a parameterized path for a base relation, diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 9df9828df8..857fe058c3 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -1094,6 +1094,12 @@ getRTEForSpecialRelationTypes(ParseState *pstate, RangeVar *rv) * * *top_rti: receives the rangetable index of top_rte. (Ditto.) * + * *right_rte: receives the RTE corresponding to the right side of the + * jointree. Only MERGE really needs to know about this and only MERGE passes a + * non-NULL pointer. + * + * *right_rti: receives the rangetable index of the right_rte. + * * *namespace: receives a List of ParseNamespaceItems for the RTEs exposed * as table/column names by this item. (The lateral_only flags in these items * are indeterminate and should be explicitly set by the caller before use.) diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c index 26a6692231..6df63439bb 100644 --- a/src/backend/parser/parse_merge.c +++ b/src/backend/parser/parse_merge.c @@ -279,10 +279,13 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) } /* - * Construct a query of the form SELECT relation.ctid --junk attribute - * ,relation.tableoid --junk attribute ,source_relation.<somecols> - * ,relation.<somecols> FROM relation RIGHT JOIN source_relation ON - * join_condition -- no WHERE clause - all conditions are applied in + * Construct a query of the form + * SELECT relation.ctid --junk attribute + * ,relation.tableoid --junk attribute + * ,source_relation.<somecols> + * ,relation.<somecols> + * FROM relation RIGHT JOIN source_relation + * ON join_condition; -- no WHERE clause - all conditions are applied in * executor * * stmt->relation is the target relation, given as a RangeVar @@ -353,18 +356,6 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) * If there are no INSERT actions we won't be using the non-matching * candidate rows for anything, so no need for an outer join. We do still * need an inner join for UPDATE and DELETE actions. - * - * Possible additional simplifications... - * - * XXX if we have a constant ON clause, we can skip join altogether - * - * XXX if we have a constant subquery, we can also skip join - * - * XXX if we were really keen we could look through the actionList and - * pull out common conditions, if there were no terminal clauses and put - * them into the main query as an early row filter but that seems like an - * atypical case and so checking for it would be likely to just be wasted - * effort. */ if (targetPerms & ACL_INSERT) joinexpr->jointype = JOIN_RIGHT; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index 45875ec289..e25aa9a35b 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1406,22 +1406,26 @@ rewriteTargetListMerge(Query *parsetree, Relation target_relation) parsetree->targetList = lappend(parsetree->targetList, tle); /* - * Emit TABLEOID so that executor can find the row to update or delete. + * If we are dealing with partitioned table, then emit TABLEOID so that + * executor can find the partition the row belongs to. */ - var = makeVar(parsetree->mergeTarget_relation, - TableOidAttributeNumber, - OIDOID, - -1, - InvalidOid, - 0); - - attrname = "tableoid"; - tle = makeTargetEntry((Expr *) var, - list_length(parsetree->targetList) + 1, - pstrdup(attrname), - true); + if (target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + var = makeVar(parsetree->mergeTarget_relation, + TableOidAttributeNumber, + OIDOID, + -1, + InvalidOid, + 0); + + attrname = "tableoid"; + tle = makeTargetEntry((Expr *) var, + list_length(parsetree->targetList) + 1, + pstrdup(attrname), + true); - parsetree->targetList = lappend(parsetree->targetList, tle); + parsetree->targetList = lappend(parsetree->targetList, tle); + } } /* diff --git a/src/include/executor/instrument.h b/src/include/executor/instrument.h index b72f91898a..28eb0093d4 100644 --- a/src/include/executor/instrument.h +++ b/src/include/executor/instrument.h @@ -58,8 +58,11 @@ typedef struct Instrumentation double total; /* Total total time (in seconds) */ double ntuples; /* Total tuples produced */ double nloops; /* # of run cycles for this node */ - double nfiltered1; /* # tuples removed by scanqual or joinqual */ - double nfiltered2; /* # tuples removed by "other" quals */ + double nfiltered1; /* # tuples removed by scanqual or joinqual OR + * # tuples inserted by MERGE */ + double nfiltered2; /* # tuples removed by "other" quals OR + * # tuples updated by MERGE */ + double nfiltered3; /* # tuples deleted by MERGE */ BufferUsage bufusage; /* Total buffer usage */ } Instrumentation; diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 0bf1d7aeb6..a839d53334 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -467,12 +467,12 @@ typedef struct ResultRelInfo MergeState *ri_mergeState; /* - * While executing MERGE, the target relation is processed twice; once to - * as a result relation and to run a join between the target and the + * While executing MERGE, the target relation is processed twice; once + * as a target relation and once to run a join between the target and the * source. We generate two different RTEs for these two purposes, one with * rte->inh set to false and other with rte->inh set to true. * - * Since the plan re-evaluated by EvalPlanQual uses the second RTE, we must + * Since the plan re-evaluated by EvalPlanQual uses the join RTE, we must * install the updated tuple in the scan corresponding to that RTE. The * following member tracks the index of the second RTE for EvalPlanQual * purposes. ri_mergeTargetRTI is non-zero only when MERGE is in-progress. @@ -1005,6 +1005,11 @@ typedef struct PlanState if (((PlanState *)(node))->instrument) \ ((PlanState *)(node))->instrument->nfiltered2 += (delta); \ } while(0) +#define InstrCountFiltered3(node, delta) \ + do { \ + if (((PlanState *)(node))->instrument) \ + ((PlanState *)(node))->instrument->nfiltered3 += (delta); \ + } while(0) /* * EPQState is state for executing an EvalPlanQual recheck on a candidate diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out index 411ccd0e66..90f3177743 100644 --- a/src/test/regress/expected/merge.out +++ b/src/test/regress/expected/merge.out @@ -822,6 +822,7 @@ NOTICE: AFTER INSERT STATEMENT trigger Tuples Inserted: 1 Tuples Updated: 1 Tuples Deleted: 1 + Tuples Skipped: 0 -> Hash Left Join (actual rows=3 loops=1) Hash Cond: (s.sid = t_1.tid) -> Seq Scan on source s (actual rows=3 loops=1) @@ -840,7 +841,7 @@ NOTICE: AFTER INSERT STATEMENT trigger Trigger merge_bsd: calls=1 Trigger merge_bsi: calls=1 Trigger merge_bsu: calls=1 -(22 rows) +(23 rows) SELECT * FROM target ORDER BY tid; tid | balance @@ -1085,6 +1086,7 @@ NOTICE: AFTER UPDATE STATEMENT trigger Tuples Inserted: 0 Tuples Updated: 1 Tuples Deleted: 0 + Tuples Skipped: 0 -> Seq Scan on target t_1 (actual rows=1 loops=1) Filter: (tid = 1) Rows Removed by Filter: 2 @@ -1092,7 +1094,7 @@ NOTICE: AFTER UPDATE STATEMENT trigger Trigger merge_asu: calls=1 Trigger merge_bru: calls=1 Trigger merge_bsu: calls=1 -(11 rows) +(12 rows) SELECT * FROM target ORDER BY tid; tid | balance @@ -1210,6 +1212,138 @@ ERROR: syntax error at or near "RETURNING" LINE 10: RETURNING * ^ ROLLBACK; +-- EXPLAIN +CREATE TABLE ex_mtarget (a int, b int); +CREATE TABLE ex_msource (a int, b int); +INSERT INTO ex_mtarget SELECT i, i*10 FROM generate_series(1,100,2) i; +INSERT INTO ex_msource SELECT i, i*10 FROM generate_series(1,100,1) i; +-- only updates +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b + 1; + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 50 + Tuples Deleted: 0 + Tuples Skipped: 0 + -> Merge Join (actual rows=50 loops=1) + Merge Cond: (t_1.a = s.a) + -> Sort (actual rows=50 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=50 loops=1) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) +(15 rows) + +-- only updates to selected tuples +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1; + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 5 + Tuples Deleted: 0 + Tuples Skipped: 45 + -> Merge Join (actual rows=50 loops=1) + Merge Cond: (t_1.a = s.a) + -> Sort (actual rows=50 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=50 loops=1) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) +(15 rows) + +-- updates + deletes +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1 +WHEN MATCHED AND t.a >= 10 AND t.a <= 20 THEN + DELETE; + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 5 + Tuples Deleted: 5 + Tuples Skipped: 40 + -> Merge Join (actual rows=50 loops=1) + Merge Cond: (t_1.a = s.a) + -> Sort (actual rows=50 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=50 loops=1) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) +(15 rows) + +-- only inserts +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN NOT MATCHED AND s.a < 10 THEN + INSERT VALUES (a, b); + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 4 + Tuples Updated: 0 + Tuples Deleted: 0 + Tuples Skipped: 96 + -> Merge Left Join (actual rows=100 loops=1) + Merge Cond: (s.a = t_1.a) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) + -> Sort (actual rows=45 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=45 loops=1) +(15 rows) + +-- all three +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1 +WHEN MATCHED AND t.a >= 30 AND t.a <= 40 THEN + DELETE +WHEN NOT MATCHED AND s.a < 20 THEN + INSERT VALUES (a, b); + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 10 + Tuples Updated: 9 + Tuples Deleted: 5 + Tuples Skipped: 76 + -> Merge Left Join (actual rows=100 loops=1) + Merge Cond: (s.a = t_1.a) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) + -> Sort (actual rows=49 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=49 loops=1) +(15 rows) + +DROP TABLE ex_msource, ex_mtarget; -- Subqueries BEGIN; MERGE INTO sq_target t diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out index 543ca4f272..ba2c937bca 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -1932,10 +1932,10 @@ WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); -> Result Output: 1, 'cte_basic val'::text -> Hash Right Join - Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Output: o.k, o.v, o.*, m_1.ctid Hash Cond: (m_1.k = o.k) -> Seq Scan on public.m m_1 - Output: m_1.ctid, m_1.tableoid, m_1.k + Output: m_1.ctid, m_1.k -> Hash Output: o.k, o.v, o.* -> Subquery Scan on o @@ -1981,10 +1981,10 @@ WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); Output: (cte_init.b || ' merge update'::text) Filter: (cte_init.a = 1) -> Hash Right Join - Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Output: o.k, o.v, o.*, m_1.ctid Hash Cond: (m_1.k = o.k) -> Seq Scan on public.m m_1 - Output: m_1.ctid, m_1.tableoid, m_1.k + Output: m_1.ctid, m_1.k -> Hash Output: o.k, o.v, o.* -> Subquery Scan on o @@ -2011,8 +2011,8 @@ WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); - QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------ + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- Merge on public.m CTE merge_source_cte -> Result @@ -2025,10 +2025,10 @@ WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text -> CTE Scan on merge_source_cte merge_source_cte_2 Output: ((merge_source_cte_2.*)::text || ' merge insert'::text) -> Hash Right Join - Output: merge_source_cte.a, merge_source_cte.b, ROW(merge_source_cte.a, merge_source_cte.b), m_1.ctid, m_1.tableoid + Output: merge_source_cte.a, merge_source_cte.b, ROW(merge_source_cte.a, merge_source_cte.b), m_1.ctid Hash Cond: (m_1.k = merge_source_cte.a) -> Seq Scan on public.m m_1 - Output: m_1.ctid, m_1.tableoid, m_1.k + Output: m_1.ctid, m_1.k -> Hash Output: merge_source_cte.a, merge_source_cte.b -> CTE Scan on merge_source_cte diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql index 8b5244fc63..cd6144bb5f 100644 --- a/src/test/regress/sql/merge.sql +++ b/src/test/regress/sql/merge.sql @@ -790,6 +790,50 @@ RETURNING * ; ROLLBACK; +-- EXPLAIN +CREATE TABLE ex_mtarget (a int, b int); +CREATE TABLE ex_msource (a int, b int); +INSERT INTO ex_mtarget SELECT i, i*10 FROM generate_series(1,100,2) i; +INSERT INTO ex_msource SELECT i, i*10 FROM generate_series(1,100,1) i; + +-- only updates +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b + 1; + +-- only updates to selected tuples +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1; + +-- updates + deletes +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1 +WHEN MATCHED AND t.a >= 10 AND t.a <= 20 THEN + DELETE; + +-- only inserts +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN NOT MATCHED AND s.a < 10 THEN + INSERT VALUES (a, b); + +-- all three +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1 +WHEN MATCHED AND t.a >= 30 AND t.a <= 40 THEN + DELETE +WHEN NOT MATCHED AND s.a < 20 THEN + INSERT VALUES (a, b); + +DROP TABLE ex_msource, ex_mtarget; + -- Subqueries BEGIN; MERGE INTO sq_target t -- 2.14.3 (Apple Git-98) [application/octet-stream] v26-0002-Add-support-for-CTE.patch (13.5K, ../../CABOikdOwA4+CJO4MwzF250huNuBWejWugf-GZwrbX-k=NDc2rw@mail.gmail.com/5-v26-0002-Add-support-for-CTE.patch) download | inline diff: From 284b55ccb5dc7c4403063cd38f1dbebc7fbd247f Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Mon, 26 Mar 2018 18:56:50 +0530 Subject: [PATCH v26 2/4] Add support for CTE --- src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/parser/gram.y | 11 +-- src/backend/parser/parse_merge.c | 9 +++ src/include/nodes/parsenodes.h | 1 + src/test/regress/expected/merge.out | 3 - src/test/regress/expected/with.out | 132 ++++++++++++++++++++++++++++++++++++ src/test/regress/sql/with.sql | 51 ++++++++++++++ 9 files changed, 203 insertions(+), 8 deletions(-) diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 770ed3b1a8..c3efca3c45 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -3055,6 +3055,7 @@ _copyMergeStmt(const MergeStmt *from) COPY_NODE_FIELD(source_relation); COPY_NODE_FIELD(join_condition); COPY_NODE_FIELD(mergeActionList); + COPY_NODE_FIELD(withClause); return newnode; } diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 5a0151eece..45ceba2830 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -1051,6 +1051,7 @@ _equalMergeStmt(const MergeStmt *a, const MergeStmt *b) COMPARE_NODE_FIELD(source_relation); COMPARE_NODE_FIELD(join_condition); COMPARE_NODE_FIELD(mergeActionList); + COMPARE_NODE_FIELD(withClause); return true; } diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 68e2cec66e..7106765e2b 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3446,6 +3446,8 @@ raw_expression_tree_walker(Node *node, return true; if (walker(stmt->mergeActionList, context)) return true; + if (walker(stmt->withClause, context)) + return true; } break; case T_SelectStmt: diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index ebca5f3eb7..2f21571915 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -11105,17 +11105,18 @@ set_target_list: *****************************************************************************/ MergeStmt: - MERGE INTO relation_expr_opt_alias + opt_with_clause MERGE INTO relation_expr_opt_alias USING table_ref ON a_expr merge_when_list { MergeStmt *m = makeNode(MergeStmt); - m->relation = $3; - m->source_relation = $5; - m->join_condition = $7; - m->mergeActionList = $8; + m->withClause = $1; + m->relation = $4; + m->source_relation = $6; + m->join_condition = $8; + m->mergeActionList = $9; $$ = (Node *)m; } diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c index b6e0c46656..26a6692231 100644 --- a/src/backend/parser/parse_merge.c +++ b/src/backend/parser/parse_merge.c @@ -24,6 +24,7 @@ #include "parser/parsetree.h" #include "parser/parser.h" #include "parser/parse_clause.h" +#include "parser/parse_cte.h" #include "parser/parse_merge.h" #include "parser/parse_relation.h" #include "parser/parse_target.h" @@ -203,6 +204,14 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) qry->commandType = CMD_MERGE; + /* process the WITH clause independently of all else */ + if (stmt->withClause) + { + qry->hasRecursive = stmt->withClause->recursive; + qry->cteList = transformWithClause(pstate, stmt->withClause); + qry->hasModifyingCTE = pstate->p_hasModifyingCTE; + } + /* * Check WHEN clauses for permissions and sanity */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 0c904f4d7f..36e6e2e976 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -1519,6 +1519,7 @@ typedef struct MergeStmt Node *source_relation; /* source relation */ Node *join_condition; /* join condition between source and target */ List *mergeActionList; /* list of MergeAction(s) */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeAction diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out index 05c4287078..411ccd0e66 100644 --- a/src/test/regress/expected/merge.out +++ b/src/test/regress/expected/merge.out @@ -1191,9 +1191,6 @@ WHEN NOT MATCHED THEN WHEN MATCHED AND tid < 2 THEN DELETE ; -ERROR: syntax error at or near "MERGE" -LINE 4: MERGE INTO sq_target t - ^ ROLLBACK; -- RETURNING BEGIN; diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out index 2a2085556b..543ca4f272 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -1904,6 +1904,138 @@ RETURNING k, v; (0 rows) DROP TABLE withz; +-- WITH referenced by MERGE statement +CREATE TABLE m AS SELECT i AS k, (i || ' v')::text v FROM generate_series(1, 16, 3) i; +ALTER TABLE m ADD UNIQUE (k); +-- Basic: +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 0; + k | v +---+---------------------- + 0 | merge source SubPlan +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + QUERY PLAN +------------------------------------------------------------------- + Merge on public.m + CTE cte_basic + -> Result + Output: 1, 'cte_basic val'::text + -> Hash Right Join + Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = o.k) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: o.k, o.v, o.* + -> Subquery Scan on o + Output: o.k, o.v, o.* + -> Result + Output: 0, 'merge source SubPlan'::text + SubPlan 2 + -> Limit + Output: ((cte_basic.b || ' merge update'::text)) + -> CTE Scan on cte_basic + Output: (cte_basic.b || ' merge update'::text) + Filter: (cte_basic.a = m.k) +(21 rows) + +-- InitPlan +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 1; + k | v +---+--------------------------- + 1 | cte_init val merge update +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + QUERY PLAN +-------------------------------------------------------------------- + Merge on public.m + CTE cte_init + -> Result + Output: 1, 'cte_init val'::text + InitPlan 2 (returns $1) + -> Limit + Output: ((cte_init.b || ' merge update'::text)) + -> CTE Scan on cte_init + Output: (cte_init.b || ' merge update'::text) + Filter: (cte_init.a = 1) + -> Hash Right Join + Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = o.k) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: o.k, o.v, o.* + -> Subquery Scan on o + Output: o.k, o.v, o.* + -> Result + Output: 1, 'merge source InitPlan'::text +(21 rows) + +-- MERGE source comes from CTE: +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); +-- Examine +SELECT * FROM m where k = 15; + k | v +----+-------------------------------------------------------------- + 15 | merge_source_cte val(15,"merge_source_cte val") merge insert +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------- + Merge on public.m + CTE merge_source_cte + -> Result + Output: 15, 'merge_source_cte val'::text + InitPlan 2 (returns $1) + -> CTE Scan on merge_source_cte merge_source_cte_1 + Output: ((merge_source_cte_1.b || (merge_source_cte_1.*)::text) || ' merge update'::text) + Filter: (merge_source_cte_1.a = 15) + InitPlan 3 (returns $2) + -> CTE Scan on merge_source_cte merge_source_cte_2 + Output: ((merge_source_cte_2.*)::text || ' merge insert'::text) + -> Hash Right Join + Output: merge_source_cte.a, merge_source_cte.b, ROW(merge_source_cte.a, merge_source_cte.b), m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = merge_source_cte.a) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: merge_source_cte.a, merge_source_cte.b + -> CTE Scan on merge_source_cte + Output: merge_source_cte.a, merge_source_cte.b +(20 rows) + +DROP TABLE m; -- check that run to completion happens in proper ordering TRUNCATE TABLE y; INSERT INTO y SELECT generate_series(1, 3); diff --git a/src/test/regress/sql/with.sql b/src/test/regress/sql/with.sql index f85645efde..dd73b334de 100644 --- a/src/test/regress/sql/with.sql +++ b/src/test/regress/sql/with.sql @@ -862,6 +862,57 @@ RETURNING k, v; DROP TABLE withz; +-- WITH referenced by MERGE statement +CREATE TABLE m AS SELECT i AS k, (i || ' v')::text v FROM generate_series(1, 16, 3) i; +ALTER TABLE m ADD UNIQUE (k); + +-- Basic: +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 0; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + +-- InitPlan +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 1; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + +-- MERGE source comes from CTE: +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); +-- Examine +SELECT * FROM m where k = 15; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); + +DROP TABLE m; + -- check that run to completion happens in proper ordering TRUNCATE TABLE y; -- 2.14.3 (Apple Git-98) [application/octet-stream] v26-0001-Version-25c-of-MERGE-patch-based-on-ON-CONFLICT-.patch (345.7K, ../../CABOikdOwA4+CJO4MwzF250huNuBWejWugf-GZwrbX-k=NDc2rw@mail.gmail.com/6-v26-0001-Version-25c-of-MERGE-patch-based-on-ON-CONFLICT-.patch) download | inline diff: From ac0d984c8c0ba587e90216f6fab13ee1d85a0fa0 Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Sat, 24 Mar 2018 17:25:04 +0530 Subject: [PATCH v26 1/4] Version 25c of MERGE patch, based on ON CONFLICT DO UPDATE work Add a check for consistent lookup for result and mergeTarget relation, some comments and fixes Add tests to privileges.sql and ensure that we don't demand SELECT privileges on columns from source side Add additional test case for RLS --- contrib/test_decoding/expected/ddl.out | 46 + contrib/test_decoding/sql/ddl.sql | 16 + doc/src/sgml/libpq.sgml | 8 +- doc/src/sgml/mvcc.sgml | 28 +- doc/src/sgml/plpgsql.sgml | 3 +- doc/src/sgml/ref/allfiles.sgml | 1 + doc/src/sgml/ref/insert.sgml | 11 +- doc/src/sgml/ref/merge.sgml | 599 ++++++++ doc/src/sgml/reference.sgml | 1 + doc/src/sgml/trigger.sgml | 20 + src/backend/access/heap/heapam.c | 3 + src/backend/catalog/sql_features.txt | 6 +- src/backend/commands/explain.c | 30 + src/backend/commands/prepare.c | 1 + src/backend/commands/trigger.c | 156 +- src/backend/executor/Makefile | 2 +- src/backend/executor/README | 10 + src/backend/executor/execMain.c | 17 + src/backend/executor/execPartition.c | 116 ++ src/backend/executor/execReplication.c | 4 +- src/backend/executor/nodeMerge.c | 566 +++++++ src/backend/executor/nodeModifyTable.c | 378 ++++- src/backend/executor/spi.c | 3 + src/backend/nodes/copyfuncs.c | 40 + src/backend/nodes/equalfuncs.c | 32 + src/backend/nodes/nodeFuncs.c | 48 +- src/backend/nodes/outfuncs.c | 25 + src/backend/nodes/readfuncs.c | 6 + src/backend/optimizer/plan/createplan.c | 22 +- src/backend/optimizer/plan/planner.c | 29 +- src/backend/optimizer/plan/setrefs.c | 59 + src/backend/optimizer/prep/preptlist.c | 40 +- src/backend/optimizer/util/pathnode.c | 11 +- src/backend/optimizer/util/plancat.c | 4 + src/backend/optimizer/util/relnode.c | 1 - src/backend/parser/Makefile | 2 +- src/backend/parser/analyze.c | 18 +- src/backend/parser/gram.y | 158 +- src/backend/parser/parse_agg.c | 10 + src/backend/parser/parse_clause.c | 39 +- src/backend/parser/parse_collate.c | 1 + src/backend/parser/parse_expr.c | 3 + src/backend/parser/parse_func.c | 3 + src/backend/parser/parse_merge.c | 670 ++++++++ src/backend/parser/parse_relation.c | 10 + src/backend/rewrite/rewriteHandler.c | 112 +- src/backend/rewrite/rowsecurity.c | 97 ++ src/backend/tcop/pquery.c | 5 + src/backend/tcop/utility.c | 16 + src/include/access/heapam.h | 1 + src/include/commands/trigger.h | 6 +- src/include/executor/execPartition.h | 1 + src/include/executor/nodeMerge.h | 22 + src/include/executor/nodeModifyTable.h | 21 + src/include/executor/spi.h | 1 + src/include/nodes/execnodes.h | 64 +- src/include/nodes/nodes.h | 6 +- src/include/nodes/parsenodes.h | 39 +- src/include/nodes/plannodes.h | 8 +- src/include/nodes/relation.h | 7 +- src/include/optimizer/pathnode.h | 7 +- src/include/parser/analyze.h | 5 + src/include/parser/kwlist.h | 2 + src/include/parser/parse_clause.h | 5 +- src/include/parser/parse_merge.h | 19 + src/include/parser/parse_node.h | 6 +- src/include/rewrite/rewriteHandler.h | 1 + src/interfaces/libpq/fe-exec.c | 9 +- src/pl/plpgsql/src/pl_exec.c | 5 +- src/pl/plpgsql/src/pl_gram.y | 8 + src/pl/plpgsql/src/pl_scanner.c | 1 + src/pl/plpgsql/src/plpgsql.h | 4 +- src/test/isolation/expected/merge-delete.out | 97 ++ .../isolation/expected/merge-insert-update.out | 84 + .../isolation/expected/merge-match-recheck.out | 106 ++ src/test/isolation/expected/merge-update.out | 213 +++ src/test/isolation/isolation_schedule | 4 + src/test/isolation/specs/merge-delete.spec | 51 + src/test/isolation/specs/merge-insert-update.spec | 52 + src/test/isolation/specs/merge-match-recheck.spec | 79 + src/test/isolation/specs/merge-update.spec | 132 ++ src/test/regress/expected/identity.out | 55 + src/test/regress/expected/merge.out | 1599 ++++++++++++++++++++ src/test/regress/expected/privileges.out | 98 ++ src/test/regress/expected/rowsecurity.out | 182 +++ src/test/regress/expected/rules.out | 31 + src/test/regress/expected/triggers.out | 48 + src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/identity.sql | 45 + src/test/regress/sql/merge.sql | 1068 +++++++++++++ src/test/regress/sql/privileges.sql | 108 ++ src/test/regress/sql/rowsecurity.sql | 156 ++ src/test/regress/sql/rules.sql | 33 + src/test/regress/sql/triggers.sql | 47 + src/tools/pgindent/typedefs.list | 3 + 96 files changed, 7879 insertions(+), 149 deletions(-) create mode 100644 doc/src/sgml/ref/merge.sgml create mode 100644 src/backend/executor/nodeMerge.c create mode 100644 src/backend/parser/parse_merge.c create mode 100644 src/include/executor/nodeMerge.h create mode 100644 src/include/parser/parse_merge.h create mode 100644 src/test/isolation/expected/merge-delete.out create mode 100644 src/test/isolation/expected/merge-insert-update.out create mode 100644 src/test/isolation/expected/merge-match-recheck.out create mode 100644 src/test/isolation/expected/merge-update.out create mode 100644 src/test/isolation/specs/merge-delete.spec create mode 100644 src/test/isolation/specs/merge-insert-update.spec create mode 100644 src/test/isolation/specs/merge-match-recheck.spec create mode 100644 src/test/isolation/specs/merge-update.spec create mode 100644 src/test/regress/expected/merge.out create mode 100644 src/test/regress/sql/merge.sql diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index b7c76469fc..79c359d6e3 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -192,6 +192,52 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (33 rows) +-- MERGE support +BEGIN; +MERGE INTO replication_example t + USING (SELECT i as id, i as data, i as num FROM generate_series(-20, 5) i) s + ON t.id = s.id + WHEN MATCHED AND t.id < 0 THEN + UPDATE SET somenum = somenum + 1 + WHEN MATCHED AND t.id >= 0 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.*); +COMMIT; +/* display results */ +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); + data +-------------------------------------------------------------------------------------------------------------------------------------------------- + BEGIN + table public.replication_example: INSERT: id[integer]:-20 somedata[integer]:-20 somenum[integer]:-20 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-19 somedata[integer]:-19 somenum[integer]:-19 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-18 somedata[integer]:-18 somenum[integer]:-18 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-17 somedata[integer]:-17 somenum[integer]:-17 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-16 somedata[integer]:-16 somenum[integer]:-16 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-15 somedata[integer]:-15 somenum[integer]:-14 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-14 somedata[integer]:-14 somenum[integer]:-13 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-13 somedata[integer]:-13 somenum[integer]:-12 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-12 somedata[integer]:-12 somenum[integer]:-11 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-11 somedata[integer]:-11 somenum[integer]:-10 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-10 somedata[integer]:-10 somenum[integer]:-9 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-9 somedata[integer]:-9 somenum[integer]:-8 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-8 somedata[integer]:-8 somenum[integer]:-7 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-7 somedata[integer]:-7 somenum[integer]:-6 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-6 somedata[integer]:-6 somenum[integer]:-5 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-5 somedata[integer]:-5 somenum[integer]:-4 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-4 somedata[integer]:-4 somenum[integer]:-3 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-3 somedata[integer]:-3 somenum[integer]:-2 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-2 somedata[integer]:-2 somenum[integer]:-1 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-1 somedata[integer]:-1 somenum[integer]:0 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: DELETE: id[integer]:0 + table public.replication_example: DELETE: id[integer]:1 + table public.replication_example: DELETE: id[integer]:2 + table public.replication_example: DELETE: id[integer]:3 + table public.replication_example: DELETE: id[integer]:4 + table public.replication_example: DELETE: id[integer]:5 + COMMIT +(28 rows) + CREATE TABLE tr_unique(id2 serial unique NOT NULL, data int); INSERT INTO tr_unique(data) VALUES(10); ALTER TABLE tr_unique RENAME TO tr_pkey; diff --git a/contrib/test_decoding/sql/ddl.sql b/contrib/test_decoding/sql/ddl.sql index c4b10a4cf9..0e608b252f 100644 --- a/contrib/test_decoding/sql/ddl.sql +++ b/contrib/test_decoding/sql/ddl.sql @@ -93,6 +93,22 @@ COMMIT; /* display results */ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- MERGE support +BEGIN; +MERGE INTO replication_example t + USING (SELECT i as id, i as data, i as num FROM generate_series(-20, 5) i) s + ON t.id = s.id + WHEN MATCHED AND t.id < 0 THEN + UPDATE SET somenum = somenum + 1 + WHEN MATCHED AND t.id >= 0 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.*); +COMMIT; + +/* display results */ +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); + CREATE TABLE tr_unique(id2 serial unique NOT NULL, data int); INSERT INTO tr_unique(data) VALUES(10); ALTER TABLE tr_unique RENAME TO tr_pkey; diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml index 1fd5dd9fca..de03046f5c 100644 --- a/doc/src/sgml/libpq.sgml +++ b/doc/src/sgml/libpq.sgml @@ -3873,9 +3873,11 @@ char *PQcmdTuples(PGresult *res); <structname>PGresult</structname>. This function can only be used following the execution of a <command>SELECT</command>, <command>CREATE TABLE AS</command>, <command>INSERT</command>, <command>UPDATE</command>, <command>DELETE</command>, - <command>MOVE</command>, <command>FETCH</command>, or <command>COPY</command> statement, - or an <command>EXECUTE</command> of a prepared query that contains an - <command>INSERT</command>, <command>UPDATE</command>, or <command>DELETE</command> statement. + <command>MERGE</command>, <command>MOVE</command>, <command>FETCH</command>, + or <command>COPY</command> statement, or an <command>EXECUTE</command> of a + prepared query that contains an <command>INSERT</command>, + <command>UPDATE</command>, <command>DELETE</command> + or <command>MERGE</command> statement. If the command that generated the <structname>PGresult</structname> was anything else, <function>PQcmdTuples</function> returns an empty string. The caller should not free the return value directly. It will be freed when diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 24613e3c75..0e3e89af56 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -422,6 +422,31 @@ COMMIT; <literal>11</literal>, which no longer matches the criteria. </para> + <para> + The <command>MERGE</command> allows the user to specify various combinations + of <command>INSERT</command>, <command>UPDATE</command> or + <command>DELETE</command> subcommands. A <command>MERGE</command> command + with both <command>INSERT</command> and <command>UPDATE</command> + subcommands looks similar to <command>INSERT</command> with an + <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee + that either <command>INSERT</command> and <command>UPDATE</command> will occur. + + If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + but the join condition still passes for the current target and the current + source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + and perform its action on the latest version of the row, using standard + EvalPlanQual. MERGE actions can be conditional, so conditions must be + re-evaluated on the latest row, starting from the first action. + + On the other hand, if the row is concurrently updated or deleted so that + the join condition fails, then MERGE will execute a NOT MATCHED action, if it + exists and the AND WHEN qual evaluates to true. + + If MERGE attempts an INSERT and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. MERGE + does not attempt to avoid the ERROR by attempting an UPDATE. + </para> + <para> Because Read Committed mode starts each command with a new snapshot that includes all transactions committed up to that instant, @@ -900,7 +925,8 @@ ERROR: could not serialize access due to read/write dependencies among transact <para> The commands <command>UPDATE</command>, - <command>DELETE</command>, and <command>INSERT</command> + <command>DELETE</command>, <command>INSERT</command> and + <command>MERGE</command> acquire this lock mode on the target table (in addition to <literal>ACCESS SHARE</literal> locks on any other referenced tables). In general, this lock mode will be acquired by any diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml index 7ed926fd51..67b22a0d04 100644 --- a/doc/src/sgml/plpgsql.sgml +++ b/doc/src/sgml/plpgsql.sgml @@ -1246,7 +1246,7 @@ EXECUTE format('SELECT count(*) FROM %I ' </programlisting> Another restriction on parameter symbols is that they only work in <command>SELECT</command>, <command>INSERT</command>, <command>UPDATE</command>, and - <command>DELETE</command> commands. In other statement + <command>DELETE</command> and <command>MERGE</command> commands. In other statement types (generically called utility statements), you must insert values textually even if they are just data values. </para> @@ -1529,6 +1529,7 @@ GET DIAGNOSTICS integer_var = ROW_COUNT; <listitem> <para> <command>UPDATE</command>, <command>INSERT</command>, and <command>DELETE</command> + and <command>MERGE</command> statements set <literal>FOUND</literal> true if at least one row is affected, false if no row is affected. </para> diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 22e6893211..4e01e5641c 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -159,6 +159,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY load SYSTEM "load.sgml"> <!ENTITY lock SYSTEM "lock.sgml"> <!ENTITY move SYSTEM "move.sgml"> +<!ENTITY merge SYSTEM "merge.sgml"> <!ENTITY notify SYSTEM "notify.sgml"> <!ENTITY prepare SYSTEM "prepare.sgml"> <!ENTITY prepareTransaction SYSTEM "prepare_transaction.sgml"> diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml index 62e142fd8e..da294aaa46 100644 --- a/doc/src/sgml/ref/insert.sgml +++ b/doc/src/sgml/ref/insert.sgml @@ -579,6 +579,13 @@ INSERT <replaceable>oid</replaceable> <replaceable class="parameter">count</repl is a partition, an error will occur if one of the input rows violates the partition constraint. </para> + + <para> + You may also wish to consider using <command>MERGE</command>, since that + allows mixed <command>INSERT</command>, <command>UPDATE</command> and + <command>DELETE</command> within a single statement. + See <xref linkend="sql-merge"/>. + </para> </refsect1> <refsect1> @@ -749,7 +756,9 @@ INSERT INTO distributors (did, dname) VALUES (10, 'Conrad International') Also, the case in which a column name list is omitted, but not all the columns are filled from the <literal>VALUES</literal> clause or <replaceable>query</replaceable>, - is disallowed by the standard. + is disallowed by the standard. If you prefer a more SQL Standard + conforming statement than <literal>ON CONFLICT</literal>, see + <xref linkend="sql-merge"/>. </para> <para> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml new file mode 100644 index 0000000000..405a4cee29 --- /dev/null +++ b/doc/src/sgml/ref/merge.sgml @@ -0,0 +1,599 @@ +<!-- +doc/src/sgml/ref/merge.sgml +PostgreSQL documentation +--> + +<refentry id="sql-merge"> + + <refmeta> + <refentrytitle>MERGE</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>MERGE</refname> + <refpurpose>insert, update, or delete rows of a table based upon source data</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +MERGE INTO <replaceable class="parameter">target_table_name</replaceable> [ [ AS ] <replaceable class="parameter">target_alias</replaceable> ] +USING <replaceable class="parameter">data_source</replaceable> +ON <replaceable class="parameter">join_condition</replaceable> +<replaceable class="parameter">when_clause</replaceable> [...] + +where <replaceable class="parameter">data_source</replaceable> is + +{ <replaceable class="parameter">source_table_name</replaceable> | + ( source_query ) +} +[ [ AS ] <replaceable class="parameter">source_alias</replaceable> ] + +and <replaceable class="parameter">when_clause</replaceable> is + +{ WHEN MATCHED [ AND <replaceable class="parameter">condition</replaceable> ] THEN { <replaceable class="parameter">merge_update</replaceable> | <replaceable class="parameter">merge_delete</replaceable> } | + WHEN NOT MATCHED [ AND <replaceable class="parameter">condition</replaceable> ] THEN { <replaceable class="parameter">merge_insert</replaceable> | DO NOTHING } +} + +and <replaceable class="parameter">merge_update</replaceable> is + +UPDATE SET { <replaceable class="parameter">column_name</replaceable> = { <replaceable class="parameter">expression</replaceable> | DEFAULT } | + ( <replaceable class="parameter">column_name</replaceable> [, ...] ) = ( { <replaceable class="parameter">expression</replaceable> | DEFAULT } [, ...] ) + } [, ...] + +and <replaceable class="parameter">merge_insert</replaceable> is + +INSERT [( <replaceable class="parameter">column_name</replaceable> [, ...] )] +[ OVERRIDING { SYSTEM | USER } VALUE ] +{ VALUES ( { <replaceable class="parameter">expression</replaceable> | DEFAULT } [, ...] ) | DEFAULT VALUES } + +and <replaceable class="parameter">merge_delete</replaceable> is + +DELETE +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + <command>MERGE</command> performs actions that modify rows in the + <replaceable class="parameter">target_table_name</replaceable>, + using the <replaceable class="parameter">data_source</replaceable>. + <command>MERGE</command> provides a single <acronym>SQL</acronym> + statement that can conditionally <command>INSERT</command>, + <command>UPDATE</command> or <command>DELETE</command> rows, a task + that would otherwise require multiple procedural language statements. + </para> + + <para> + First, the <command>MERGE</command> command performs a join + from <replaceable class="parameter">data_source</replaceable> to + <replaceable class="parameter">target_table_name</replaceable> + producing zero or more candidate change rows. For each candidate change + row the status of <literal>MATCHED</literal> or <literal>NOT MATCHED</literal> is set + just once, after which <literal>WHEN</literal> clauses are evaluated + in the order specified. If one of them is activated, the specified + action occurs. No more than one <literal>WHEN</literal> clause can be + activated for any candidate change row. + </para> + + <para> + <command>MERGE</command> actions have the same effect as + regular <command>UPDATE</command>, <command>INSERT</command>, or + <command>DELETE</command> commands of the same names. The syntax of + those commands is different, notably that there is no <literal>WHERE</literal> + clause and no tablename is specified. All actions refer to the + <replaceable class="parameter">target_table_name</replaceable>, + though modifications to other tables may be made using triggers. + </para> + + <para> + When <literal>DO NOTHING</literal> action is specified, the source row is + skipped. Since actions are evaluated in the given order, <literal>DO + NOTHING</literal> can be handy to skip non-interesting source rows before + more fine-grained handling. + </para> + + <para> + There is no MERGE privilege. + You must have the <literal>UPDATE</literal> privilege on the column(s) + of the <replaceable class="parameter">target_table_name</replaceable> + referred to in the <literal>SET</literal> clause + if you specify an update action, the <literal>INSERT</literal> privilege + on the <replaceable class="parameter">target_table_name</replaceable> + if you specify an insert action and/or the <literal>DELETE</literal> + privilege on the <replaceable class="parameter">target_table_name</replaceable> + if you specify a delete action on the + <replaceable class="parameter">target_table_name</replaceable>. + Privileges are tested once at statement start and are checked + whether or not particular <literal>WHEN</literal> clauses are activated + during the subsequent execution. + You will require the <literal>SELECT</literal> privilege on the + <replaceable class="parameter">data_source</replaceable> and any column(s) + of the <replaceable class="parameter">target_table_name</replaceable> + referred to in a <literal>condition</literal>. + </para> + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry> + <term><replaceable class="parameter">target_table_name</replaceable></term> + <listitem> + <para> + The name (optionally schema-qualified) of the target table or materialized + view to merge into. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">target_alias</replaceable></term> + <listitem> + <para> + A substitute name for the target table. When an alias is + provided, it completely hides the actual name of the table. For + example, given <literal>MERGE foo AS f</literal>, the remainder of the + <command>MERGE</command> statement must refer to this table as + <literal>f</literal> not <literal>foo</literal>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">source_table_name</replaceable></term> + <listitem> + <para> + The name (optionally schema-qualified) of the source table, view or + transition table. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">source_query</replaceable></term> + <listitem> + <para> + A query (<command>SELECT</command> statement or <command>VALUES</command> + statement) that supplies the rows to be merged into the + <replaceable class="parameter">target_table_name</replaceable>. + Refer to the <xref linkend="sql-select"/> + statement or <xref linkend="sql-values"/> + statement for a description of the syntax. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">source_alias</replaceable></term> + <listitem> + <para> + A substitute name for the data source. When an alias is + provided, it completely hides whether table or query was specified. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">join_condition</replaceable></term> + <listitem> + <para> + <replaceable class="parameter">join_condition</replaceable> is + an expression resulting in a value of type + <type>boolean</type> (similar to a <literal>WHERE</literal> + clause) that specifies which rows in the + <replaceable class="parameter">data_source</replaceable> + match rows in the + <replaceable class="parameter">target_table_name</replaceable>. + </para> + <warning> + <para> + Only columns from <replaceable class="parameter">target_table_name</replaceable> + that attempt to match <replaceable class="parameter">data_source</replaceable> + rows should appear in <replaceable class="parameter">join_condition</replaceable>. + <replaceable class="parameter">join_condition</replaceable> subexpressions that + only reference <replaceable class="parameter">target_table_name</replaceable> + columns can only affect which action is taken, often in surprising ways. + </para> + </warning> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">when_clause</replaceable></term> + <listitem> + <para> + At least one <literal>WHEN</literal> clause is required. + </para> + <para> + If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> + and the candidate change row matches a row in the + <replaceable class="parameter">target_table_name</replaceable> + the <literal>WHEN</literal> clause is activated if the + <replaceable class="parameter">condition</replaceable> is + absent or is present and evaluates to <literal>true</literal>. + If the <literal>WHEN</literal> clause specifies <literal>WHEN NOT MATCHED</literal> + and the candidate change row does not match a row in the + <replaceable class="parameter">target_table_name</replaceable> + the <literal>WHEN</literal> clause is activated if the + <replaceable class="parameter">condition</replaceable> is + absent or is present and evaluates to <literal>true</literal>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">condition</replaceable></term> + <listitem> + <para> + An expression that returns a value of type <type>boolean</type>. + If this expression returns <literal>true</literal> then the <literal>WHEN</literal> + clause will be activated and the corresponding action will occur for + that row. The expression may not contain functions that possibly performs + writes to the database. + </para> + <para> + A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns + in both the source and the target relation. A condition on a + <literal>WHEN NOT MATCHED</literal> clause can only refer to columns from + the source relation, since by definition there is no matching target row. + Only the system attributes from the target table are accessible. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">merge_insert</replaceable></term> + <listitem> + <para> + The specification of an <literal>INSERT</literal> action that inserts + one row into the target table. + The target column names can be listed in any order. If no list of + column names is given at all, the default is all the columns of the + table in their declared order. + </para> + <para> + Each column not present in the explicit or implicit column list will be + filled with a default value, either its declared default value + or null if there is none. + </para> + <para> + If the expression for any column is not of the correct data type, + automatic type conversion will be attempted. + </para> + <para> + If <replaceable class="parameter">target_table_name</replaceable> + is a partitioned table, each row is routed to the appropriate partition + and inserted into it. + If <replaceable class="parameter">target_table_name</replaceable> + is a partition, an error will occur if one of the input rows violates + the partition constraint. + </para> + <para> + Column names may not be specified more than once. + <command>INSERT</command> actions cannot contain sub-selects. + </para> + <para> + The <literal>VALUES</literal> clause can only refer to columns from + the source relation, since by definition there is no matching target row. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">merge_update</replaceable></term> + <listitem> + <para> + The specification of an <literal>UPDATE</literal> action that updates + the current row of the <replaceable + class="parameter">target_table_name</replaceable>. + Column names may not be specified more than once. + </para> + <para> + Do not include the table name, as you would normally do with an + <xref linkend="sql-update"/> command. + For example, <literal>UPDATE tab SET col = 1</literal> is invalid. Also, + do not include a <literal>WHERE</literal> clause, since only the current + row can be updated. For example, + <literal>UPDATE SET col = 1 WHERE key = 57</literal> is invalid. + <command>UPDATE</command> actions cannot contain sub-selects in the + <literal>SET</literal> clause. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">merge_delete</replaceable></term> + <listitem> + <para> + Specifies a <literal>DELETE</literal> action that deletes the current row + of the <replaceable class="parameter">target_table_name</replaceable>. + Do not include the tablename or any other clauses, as you would normally + do with an <xref linkend="sql-delete"/> command. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">column_name</replaceable></term> + <listitem> + <para> + The name of a column in the <replaceable + class="parameter">target_table_name</replaceable>. The column name + can be qualified with a subfield name or array subscript, if + needed. (Inserting into only some fields of a composite + column leaves the other fields null.) When referencing a + column, do not include the table's name in the specification + of a target column. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>OVERRIDING SYSTEM VALUE</literal></term> + <listitem> + <para> + Without this clause, it is an error to specify an explicit value + (other than <literal>DEFAULT</literal>) for an identity column defined + as <literal>GENERATED ALWAYS</literal>. This clause overrides that + restriction. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>OVERRIDING USER VALUE</literal></term> + <listitem> + <para> + If this clause is specified, then any values supplied for identity + columns defined as <literal>GENERATED BY DEFAULT</literal> are ignored + and the default sequence-generated values are applied. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>DEFAULT VALUES</literal></term> + <listitem> + <para> + All columns will be filled with their default values. + (An <literal>OVERRIDING</literal> clause is not permitted in this + form.) + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">expression</replaceable></term> + <listitem> + <para> + An expression to assign to the column. The expression can use the + old values of this and other columns in the table. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>DEFAULT</literal></term> + <listitem> + <para> + Set the column to its default value (which will be NULL if no + specific default expression has been assigned to it). + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title>Outputs</title> + + <para> + On successful completion, a <command>MERGE</command> command returns a command + tag of the form +<screen> +MERGE <replaceable class="parameter">total-count</replaceable> +</screen> + The <replaceable class="parameter">total-count</replaceable> is the total + number of rows changed (whether updated, inserted or deleted). + If <replaceable class="parameter">total-count</replaceable> is 0, no rows + were changed in any way. + </para> + + </refsect1> + + <refsect1> + <title>Execution</title> + + <para> + The following steps take place during the execution of + <command>MERGE</command>. + <orderedlist> + <listitem> + <para> + Perform any BEFORE STATEMENT triggers for all actions specified, whether or + not their <literal>WHEN</literal> clauses are activated during execution. + </para> + </listitem> + <listitem> + <para> + Perform a join from source to target table. + The resulting query will be optimized normally and will produce + a set of candidate change row. For each candidate change row + <orderedlist> + <listitem> + <para> + Evaluate whether each row is MATCHED or NOT MATCHED. + </para> + </listitem> + <listitem> + <para> + Test each WHEN condition in the order specified until one activates. + </para> + </listitem> + <listitem> + <para> + When activated, perform the following actions + <orderedlist> + <listitem> + <para> + Perform any BEFORE ROW triggers that fire for the action's event type. + </para> + </listitem> + <listitem> + <para> + Apply the action specified, invoking any check constraints on the + target table. + However, it will not invoke rules. + </para> + </listitem> + <listitem> + <para> + Perform any AFTER ROW triggers that fire for the action's event type. + </para> + </listitem> + </orderedlist> + </para> + </listitem> + </orderedlist> + </para> + </listitem> + <listitem> + <para> + Perform any AFTER STATEMENT triggers for actions specified, whether or + not they actually occur. This is similar to the behavior of an + <command>UPDATE</command> statement that modifies no rows. + </para> + </listitem> + </orderedlist> + In summary, statement triggers for an event type (say, INSERT) will + be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level + triggers will fire only for the one event type <emphasis>activated</emphasis>. + So a <command>MERGE</command> might fire statement triggers for both + <command>UPDATE</command> and <command>INSERT</command>, even though only + <command>UPDATE</command> row triggers were fired. + </para> + + <para> + You should ensure that the join produces at most one candidate change row + for each target row. In other words, a target row shouldn't join to more + than one data source row. If it does, then only one of the candidate change + rows will be used to modify the target row, later attempts to modify will + cause an error. This can also occur if row triggers make changes to the + target table which are then subsequently modified by <command>MERGE</command>. + If the repeated action is an <command>INSERT</command> this will + cause a uniqueness violation while a repeated <command>UPDATE</command> or + <command>DELETE</command> will cause a cardinality violation; the latter behavior + is required by the <acronym>SQL</acronym> Standard. This differs from + historical <productname>PostgreSQL</productname> behavior of joins in + <command>UPDATE</command> and <command>DELETE</command> statements where second and + subsequent attempts to modify are simply ignored. + </para> + + <para> + If a <literal>WHEN</literal> clause omits an <literal>AND</literal> clause it becomes + the final reachable clause of that kind (<literal>MATCHED</literal> or + <literal>NOT MATCHED</literal>). If a later <literal>WHEN</literal> clause of that kind + is specified it would be provably unreachable and an error is raised. + If a final reachable clause is omitted it is possible that no action + will be taken for a candidate change row. + </para> + + </refsect1> + <refsect1> + <title>Notes</title> + + <para> + The order in which rows are generated from the data source is indeterminate + by default. A <replaceable class="parameter">source_query</replaceable> + can be used to specify a consistent ordering, if required, which might be + needed to avoid deadlocks between concurrent transactions. + </para> + + <para> + There is no <literal>RETURNING</literal> clause with <command>MERGE</command>. + Actions of <command>INSERT</command>, <command>UPDATE</command> and <command>DELETE</command> + cannot contain <literal>RETURNING</literal> or <literal>WITH</literal> clauses. + </para> + + <tip> + <para> + You may also wish to consider using <command>INSERT ... ON CONFLICT</command> as an + alternative statement which offers the ability to run an <command>UPDATE</command> + if a concurrent <command>INSERT</command> occurs. There are a variety of + differences and restrictions between the two statement types and they are not + interchangeable. + </para> + </tip> + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + Perform maintenance on CustomerAccounts based upon new Transactions. + +<programlisting> +MERGE CustomerAccount CA +USING RecentTransactions T +ON T.CustomerId = CA.CustomerId +WHEN MATCHED THEN + UPDATE SET Balance = Balance + TransactionValue +WHEN NOT MATCHED THEN + INSERT (CustomerId, Balance) + VALUES (T.CustomerId, T.TransactionValue); +</programlisting> + + notice that this would be exactly equivalent to the following + statement because the <literal>MATCHED</literal> result does not change + during execution + +<programlisting> +MERGE CustomerAccount CA +USING (Select CustomerId, TransactionValue From RecentTransactions) AS T +ON CA.CustomerId = T.CustomerId +WHEN NOT MATCHED THEN + INSERT (CustomerId, Balance) + VALUES (T.CustomerId, T.TransactionValue) +WHEN MATCHED THEN + UPDATE SET Balance = Balance + TransactionValue; +</programlisting> + </para> + + <para> + Attempt to insert a new stock item along with the quantity of stock. If + the item already exists, instead update the stock count of the existing + item. Don't allow entries that have zero stock. +<programlisting> +MERGE INTO wines w +USING wine_stock_changes s +ON s.winename = w.winename +WHEN NOT MATCHED AND s.stock_delta > 0 THEN + INSERT VALUES(s.winename, s.stock_delta) +WHEN MATCHED AND w.stock + s.stock_delta > 0 THEN + UPDATE SET stock = w.stock + s.stock_delta; +WHEN MATCHED THEN + DELETE; +</programlisting> + + The wine_stock_changes table might be, for example, a temporary table + recently loaded into the database. + </para> + + </refsect1> + + <refsect1> + <title>Compatibility</title> + <para> + This command conforms to the <acronym>SQL</acronym> standard. + </para> + <para> + The DO NOTHING action is an extension to the <acronym>SQL</acronym> standard. + </para> + </refsect1> +</refentry> diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index d27fb414f7..ef2270c467 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -186,6 +186,7 @@ &listen; &load; &lock; + &merge; &move; ¬ify; &prepare; diff --git a/doc/src/sgml/trigger.sgml b/doc/src/sgml/trigger.sgml index c43dbc9786..ac662bc64d 100644 --- a/doc/src/sgml/trigger.sgml +++ b/doc/src/sgml/trigger.sgml @@ -182,6 +182,26 @@ will be fired. </para> + <para> + No separate triggers are defined for <command>MERGE</command>. Instead, + statement-level or row-level <command>UPDATE</command>, + <command>DELETE</command> and <command>INSERT</command> triggers are fired + depedning on what actions are specified in the <command>MERGE</command> query + and what actions are activated. + </para> + + <para> + While running a <command>MERGE</command> command, statement-level + <literal>BEFORE</literal> and <literal>AFTER</literal> triggers are fired for + specific actions specified in the <command>MERGE</command>, irrespective of + whether the action is finally activated or not. This is same as + an <command>UPDATE</command> statement that updates no rows, yet + statement-level triggers are fired. The row-level triggers are fired only + when a row is actually updated, inserted or deleted. So it's perfectly legal + that while statement-level triggers are fired for certain type of action, no + row-level triggers are fired for the same kind of action. + </para> + <para> Trigger functions invoked by per-statement triggers should always return <symbol>NULL</symbol>. Trigger functions invoked by per-row diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index c08ab14c02..dec97a7328 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -3241,6 +3241,7 @@ l1: result == HeapTupleUpdated || result == HeapTupleBeingUpdated); Assert(!(tp.t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = tp.t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(tp.t_data); if (result == HeapTupleSelfUpdated) @@ -3882,6 +3883,7 @@ l2: result == HeapTupleUpdated || result == HeapTupleBeingUpdated); Assert(!(oldtup.t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = oldtup.t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(oldtup.t_data); if (result == HeapTupleSelfUpdated) @@ -5086,6 +5088,7 @@ failed: Assert(result == HeapTupleSelfUpdated || result == HeapTupleUpdated || result == HeapTupleWouldBlock); Assert(!(tuple->t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = tuple->t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(tuple->t_data); if (result == HeapTupleSelfUpdated) diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt index 20d61f3780..9612c135da 100644 --- a/src/backend/catalog/sql_features.txt +++ b/src/backend/catalog/sql_features.txt @@ -229,9 +229,9 @@ F311 Schema definition statement 02 CREATE TABLE for persistent base tables YES F311 Schema definition statement 03 CREATE VIEW YES F311 Schema definition statement 04 CREATE VIEW: WITH CHECK OPTION YES F311 Schema definition statement 05 GRANT statement YES -F312 MERGE statement NO consider INSERT ... ON CONFLICT DO UPDATE -F313 Enhanced MERGE statement NO -F314 MERGE statement with DELETE branch NO +F312 MERGE statement YES consider INSERT ... ON CONFLICT DO UPDATE +F313 Enhanced MERGE statement YES +F314 MERGE statement with DELETE branch YES F321 User authorization YES F341 Usage tables NO no ROUTINE_*_USAGE tables F361 Subprogram support YES diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index c38d178cd9..dc2f727d21 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -887,6 +887,9 @@ ExplainNode(PlanState *planstate, List *ancestors, case CMD_DELETE: pname = operation = "Delete"; break; + case CMD_MERGE: + pname = operation = "Merge"; + break; default: pname = "???"; break; @@ -2948,6 +2951,10 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, operation = "Delete"; foperation = "Foreign Delete"; break; + case CMD_MERGE: + operation = "Merge"; + foperation = "Foreign Merge"; + break; default: operation = "???"; foperation = "Foreign ???"; @@ -3070,6 +3077,29 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, other_path, 0, es); } } + else if (node->operation == CMD_MERGE) + { + /* EXPLAIN ANALYZE display of actual outcome for each tuple proposed */ + if (es->analyze && mtstate->ps.instrument) + { + double total; + double insert_path; + double update_path; + double delete_path; + + InstrEndLoop(mtstate->mt_plans[0]->instrument); + + /* count the number of source rows */ + total = mtstate->mt_plans[0]->instrument->ntuples; + update_path = mtstate->ps.instrument->nfiltered1; + delete_path = mtstate->ps.instrument->nfiltered2; + insert_path = total - update_path - delete_path; + + ExplainPropertyFloat("Tuples Inserted", NULL, insert_path, 0, es); + ExplainPropertyFloat("Tuples Updated", NULL, update_path, 0, es); + ExplainPropertyFloat("Tuples Deleted", NULL, delete_path, 0, es); + } + } if (labeltargets) ExplainCloseGroup("Target Tables", "Target Tables", false, es); diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c index b945b1556a..c3610b1874 100644 --- a/src/backend/commands/prepare.c +++ b/src/backend/commands/prepare.c @@ -151,6 +151,7 @@ PrepareQuery(PrepareStmt *stmt, const char *queryString, case CMD_INSERT: case CMD_UPDATE: case CMD_DELETE: + case CMD_MERGE: /* OK */ break; default: diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c index 9d8df5986e..331c37ad67 100644 --- a/src/backend/commands/trigger.c +++ b/src/backend/commands/trigger.c @@ -85,7 +85,8 @@ static HeapTuple GetTupleForTrigger(EState *estate, ResultRelInfo *relinfo, ItemPointer tid, LockTupleMode lockmode, - TupleTableSlot **newSlot); + TupleTableSlot **newSlot, + HeapUpdateFailureData *hufdp); static bool TriggerEnabled(EState *estate, ResultRelInfo *relinfo, Trigger *trigger, TriggerEvent event, Bitmapset *modifiedCols, @@ -2729,7 +2730,8 @@ bool ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, - HeapTuple fdw_trigtuple) + HeapTuple fdw_trigtuple, + HeapUpdateFailureData *hufdp) { TriggerDesc *trigdesc = relinfo->ri_TrigDesc; bool result = true; @@ -2743,7 +2745,7 @@ ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, if (fdw_trigtuple == NULL) { trigtuple = GetTupleForTrigger(estate, epqstate, relinfo, tupleid, - LockTupleExclusive, &newSlot); + LockTupleExclusive, &newSlot, hufdp); if (trigtuple == NULL) return false; } @@ -2814,6 +2816,7 @@ ExecARDeleteTriggers(EState *estate, ResultRelInfo *relinfo, relinfo, tupleid, LockTupleExclusive, + NULL, NULL); else trigtuple = fdw_trigtuple; @@ -2951,7 +2954,8 @@ ExecBRUpdateTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, HeapTuple fdw_trigtuple, - TupleTableSlot *slot) + TupleTableSlot *slot, + HeapUpdateFailureData *hufdp) { TriggerDesc *trigdesc = relinfo->ri_TrigDesc; HeapTuple slottuple = ExecMaterializeSlot(slot); @@ -2972,7 +2976,7 @@ ExecBRUpdateTriggers(EState *estate, EPQState *epqstate, { /* get a copy of the on-disk tuple we are planning to update */ trigtuple = GetTupleForTrigger(estate, epqstate, relinfo, tupleid, - lockmode, &newSlot); + lockmode, &newSlot, hufdp); if (trigtuple == NULL) return NULL; /* cancel the update action */ } @@ -3092,6 +3096,7 @@ ExecARUpdateTriggers(EState *estate, ResultRelInfo *relinfo, relinfo, tupleid, LockTupleExclusive, + NULL, NULL); else trigtuple = fdw_trigtuple; @@ -3240,7 +3245,8 @@ GetTupleForTrigger(EState *estate, ResultRelInfo *relinfo, ItemPointer tid, LockTupleMode lockmode, - TupleTableSlot **newSlot) + TupleTableSlot **newSlot, + HeapUpdateFailureData *hufdp) { Relation relation = relinfo->ri_RelationDesc; HeapTupleData tuple; @@ -3266,6 +3272,11 @@ ltrmark:; estate->es_output_cid, lockmode, LockWaitBlock, false, &buffer, &hufd); + + /* Let the caller know about failure reason, if any. */ + if (hufdp) + *hufdp = hufd; + switch (test) { case HeapTupleSelfUpdated: @@ -3302,10 +3313,17 @@ ltrmark:; /* it was updated, so look at the updated version */ TupleTableSlot *epqslot; + /* + * If we're running MERGE then we must install the + * new tuple in the slot of the underlying join query and + * not the result relation itself. If the join does not + * yeild any tuple, the caller will take the necessary + * action. + */ epqslot = EvalPlanQual(estate, epqstate, relation, - relinfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(relinfo), lockmode, &hufd.ctid, hufd.xmax); @@ -3828,8 +3846,14 @@ struct AfterTriggersTableData bool before_trig_done; /* did we already queue BS triggers? */ bool after_trig_done; /* did we already queue AS triggers? */ AfterTriggerEventList after_trig_events; /* if so, saved list pointer */ - Tuplestorestate *old_tuplestore; /* "old" transition table, if any */ - Tuplestorestate *new_tuplestore; /* "new" transition table, if any */ + /* "old" transition table for UPDATE, if any */ + Tuplestorestate *old_upd_tuplestore; + /* "new" transition table for UPDATE, if any */ + Tuplestorestate *new_upd_tuplestore; + /* "old" transition table for DELETE, if any */ + Tuplestorestate *old_del_tuplestore; + /* "new" transition table INSERT, if any */ + Tuplestorestate *new_ins_tuplestore; }; static AfterTriggersData afterTriggers; @@ -4296,13 +4320,19 @@ AfterTriggerExecute(AfterTriggerEvent event, { if (LocTriggerData.tg_trigger->tgoldtable) { - LocTriggerData.tg_oldtable = evtshared->ats_table->old_tuplestore; + if (TRIGGER_FIRED_BY_UPDATE(evtshared->ats_event)) + LocTriggerData.tg_oldtable = evtshared->ats_table->old_upd_tuplestore; + else + LocTriggerData.tg_oldtable = evtshared->ats_table->old_del_tuplestore; evtshared->ats_table->closed = true; } if (LocTriggerData.tg_trigger->tgnewtable) { - LocTriggerData.tg_newtable = evtshared->ats_table->new_tuplestore; + if (TRIGGER_FIRED_BY_INSERT(evtshared->ats_event)) + LocTriggerData.tg_newtable = evtshared->ats_table->new_ins_tuplestore; + else + LocTriggerData.tg_newtable = evtshared->ats_table->new_upd_tuplestore; evtshared->ats_table->closed = true; } } @@ -4637,8 +4667,10 @@ TransitionCaptureState * MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) { TransitionCaptureState *state; - bool need_old, - need_new; + bool need_old_upd, + need_new_upd, + need_old_del, + need_new_ins; AfterTriggersTableData *table; MemoryContext oldcxt; ResourceOwner saveResourceOwner; @@ -4650,23 +4682,31 @@ MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) switch (cmdType) { case CMD_INSERT: - need_old = false; - need_new = trigdesc->trig_insert_new_table; + need_old_upd = need_old_del = need_new_upd = false; + need_new_ins = trigdesc->trig_insert_new_table; break; case CMD_UPDATE: - need_old = trigdesc->trig_update_old_table; - need_new = trigdesc->trig_update_new_table; + need_old_upd = trigdesc->trig_update_old_table; + need_new_upd = trigdesc->trig_update_new_table; + need_old_del = need_new_ins = false; break; case CMD_DELETE: - need_old = trigdesc->trig_delete_old_table; - need_new = false; + need_old_del = trigdesc->trig_delete_old_table; + need_old_upd = need_new_upd = need_new_ins = false; + break; + case CMD_MERGE: + need_old_upd = trigdesc->trig_update_old_table; + need_new_upd = trigdesc->trig_update_new_table; + need_old_del = trigdesc->trig_delete_old_table; + need_new_ins = trigdesc->trig_insert_new_table; break; default: elog(ERROR, "unexpected CmdType: %d", (int) cmdType); - need_old = need_new = false; /* keep compiler quiet */ + /* keep compiler quiet */ + need_old_upd = need_new_upd = need_old_del = need_new_ins = false; break; } - if (!need_old && !need_new) + if (!need_old_upd && !need_new_upd && !need_new_ins && !need_old_del) return NULL; /* Check state, like AfterTriggerSaveEvent. */ @@ -4696,10 +4736,14 @@ MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) saveResourceOwner = CurrentResourceOwner; CurrentResourceOwner = CurTransactionResourceOwner; - if (need_old && table->old_tuplestore == NULL) - table->old_tuplestore = tuplestore_begin_heap(false, false, work_mem); - if (need_new && table->new_tuplestore == NULL) - table->new_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_old_upd && table->old_upd_tuplestore == NULL) + table->old_upd_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_new_upd && table->new_upd_tuplestore == NULL) + table->new_upd_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_old_del && table->old_del_tuplestore == NULL) + table->old_del_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_new_ins && table->new_ins_tuplestore == NULL) + table->new_ins_tuplestore = tuplestore_begin_heap(false, false, work_mem); CurrentResourceOwner = saveResourceOwner; MemoryContextSwitchTo(oldcxt); @@ -4888,12 +4932,20 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs) { AfterTriggersTableData *table = (AfterTriggersTableData *) lfirst(lc); - ts = table->old_tuplestore; - table->old_tuplestore = NULL; + ts = table->old_upd_tuplestore; + table->old_upd_tuplestore = NULL; if (ts) tuplestore_end(ts); - ts = table->new_tuplestore; - table->new_tuplestore = NULL; + ts = table->new_upd_tuplestore; + table->new_upd_tuplestore = NULL; + if (ts) + tuplestore_end(ts); + ts = table->old_del_tuplestore; + table->old_del_tuplestore = NULL; + if (ts) + tuplestore_end(ts); + ts = table->new_ins_tuplestore; + table->new_ins_tuplestore = NULL; if (ts) tuplestore_end(ts); } @@ -5744,12 +5796,11 @@ AfterTriggerSaveEvent(EState *estate, ResultRelInfo *relinfo, newtup == NULL)); if (oldtup != NULL && - ((event == TRIGGER_EVENT_DELETE && delete_old_table) || - (event == TRIGGER_EVENT_UPDATE && update_old_table))) + (event == TRIGGER_EVENT_DELETE && delete_old_table)) { Tuplestorestate *old_tuplestore; - old_tuplestore = transition_capture->tcs_private->old_tuplestore; + old_tuplestore = transition_capture->tcs_private->old_del_tuplestore; if (map != NULL) { @@ -5761,13 +5812,48 @@ AfterTriggerSaveEvent(EState *estate, ResultRelInfo *relinfo, else tuplestore_puttuple(old_tuplestore, oldtup); } + if (oldtup != NULL && + (event == TRIGGER_EVENT_UPDATE && update_old_table)) + { + Tuplestorestate *old_tuplestore; + + old_tuplestore = transition_capture->tcs_private->old_upd_tuplestore; + + if (map != NULL) + { + HeapTuple converted = do_convert_tuple(oldtup, map); + + tuplestore_puttuple(old_tuplestore, converted); + pfree(converted); + } + else + tuplestore_puttuple(old_tuplestore, oldtup); + } + if (newtup != NULL && + (event == TRIGGER_EVENT_INSERT && insert_new_table)) + { + Tuplestorestate *new_tuplestore; + + new_tuplestore = transition_capture->tcs_private->new_ins_tuplestore; + + if (original_insert_tuple != NULL) + tuplestore_puttuple(new_tuplestore, original_insert_tuple); + else if (map != NULL) + { + HeapTuple converted = do_convert_tuple(newtup, map); + + tuplestore_puttuple(new_tuplestore, converted); + pfree(converted); + } + else + tuplestore_puttuple(new_tuplestore, newtup); + } if (newtup != NULL && - ((event == TRIGGER_EVENT_INSERT && insert_new_table) || - (event == TRIGGER_EVENT_UPDATE && update_new_table))) + (event == TRIGGER_EVENT_UPDATE && update_new_table)) { Tuplestorestate *new_tuplestore; - new_tuplestore = transition_capture->tcs_private->new_tuplestore; + new_tuplestore = transition_capture->tcs_private->new_upd_tuplestore; if (original_insert_tuple != NULL) tuplestore_puttuple(new_tuplestore, original_insert_tuple); diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefile index cc09895fa5..68675f9796 100644 --- a/src/backend/executor/Makefile +++ b/src/backend/executor/Makefile @@ -22,7 +22,7 @@ OBJS = execAmi.o execCurrent.o execExpr.o execExprInterp.o \ nodeCustom.o nodeFunctionscan.o nodeGather.o \ nodeHash.o nodeHashjoin.o nodeIndexscan.o nodeIndexonlyscan.o \ nodeLimit.o nodeLockRows.o nodeGatherMerge.o \ - nodeMaterial.o nodeMergeAppend.o nodeMergejoin.o nodeModifyTable.o \ + nodeMaterial.o nodeMergeAppend.o nodeMergejoin.o nodeMerge.o nodeModifyTable.o \ nodeNestloop.o nodeProjectSet.o nodeRecursiveunion.o nodeResult.o \ nodeSamplescan.o nodeSeqscan.o nodeSetOp.o nodeSort.o nodeUnique.o \ nodeValuesscan.o \ diff --git a/src/backend/executor/README b/src/backend/executor/README index 0d7cd552eb..05769772b7 100644 --- a/src/backend/executor/README +++ b/src/backend/executor/README @@ -37,6 +37,16 @@ the plan tree returns the computed tuples to be updated, plus a "junk" one. For DELETE, the plan tree need only deliver a CTID column, and the ModifyTable node visits each of those rows and marks the row deleted. +MERGE runs one generic plan that returns candidate target rows. Each row +consists of a super-row that contains all the columns needed by any of the +individual actions, plus a CTID and a TABLEOID junk columns. The CTID column is +required to know if a matching target row was found or not and the TABLEOID +column is needed to find the underlying target partition, in case when the +target table is a partition table. If the CTID column is set we attempt to +activate WHEN MATCHED actions, or if it is NULL then we will attempt to +activate WHEN NOT MATCHED actions. Once we know which action is activated we +form the final result row and apply only those changes. + XXX a great deal more documentation needs to be written here... diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 68f6450ee6..11e1d11ef3 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -233,6 +233,7 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags) case CMD_INSERT: case CMD_DELETE: case CMD_UPDATE: + case CMD_MERGE: estate->es_output_cid = GetCurrentCommandId(true); break; @@ -1357,6 +1358,9 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_onConflictArbiterIndexes = NIL; resultRelInfo->ri_onConflict = NULL; + resultRelInfo->ri_mergeTargetRTI = 0; + resultRelInfo->ri_mergeState = (MergeState *) palloc0(sizeof (MergeState)); + /* * Partition constraint, which also includes the partition constraint of * all the ancestors that are partitions. Note that it will be checked @@ -2205,6 +2209,19 @@ ExecWithCheckOptions(WCOKind kind, ResultRelInfo *resultRelInfo, errmsg("new row violates row-level security policy for table \"%s\"", wco->relname))); break; + case WCO_RLS_MERGE_UPDATE_CHECK: + case WCO_RLS_MERGE_DELETE_CHECK: + if (wco->polname != NULL) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("target row violates row-level security policy \"%s\" (USING expression) for table \"%s\"", + wco->polname, wco->relname))); + else + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("target row violates row-level security policy (USING expression) for table \"%s\"", + wco->relname))); + break; case WCO_RLS_CONFLICT_CHECK: if (wco->polname != NULL) ereport(ERROR, diff --git a/src/backend/executor/execPartition.c b/src/backend/executor/execPartition.c index 9a13188649..a6a7885abd 100644 --- a/src/backend/executor/execPartition.c +++ b/src/backend/executor/execPartition.c @@ -67,6 +67,8 @@ ExecSetupPartitionTupleRouting(ModifyTableState *mtstate, Relation rel) ResultRelInfo *update_rri = NULL; int num_update_rri = 0, update_rri_index = 0; + bool is_update = false; + bool is_merge = false; PartitionTupleRouting *proute; int nparts; ModifyTable *node = mtstate ? (ModifyTable *) mtstate->ps.plan : NULL; @@ -89,13 +91,22 @@ ExecSetupPartitionTupleRouting(ModifyTableState *mtstate, Relation rel) /* Set up details specific to the type of tuple routing we are doing. */ if (node && node->operation == CMD_UPDATE) + is_update = true; + else if (node && node->operation == CMD_MERGE) + is_merge = true; + + if (is_update) { update_rri = mtstate->resultRelInfo; num_update_rri = list_length(node->plans); proute->subplan_partition_offsets = palloc(num_update_rri * sizeof(int)); proute->num_subplan_partition_offsets = num_update_rri; + } + + if (is_update || is_merge) + { /* * We need an additional tuple slot for storing transient tuples that * are converted to the root table descriptor. @@ -299,6 +310,25 @@ ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, return result; } +/* + * Given OID of the partition leaf, return the index of the leaf in the + * partition hierarchy. + */ +int +ExecFindPartitionByOid(PartitionTupleRouting *proute, Oid partoid) +{ + int i; + + for (i = 0; i < proute->num_partitions; i++) + { + if (proute->partition_oids[i] == partoid) + break; + } + + Assert(i < proute->num_partitions); + return i; +} + /* * ExecInitPartitionInfo * Initialize ResultRelInfo and other information for a partition if not @@ -337,6 +367,8 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, rootrel, estate->es_instrument); + leaf_part_rri->ri_PartitionLeafIndex = partidx; + /* * Verify result relation is a valid target for an INSERT. An UPDATE of a * partition-key becomes a DELETE+INSERT operation, so this check is still @@ -625,6 +657,90 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, Assert(proute->partitions[partidx] == NULL); proute->partitions[partidx] = leaf_part_rri; + /* + * Initialize information about this partition that's needed to handle + * MERGE. + */ + if (node && node->operation == CMD_MERGE) + { + TupleDesc partrelDesc = RelationGetDescr(partrel); + TupleConversionMap *map = proute->parent_child_tupconv_maps[partidx]; + int firstVarno = mtstate->resultRelInfo[0].ri_RangeTableIndex; + Relation firstResultRel = mtstate->resultRelInfo[0].ri_RelationDesc; + + /* + * If the root parent and partition have the same tuple + * descriptor, just reuse the original MERGE state for partition. + */ + if (map == NULL) + { + leaf_part_rri->ri_mergeState = resultRelInfo->ri_mergeState; + } + else + { + /* Convert expressions contain partition's attnos. */ + List *conv_tl, *conv_qual; + ListCell *l; + List *matchedActionStates = NIL; + List *notMatchedActionStates = NIL; + + foreach (l, node->mergeActionList) + { + MergeAction *action = lfirst_node(MergeAction, l); + MergeActionState *action_state = makeNode(MergeActionState); + TupleDesc tupDesc; + ExprContext *econtext; + + action_state->matched = action->matched; + action_state->commandType = action->commandType; + + conv_qual = (List *) action->qual; + conv_qual = map_partition_varattnos(conv_qual, + firstVarno, partrel, + firstResultRel, NULL); + + action_state->whenqual = ExecInitQual(conv_qual, &mtstate->ps); + + conv_tl = (List *) action->targetList; + conv_tl = map_partition_varattnos(conv_tl, + firstVarno, partrel, + firstResultRel, NULL); + + conv_tl = adjust_partition_tlist( conv_tl, map); + + tupDesc = ExecTypeFromTL(conv_tl, partrelDesc->tdhasoid); + action_state->tupDesc = tupDesc; + + /* build action projection state */ + econtext = mtstate->ps.ps_ExprContext; + action_state->proj = + ExecBuildProjectionInfo(conv_tl, econtext, + mtstate->mt_mergeproj, + &mtstate->ps, + partrelDesc); + + if (action_state->matched) + matchedActionStates = + lappend(matchedActionStates, action_state); + else + notMatchedActionStates = + lappend(notMatchedActionStates, action_state); + } + leaf_part_rri->ri_mergeState->matchedActionStates = + matchedActionStates; + leaf_part_rri->ri_mergeState->notMatchedActionStates = + notMatchedActionStates; + } + + /* + * get_partition_dispatch_recurse() and expand_partitioned_rtentry() + * fetch the leaf OIDs in the same order. So we can safely derive the + * index of the merge target relation corresponding to this partition + * by simply adding partidx + 1 to the root's merge target relation. + */ + leaf_part_rri->ri_mergeTargetRTI = node->mergeTargetRelation + + partidx + 1; + } MemoryContextSwitchTo(oldContext); return leaf_part_rri; diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 32891abbdf..971f92a938 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -454,7 +454,7 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate, { slot = ExecBRUpdateTriggers(estate, epqstate, resultRelInfo, &searchslot->tts_tuple->t_self, - NULL, slot); + NULL, slot, NULL); if (slot == NULL) /* "do nothing" */ skip_tuple = true; @@ -515,7 +515,7 @@ ExecSimpleRelationDelete(EState *estate, EPQState *epqstate, { skip_tuple = !ExecBRDeleteTriggers(estate, epqstate, resultRelInfo, &searchslot->tts_tuple->t_self, - NULL); + NULL, NULL); } if (!skip_tuple) diff --git a/src/backend/executor/nodeMerge.c b/src/backend/executor/nodeMerge.c new file mode 100644 index 0000000000..e633af7704 --- /dev/null +++ b/src/backend/executor/nodeMerge.c @@ -0,0 +1,566 @@ +/*------------------------------------------------------------------------- + * + * nodeMerge.c + * routines to handle Merge nodes. + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/executor/nodeMerge.c + * + *------------------------------------------------------------------------- + */ + + +#include "postgres.h" + +#include "access/htup_details.h" +#include "access/xact.h" +#include "commands/trigger.h" +#include "executor/execPartition.h" +#include "executor/executor.h" +#include "executor/nodeModifyTable.h" +#include "executor/nodeMerge.h" +#include "miscadmin.h" +#include "nodes/nodeFuncs.h" +#include "storage/bufmgr.h" +#include "storage/lmgr.h" +#include "utils/builtins.h" +#include "utils/memutils.h" +#include "utils/rel.h" +#include "utils/tqual.h" + + +/* + * Check and execute the first qualifying MATCHED action. The current target + * tuple is identified by tupleid. + * + * We start from the first WHEN MATCHED action and check if the additional WHEN + * quals pass. If the additional quals for the first action do not pass, we + * check the second, then the third and so on. If we reach to the end, no + * action is taken and we return true, indicating that no further action is + * required for this tuple. + * + * If we do find a qualifying action, then we attempt to execute the action. In + * case the tuple is concurrently updated, EvalPlanQual is run with the updated + * tuple to recheck the join quals. Note that the additional quals associated + * with individual actions are evaluated separately by the MERGE code, while + * EvalPlanQual checks for the join quals. If EvalPlanQual tells us that the + * updated tuple still passes the join quals, then we restart from the top and + * again look for a qualifying action. Otherwise, we return false indicating + * that a NOT MATCHED action must now be executed for the current source tuple. + */ +static bool +ExecMergeMatched(ModifyTableState *mtstate, EState *estate, + TupleTableSlot *slot, JunkFilter *junkfilter, + ItemPointer tupleid) +{ + ExprContext *econtext = mtstate->ps.ps_ExprContext; + bool isNull; + Datum datum; + Oid tableoid = InvalidOid; + List *mergeMatchedActionStates = NIL; + HeapUpdateFailureData hufd; + bool tuple_updated, + tuple_deleted; + Buffer buffer; + HeapTupleData tuple; + EPQState *epqstate = &mtstate->mt_epqstate; + ResultRelInfo *saved_resultRelInfo; + ResultRelInfo *resultRelInfo = estate->es_result_relation_info; + ListCell *l; + TupleTableSlot *saved_slot = slot; + + + /* + * We always fetch the tableoid while performing MATCHED MERGE action. + * This is strictly not required if the target table is not a partitioned + * table. But we are not yet optimising for that case. + */ + datum = ExecGetJunkAttribute(slot, junkfilter->jf_otherJunkAttNo, + &isNull); + Assert(!isNull); + tableoid = DatumGetObjectId(datum); + + if (mtstate->mt_partition_tuple_routing) + { + int leaf_part_index; + PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing; + + /* + * If we're dealing with a MATCHED tuple, then tableoid must have been + * set correctly. In case of partitioned table, we must now fetch the + * correct result relation corresponding to the child table emitting + * the matching target row. For normal table, there is just one result + * relation and it must be the one emitting the matching row. + */ + leaf_part_index = ExecFindPartitionByOid(proute, tableoid); + + resultRelInfo = proute->partitions[leaf_part_index]; + if (resultRelInfo == NULL) + { + resultRelInfo = ExecInitPartitionInfo(mtstate, + mtstate->resultRelInfo, + proute, estate, leaf_part_index); + Assert(resultRelInfo != NULL); + } + } + + /* + * Save the current information and work with the correct result relation. + */ + saved_resultRelInfo = resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + + /* + * And get the correct action lists. + */ + mergeMatchedActionStates = + resultRelInfo->ri_mergeState->matchedActionStates; + + /* + * If there are not WHEN MATCHED actions, we are done. + */ + if (mergeMatchedActionStates == NIL) + return true; + + /* + * Make tuple and any needed join variables available to ExecQual and + * ExecProject. The target's existing tuple is installed in the scantuple. + * Again, this target relation's slot is required only in the case of a + * MATCHED tuple and UPDATE/DELETE actions. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_existing, + resultRelInfo->ri_RelationDesc->rd_att); + econtext->ecxt_scantuple = mtstate->mt_existing; + econtext->ecxt_innertuple = slot; + econtext->ecxt_outertuple = NULL; + +lmerge_matched:; + slot = saved_slot; + buffer = InvalidBuffer; + + /* + * UPDATE/DELETE is only invoked for matched rows. And we must have found + * the tupleid of the target row in that case. We fetch using SnapshotAny + * because we might get called again after EvalPlanQual returns us a new + * tuple. This tuple may not be visible to our MVCC snapshot. + */ + Assert(tupleid != NULL); + + tuple.t_self = *tupleid; + if (!heap_fetch(resultRelInfo->ri_RelationDesc, SnapshotAny, &tuple, + &buffer, true, NULL)) + elog(ERROR, "Failed to fetch the target tuple"); + + /* Store target's existing tuple in the state's dedicated slot */ + ExecStoreTuple(&tuple, mtstate->mt_existing, buffer, false); + + foreach(l, mergeMatchedActionStates) + { + MergeActionState *action = (MergeActionState *) lfirst(l); + + /* + * Test condition, if any + * + * In the absence of a condition we perform the action unconditionally + * (no need to check separately since ExecQual() will return true if + * there are no conditions to evaluate). + */ + if (!ExecQual(action->whenqual, econtext)) + continue; + + /* + * Check if the existing target tuple meet the USING checks of + * UPDATE/DELETE RLS policies. If those checks fail, we throw an + * error. + * + * The WITH CHECK quals are applied in ExecUpdate() and hence we need + * not do anything special to handle them. + * + * NOTE: We must do this after WHEN quals are evaluated so that we + * check policies only when they matter. + */ + if (resultRelInfo->ri_WithCheckOptions) + { + ExecWithCheckOptions(action->commandType == CMD_UPDATE ? + WCO_RLS_MERGE_UPDATE_CHECK : WCO_RLS_MERGE_DELETE_CHECK, + resultRelInfo, + mtstate->mt_existing, + mtstate->ps.state); + } + + /* Perform stated action */ + switch (action->commandType) + { + case CMD_UPDATE: + + /* + * We set up the projection earlier, so all we do here is + * Project, no need for any other tasks prior to the + * ExecUpdate. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_mergeproj, action->tupDesc); + ExecProject(action->proj); + + /* + * We don't call ExecFilterJunk() because the projected tuple + * using the UPDATE action's targetlist doesn't have a junk + * attribute. + */ + slot = ExecUpdate(mtstate, tupleid, NULL, + mtstate->mt_mergeproj, + slot, epqstate, estate, + &tuple_updated, &hufd, + action, mtstate->canSetTag); + break; + + case CMD_DELETE: + /* Nothing to Project for a DELETE action */ + slot = ExecDelete(mtstate, tupleid, NULL, + slot, epqstate, estate, + &tuple_deleted, false, &hufd, action, + mtstate->canSetTag); + + break; + + default: + elog(ERROR, "unknown action in MERGE WHEN MATCHED clause"); + + } + + /* + * Check for any concurrent update/delete operation which may have + * prevented our update/delete. We also check for situations where we + * might be trying to update/delete the same tuple twice. + */ + if ((action->commandType == CMD_UPDATE && !tuple_updated) || + (action->commandType == CMD_DELETE && !tuple_deleted)) + + { + switch (hufd.result) + { + case HeapTupleMayBeUpdated: + break; + case HeapTupleInvisible: + + /* + * This state should never be reached since the underlying + * JOIN runs with a MVCC snapshot and should only return + * rows visible to us. + */ + elog(ERROR, "unexpected invisible tuple"); + break; + + case HeapTupleSelfUpdated: + + /* + * SQLStandard disallows this for MERGE. + */ + if (TransactionIdIsCurrentTransactionId(hufd.xmax)) + ereport(ERROR, + (errcode(ERRCODE_CARDINALITY_VIOLATION), + errmsg("MERGE command cannot affect row a second time"), + errhint("Ensure that not more than one source rows match any one target row"))); + /* This shouldn't happen */ + elog(ERROR, "attempted to update or delete invisible tuple"); + break; + + case HeapTupleUpdated: + + /* + * The target tuple was concurrently updated/deleted by + * some other transaction. + * + * If the current tuple is that last tuple in the update + * chain, then we know that the tuple was concurrently + * deleted. Just return and let the caller try NOT MATCHED + * actions. + * + * If the current tuple was concurrently updated, then we + * must run the EvalPlanQual() with the new version of the + * tuple. If EvalPlanQual() does not return a tuple (can + * that happen?), then again we switch to NOT MATCHED + * action. If it does return a tuple and the join qual is + * still satified, then we just need to recheck the + * MATCHED actions, starting from the top, and execute the + * first qualifying action. + */ + if (!ItemPointerEquals(tupleid, &hufd.ctid)) + { + TupleTableSlot *epqslot; + + /* + * Since we generate a JOIN query with a target table + * RTE different than the result relation RTE, we must + * pass in the RTI of the relation used in the join + * query and not the one from result relation. + */ + Assert(resultRelInfo->ri_mergeTargetRTI > 0); + epqslot = EvalPlanQual(estate, + epqstate, + resultRelInfo->ri_RelationDesc, + GetEPQRangeTableIndex(resultRelInfo), + LockTupleExclusive, + &hufd.ctid, + hufd.xmax); + + if (!TupIsNull(epqslot)) + { + (void) ExecGetJunkAttribute(epqslot, + resultRelInfo->ri_junkFilter->jf_junkAttNo, + &isNull); + + /* + * A valid ctid means that we are still dealing + * with MATCHED case. But we must retry from the + * start with the updated tuple to ensure that the + * first qualifying WHEN MATCHED action is + * executed. + * + * We don't use the new slot returned by + * EvalPlanQual because we anyways re-install the + * new target tuple in econtext->ecxt_scantuple + * before re-evaluating WHEN AND conditions and + * re-projecting the update targetlists. The + * source side tuple does not change and hence we + * can safely continue to use the old slot. + */ + if (!isNull) + { + /* + * Must update *tupleid to the TID of the + * newer tuple found in the update chain. + */ + *tupleid = hufd.ctid; + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + goto lmerge_matched; + } + } + } + + /* + * Tell the caller about the updated TID, restore the + * state back and return. + */ + *tupleid = hufd.ctid; + estate->es_result_relation_info = saved_resultRelInfo; + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + return false; + + default: + break; + + } + } + + if (action->commandType == CMD_UPDATE && tuple_updated) + InstrCountFiltered1(&mtstate->ps, 1); + if (action->commandType == CMD_DELETE && tuple_deleted) + InstrCountFiltered2(&mtstate->ps, 1); + + /* + * We've activated one of the WHEN clauses, so we don't search + * further. This is required behaviour, not an optimisation. + */ + estate->es_result_relation_info = saved_resultRelInfo; + break; + } + + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + + /* + * Successfully executed an action or no qualifying action was found. + */ + return true; +} + +/* + * Execute the first qualifying NOT MATCHED action. + */ +static void +ExecMergeNotMatched(ModifyTableState *mtstate, EState *estate, + TupleTableSlot *slot) +{ + PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing; + ExprContext *econtext = mtstate->ps.ps_ExprContext; + List *mergeNotMatchedActionStates = NIL; + ResultRelInfo *resultRelInfo; + ListCell *l; + TupleTableSlot *myslot; + + /* + * We are dealing with NOT MATCHED tuple. Since for MERGE, partition tree + * is not expanded for the result relation, we continue to work with the + * currently active result relation, which should be of the root of the + * partition tree. + */ + resultRelInfo = mtstate->resultRelInfo; + + /* + * For INSERT actions, root relation's merge action is OK since the the + * INSERT's targetlist and the WHEN conditions can only refer to the + * source relation and hence it does not matter which result relation we + * work with. + */ + mergeNotMatchedActionStates = + resultRelInfo->ri_mergeState->notMatchedActionStates; + + /* + * Make source tuple available to ExecQual and ExecProject. We don't need + * the target tuple since the WHEN quals and the targetlist can't refer to + * the target columns. + */ + econtext->ecxt_scantuple = NULL; + econtext->ecxt_innertuple = slot; + econtext->ecxt_outertuple = NULL; + + foreach(l, mergeNotMatchedActionStates) + { + MergeActionState *action = (MergeActionState *) lfirst(l); + + /* + * Test condition, if any + * + * In the absence of a condition we perform the action unconditionally + * (no need to check separately since ExecQual() will return true if + * there are no conditions to evaluate). + */ + if (!ExecQual(action->whenqual, econtext)) + continue; + + /* Perform stated action */ + switch (action->commandType) + { + case CMD_INSERT: + + /* + * We set up the projection earlier, so all we do here is + * Project, no need for any other tasks prior to the + * ExecInsert. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_mergeproj, action->tupDesc); + ExecProject(action->proj); + + /* + * ExecPrepareTupleRouting may modify the passed-in slot. Hence + * pass a local reference so that action->slot is not modified. + */ + myslot = mtstate->mt_mergeproj; + + /* Prepare for tuple routing if needed. */ + if (proute) + myslot = ExecPrepareTupleRouting(mtstate, estate, proute, + resultRelInfo, myslot); + slot = ExecInsert(mtstate, myslot, slot, + estate, action, + mtstate->canSetTag); + /* Revert ExecPrepareTupleRouting's state change. */ + if (proute) + estate->es_result_relation_info = resultRelInfo; + break; + case CMD_NOTHING: + /* Do Nothing */ + break; + default: + elog(ERROR, "unknown action in MERGE WHEN NOT MATCHED clause"); + } + + break; + } +} + +/* + * Perform MERGE. + */ +void +ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, + JunkFilter *junkfilter, ResultRelInfo *resultRelInfo) +{ + ItemPointer tupleid; + ItemPointerData tuple_ctid; + bool matched = false; + char relkind; + Datum datum; + bool isNull; + + relkind = resultRelInfo->ri_RelationDesc->rd_rel->relkind; + Assert(relkind == RELKIND_RELATION || + relkind == RELKIND_MATVIEW || + relkind == RELKIND_PARTITIONED_TABLE); + + /* + * We run a JOIN between the target relation and the source relation to + * find a set of candidate source rows that has matching row in the target + * table and a set of candidate source rows that does not have matching + * row in the target table. If the join returns us a tuple with target + * relation's tid set, that implies that the join found a matching row for + * the given source tuple. This case triggers the WHEN MATCHED clause of + * the MERGE. Whereas a NULL in the target relation's ctid column + * indicates a NOT MATCHED case. + */ + datum = ExecGetJunkAttribute(slot, junkfilter->jf_junkAttNo, &isNull); + + if (!isNull) + { + matched = true; + tupleid = (ItemPointer) DatumGetPointer(datum); + tuple_ctid = *tupleid; /* be sure we don't free ctid!! */ + tupleid = &tuple_ctid; + } + else + { + matched = false; + tupleid = NULL; /* we don't need it for INSERT actions */ + } + + /* + * If we are dealing with a WHEN MATCHED case, we look at the given WHEN + * MATCHED actions in an order and execute the first action which also + * satisfies the additional WHEN MATCHED AND quals. If an action without + * any additional quals is found, that action is executed. + * + * Similarly, if we are dealing with WHEN NOT MATCHED case, we look at the + * given WHEN NOT MATCHED actions in an ordr and execute the first + * qualifying action. + * + * Things get interesting in case of concurrent update/delete of the + * target tuple. Such concurrent update/delete is detected while we are + * executing a WHEN MATCHED action. + * + * A concurrent update for example can: + * + * 1. modify the target tuple so that it no longer satisfies the + * additional quals attached to the current WHEN MATCHED action OR 2. + * modify the target tuple so that the join quals no longer pass and hence + * the source tuple no longer has a match. + * + * In the first case, we are still dealing with a WHEN MATCHED case, but + * we should recheck the list of WHEN MATCHED actions and choose the first + * one that satisfies the new target tuple. In the second case, since the + * source tuple no longer matches the target tuple, we now instead find a + * qualifying WHEN NOT MATCHED action and execute that. + * + * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. + * + * ExecMergeMatched takes care of following the update chain and + * re-finding the qualifying WHEN MATCHED action, as long as the updated + * target tuple still satisfies the join quals i.e. it still remains a + * WHEN MATCHED case. If the tuple gets deleted or the join quals fail, it + * returns and we try ExecMergeNotMatched. Given that ExecMergeMatched + * always make progress by following the update chain and we never switch + * from ExecMergeNotMatched to ExecMergeMatched, there is no risk of a + * livelock. + */ + if (!matched || + !ExecMergeMatched(mtstate, estate, slot, junkfilter, tupleid)) + ExecMergeNotMatched(mtstate, estate, slot); +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 1b09868ff8..def7eec223 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -42,6 +42,7 @@ #include "commands/trigger.h" #include "executor/execPartition.h" #include "executor/executor.h" +#include "executor/nodeMerge.h" #include "executor/nodeModifyTable.h" #include "foreign/fdwapi.h" #include "miscadmin.h" @@ -62,17 +63,17 @@ static bool ExecOnConflictUpdate(ModifyTableState *mtstate, EState *estate, bool canSetTag, TupleTableSlot **returning); -static TupleTableSlot *ExecPrepareTupleRouting(ModifyTableState *mtstate, - EState *estate, - PartitionTupleRouting *proute, - ResultRelInfo *targetRelInfo, - TupleTableSlot *slot); static ResultRelInfo *getTargetResultRelInfo(ModifyTableState *node); static void ExecSetupChildParentMapForTcs(ModifyTableState *mtstate); static void ExecSetupChildParentMapForSubplan(ModifyTableState *mtstate); static TupleConversionMap *tupconv_map_for_subplan(ModifyTableState *node, int whichplan); +/* flags for mt_merge_subcommands */ +#define MERGE_INSERT 0x01 +#define MERGE_UPDATE 0x02 +#define MERGE_DELETE 0x04 + /* * Verify that the tuples to be produced by INSERT or UPDATE match the * target relation's rowtype @@ -259,11 +260,12 @@ ExecCheckTIDVisible(EState *estate, * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +extern TupleTableSlot * ExecInsert(ModifyTableState *mtstate, TupleTableSlot *slot, TupleTableSlot *planSlot, EState *estate, + MergeActionState *actionState, bool canSetTag) { HeapTuple tuple; @@ -390,9 +392,17 @@ ExecInsert(ModifyTableState *mtstate, * partition, we should instead check UPDATE policies, because we are * executing policies defined on the target table, and not those * defined on the child partitions. + * + * If we're running MERGE, we refer to the action that we're executing + * to know if we're doing an INSERT or UPDATE to a partition table. */ - wco_kind = (mtstate->operation == CMD_UPDATE) ? - WCO_RLS_UPDATE_CHECK : WCO_RLS_INSERT_CHECK; + if (mtstate->operation == CMD_UPDATE) + wco_kind = WCO_RLS_UPDATE_CHECK; + else if (mtstate->operation == CMD_INSERT) + wco_kind = WCO_RLS_INSERT_CHECK; + else if (mtstate->operation == CMD_MERGE) + wco_kind = (actionState->commandType == CMD_UPDATE) ? + WCO_RLS_UPDATE_CHECK : WCO_RLS_INSERT_CHECK; /* * ExecWithCheckOptions() will skip any WCOs which are not of the kind @@ -617,10 +627,19 @@ ExecInsert(ModifyTableState *mtstate, * passed to foreign table triggers; it is NULL when the foreign * table has no relevant triggers. * + * MERGE passes actionState of the action it's currently executing; + * regular DELETE passes NULL. This is used by ExecDelete to know if it's + * being called from MERGE or regular DELETE operation. + * + * If the DELETE fails because the tuple is concurrently updated/deleted + * by this or some other transaction, hufdp is filled with the reason as + * well as other important information. Currently only MERGE needs this + * information. + * * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +TupleTableSlot * ExecDelete(ModifyTableState *mtstate, ItemPointer tupleid, HeapTuple oldtuple, @@ -629,6 +648,8 @@ ExecDelete(ModifyTableState *mtstate, EState *estate, bool *tupleDeleted, bool processReturning, + HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag) { ResultRelInfo *resultRelInfo; @@ -641,6 +662,14 @@ ExecDelete(ModifyTableState *mtstate, if (tupleDeleted) *tupleDeleted = false; + /* + * Initialize hufdp. Since the caller is only interested in the failure + * status, initialize with the state that is used to indicate successful + * operation. + */ + if (hufdp) + hufdp->result = HeapTupleMayBeUpdated; + /* * get information on the (current) result relation */ @@ -654,7 +683,7 @@ ExecDelete(ModifyTableState *mtstate, bool dodelete; dodelete = ExecBRDeleteTriggers(estate, epqstate, resultRelInfo, - tupleid, oldtuple); + tupleid, oldtuple, hufdp); if (!dodelete) /* "do nothing" */ return NULL; @@ -721,6 +750,15 @@ ldelete:; estate->es_crosscheck_snapshot, true /* wait for commit */ , &hufd); + + /* + * Copy the necessary information, if the caller has asked for it. We + * must do this irrespective of whether the tuple was updated or + * deleted. + */ + if (hufdp) + *hufdp = hufd; + switch (result) { case HeapTupleSelfUpdated: @@ -755,7 +793,11 @@ ldelete:; errmsg("tuple to be updated was already modified by an operation triggered by the current command"), errhint("Consider using an AFTER trigger instead of a BEFORE trigger to propagate changes to other rows."))); - /* Else, already deleted by self; nothing to do */ + /* + * Else, already deleted by self; nothing to do but inform + * MERGE about it anyways so that it can take necessary + * action. + */ return NULL; case HeapTupleMayBeUpdated: @@ -766,14 +808,24 @@ ldelete:; ereport(ERROR, (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE), errmsg("could not serialize access due to concurrent update"))); + if (!ItemPointerEquals(tupleid, &hufd.ctid)) { TupleTableSlot *epqslot; + /* + * If we're executing MERGE, then the onus of running + * EvalPlanQual() and handling its outcome lies with the + * caller. + */ + if (actionState != NULL) + return NULL; + + /* Normal DELETE path. */ epqslot = EvalPlanQual(estate, epqstate, resultRelationDesc, - resultRelInfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(resultRelInfo), LockTupleExclusive, &hufd.ctid, hufd.xmax); @@ -783,7 +835,12 @@ ldelete:; goto ldelete; } } - /* tuple already deleted; nothing to do */ + + /* + * tuple already deleted; nothing to do. But MERGE might want + * to handle it differently. We've already filled-in hufdp + * with sufficient information for MERGE to look at. + */ return NULL; default: @@ -911,10 +968,21 @@ ldelete:; * foreign table triggers; it is NULL when the foreign table has * no relevant triggers. * + * MERGE passes actionState of the action it's currently executing; + * regular UPDATE passes NULL. This is used by ExecUpdate to know if it's + * being called from MERGE or regular UPDATE operation. ExecUpdate may + * pass this information to ExecInsert if it ends up running DELETE+INSERT + * for partition key updates. + * + * If the UPDATE fails because the tuple is concurrently updated/deleted + * by this or some other transaction, hufdp is filled with the reason as + * well as other important information. Currently only MERGE needs this + * information. + * * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +extern TupleTableSlot * ExecUpdate(ModifyTableState *mtstate, ItemPointer tupleid, HeapTuple oldtuple, @@ -922,6 +990,9 @@ ExecUpdate(ModifyTableState *mtstate, TupleTableSlot *planSlot, EPQState *epqstate, EState *estate, + bool *tuple_updated, + HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag) { HeapTuple tuple; @@ -938,6 +1009,17 @@ ExecUpdate(ModifyTableState *mtstate, if (IsBootstrapProcessingMode()) elog(ERROR, "cannot UPDATE during bootstrap"); + if (tuple_updated) + *tuple_updated = false; + + /* + * Initialize hufdp. Since the caller is only interested in the failure + * status, initialize with the state that is used to indicate successful + * operation. + */ + if (hufdp) + hufdp->result = HeapTupleMayBeUpdated; + /* * get the heap tuple out of the tuple table slot, making sure we have a * writable copy @@ -955,7 +1037,7 @@ ExecUpdate(ModifyTableState *mtstate, resultRelInfo->ri_TrigDesc->trig_update_before_row) { slot = ExecBRUpdateTriggers(estate, epqstate, resultRelInfo, - tupleid, oldtuple, slot); + tupleid, oldtuple, slot, hufdp); if (slot == NULL) /* "do nothing" */ return NULL; @@ -1079,8 +1161,9 @@ lreplace:; * Row movement, part 1. Delete the tuple, but skip RETURNING * processing. We want to return rows from INSERT. */ - ExecDelete(mtstate, tupleid, oldtuple, planSlot, epqstate, estate, - &tuple_deleted, false, false); + ExecDelete(mtstate, tupleid, oldtuple, planSlot, epqstate, + estate, &tuple_deleted, false, hufdp, NULL, + false); /* * For some reason if DELETE didn't happen (e.g. trigger prevented @@ -1116,16 +1199,36 @@ lreplace:; saved_tcs_map = mtstate->mt_transition_capture->tcs_map; /* - * resultRelInfo is one of the per-subplan resultRelInfos. So we - * should convert the tuple into root's tuple descriptor, since - * ExecInsert() starts the search from root. The tuple conversion - * map list is in the order of mtstate->resultRelInfo[], so to - * retrieve the one for this resultRel, we need to know the - * position of the resultRel in mtstate->resultRelInfo[]. + * We should convert the tuple into root's tuple descriptor, since + * ExecInsert() starts the search from root. To do that, we need to + * retrieve the tuple conversion map for this resultRelInfo. + * + * If we're running MERGE then resultRelInfo is per-partition + * resultRelInfo as initialised in ExecInitPartitionInfo(). Note + * that we don't expand inheritance for the resultRelation in case + * of MERGE and hence there is just one subplan. Whereas for + * regular UPDATE, resultRelInfo is one of the per-subplan + * resultRelInfos. In either case the position of this partition in + * tracked in ri_PartitionLeafIndex; + * + * Retrieve the map either by looking at the resultRelInfo's + * position in mtstate->resultRelInfo[] (for UPDATE) or by simply + * using the ri_PartitionLeafIndex value (for MERGE). */ - map_index = resultRelInfo - mtstate->resultRelInfo; - Assert(map_index >= 0 && map_index < mtstate->mt_nplans); - tupconv_map = tupconv_map_for_subplan(mtstate, map_index); + if (mtstate->operation == CMD_MERGE) + { + map_index = resultRelInfo->ri_PartitionLeafIndex; + Assert(mtstate->rootResultRelInfo == NULL); + tupconv_map = TupConvMapForLeaf(proute, + mtstate->resultRelInfo, + map_index); + } + else + { + map_index = resultRelInfo - mtstate->resultRelInfo; + Assert(map_index >= 0 && map_index < mtstate->mt_nplans); + tupconv_map = tupconv_map_for_subplan(mtstate, map_index); + } tuple = ConvertPartitionTupleSlot(tupconv_map, tuple, proute->root_tuple_slot, @@ -1135,12 +1238,16 @@ lreplace:; * Prepare for tuple routing, making it look like we're inserting * into the root. */ - Assert(mtstate->rootResultRelInfo != NULL); slot = ExecPrepareTupleRouting(mtstate, estate, proute, - mtstate->rootResultRelInfo, slot); + getTargetResultRelInfo(mtstate), + slot); ret_slot = ExecInsert(mtstate, slot, planSlot, - estate, canSetTag); + estate, actionState, canSetTag); + + /* Update is successful. */ + if (tuple_updated) + *tuple_updated = true; /* Revert ExecPrepareTupleRouting's node change. */ estate->es_result_relation_info = resultRelInfo; @@ -1179,6 +1286,15 @@ lreplace:; estate->es_crosscheck_snapshot, true /* wait for commit */ , &hufd, &lockmode); + + /* + * Copy the necessary information, if the caller has asked for it. We + * must do this irrespective of whether the tuple was updated or + * deleted. + */ + if (hufdp) + *hufdp = hufd; + switch (result) { case HeapTupleSelfUpdated: @@ -1223,26 +1339,42 @@ lreplace:; ereport(ERROR, (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE), errmsg("could not serialize access due to concurrent update"))); + if (!ItemPointerEquals(tupleid, &hufd.ctid)) { TupleTableSlot *epqslot; + /* + * If we're executing MERGE, then the onus of running + * EvalPlanQual() and handling its outcome lies with the + * caller. + */ + if (actionState != NULL) + return NULL; + + /* Regular UPDATE path. */ epqslot = EvalPlanQual(estate, epqstate, resultRelationDesc, - resultRelInfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(resultRelInfo), lockmode, &hufd.ctid, hufd.xmax); if (!TupIsNull(epqslot)) { *tupleid = hufd.ctid; + /* Normal UPDATE path */ slot = ExecFilterJunk(resultRelInfo->ri_junkFilter, epqslot); tuple = ExecMaterializeSlot(slot); goto lreplace; } } - /* tuple already deleted; nothing to do */ + + /* + * tuple already deleted; nothing to do. But MERGE might want + * to handle it differently. We've already filled-in hufdp + * with sufficient information for MERGE to look at. + */ return NULL; default: @@ -1271,6 +1403,9 @@ lreplace:; estate, false, NULL, NIL); } + if (tuple_updated) + *tuple_updated = true; + if (canSetTag) (estate->es_processed)++; @@ -1365,9 +1500,9 @@ ExecOnConflictUpdate(ModifyTableState *mtstate, * there's no historical behavior to break. * * It is the user's responsibility to prevent this situation from - * occurring. These problems are why SQL-2003 similarly specifies - * that for SQL MERGE, an exception must be raised in the event of - * an attempt to update the same row twice. + * occurring. These problems are why SQL Standard similarly + * specifies that for SQL MERGE, an exception must be raised in + * the event of an attempt to update the same row twice. */ if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(tuple.t_data))) ereport(ERROR, @@ -1489,7 +1624,7 @@ ExecOnConflictUpdate(ModifyTableState *mtstate, *returning = ExecUpdate(mtstate, &tuple.t_self, NULL, mtstate->mt_conflproj, planSlot, &mtstate->mt_epqstate, mtstate->ps.state, - canSetTag); + NULL, NULL, NULL, canSetTag); ReleaseBuffer(buffer); return true; @@ -1527,6 +1662,14 @@ fireBSTriggers(ModifyTableState *node) case CMD_DELETE: ExecBSDeleteTriggers(node->ps.state, resultRelInfo); break; + case CMD_MERGE: + if (node->mt_merge_subcommands & MERGE_INSERT) + ExecBSInsertTriggers(node->ps.state, resultRelInfo); + if (node->mt_merge_subcommands & MERGE_UPDATE) + ExecBSUpdateTriggers(node->ps.state, resultRelInfo); + if (node->mt_merge_subcommands & MERGE_DELETE) + ExecBSDeleteTriggers(node->ps.state, resultRelInfo); + break; default: elog(ERROR, "unknown operation"); break; @@ -1582,6 +1725,17 @@ fireASTriggers(ModifyTableState *node) ExecASDeleteTriggers(node->ps.state, resultRelInfo, node->mt_transition_capture); break; + case CMD_MERGE: + if (node->mt_merge_subcommands & MERGE_DELETE) + ExecASDeleteTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + if (node->mt_merge_subcommands & MERGE_UPDATE) + ExecASUpdateTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + if (node->mt_merge_subcommands & MERGE_INSERT) + ExecASInsertTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + break; default: elog(ERROR, "unknown operation"); break; @@ -1644,7 +1798,7 @@ ExecSetupTransitionCaptureState(ModifyTableState *mtstate, EState *estate) * * Returns a slot holding the tuple of the partition rowtype. */ -static TupleTableSlot * +TupleTableSlot * ExecPrepareTupleRouting(ModifyTableState *mtstate, EState *estate, PartitionTupleRouting *proute, @@ -1967,6 +2121,7 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; + if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2015,6 +2170,12 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + if (operation == CMD_MERGE) + { + ExecMerge(node, estate, slot, junkfilter, resultRelInfo); + continue; + } + tupleid = NULL; oldtuple = NULL; if (junkfilter != NULL) @@ -2096,19 +2257,20 @@ ExecModifyTable(PlanState *pstate) slot = ExecPrepareTupleRouting(node, estate, proute, resultRelInfo, slot); slot = ExecInsert(node, slot, planSlot, - estate, node->canSetTag); + estate, NULL, node->canSetTag); /* Revert ExecPrepareTupleRouting's state change. */ if (proute) estate->es_result_relation_info = resultRelInfo; break; case CMD_UPDATE: slot = ExecUpdate(node, tupleid, oldtuple, slot, planSlot, - &node->mt_epqstate, estate, node->canSetTag); + &node->mt_epqstate, estate, + NULL, NULL, NULL, node->canSetTag); break; case CMD_DELETE: slot = ExecDelete(node, tupleid, oldtuple, planSlot, &node->mt_epqstate, estate, - NULL, true, node->canSetTag); + NULL, true, NULL, NULL, node->canSetTag); break; default: elog(ERROR, "unknown operation"); @@ -2198,6 +2360,16 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) saved_resultRelInfo = estate->es_result_relation_info; resultRelInfo = mtstate->resultRelInfo; + + /* + * mergeTargetRelation must be set if we're running MERGE and mustn't be + * set if we're not. + */ + Assert(operation != CMD_MERGE || node->mergeTargetRelation > 0); + Assert(operation == CMD_MERGE || node->mergeTargetRelation == 0); + + resultRelInfo->ri_mergeTargetRTI = node->mergeTargetRelation; + i = 0; foreach(l, node->plans) { @@ -2276,7 +2448,8 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) * partition key. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE && - (operation == CMD_INSERT || update_tuple_routing_needed)) + (operation == CMD_INSERT || operation == CMD_MERGE || + update_tuple_routing_needed)) mtstate->mt_partition_tuple_routing = ExecSetupPartitionTupleRouting(mtstate, rel); @@ -2287,6 +2460,15 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) if (!(eflags & EXEC_FLAG_EXPLAIN_ONLY)) ExecSetupTransitionCaptureState(mtstate, estate); + /* + * If we are doing MERGE then setup child-parent mapping. This will be + * required in case we end up doing a partition-key update, triggering a + * tuple routing. + */ + if (mtstate->operation == CMD_MERGE && + mtstate->mt_partition_tuple_routing != NULL) + ExecSetupChildParentMapForLeaf(mtstate->mt_partition_tuple_routing); + /* * Construct mapping from each of the per-subplan partition attnos to the * root attno. This is required when during update row movement the tuple @@ -2478,6 +2660,102 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) } } + resultRelInfo = mtstate->resultRelInfo; + + if (node->mergeActionList) + { + ListCell *l; + ExprContext *econtext; + List *mergeMatchedActionStates = NIL; + List *mergeNotMatchedActionStates = NIL; + TupleDesc relationDesc = resultRelInfo->ri_RelationDesc->rd_att; + + mtstate->mt_merge_subcommands = 0; + + if (mtstate->ps.ps_ExprContext == NULL) + ExecAssignExprContext(estate, &mtstate->ps); + + econtext = mtstate->ps.ps_ExprContext; + + /* initialize slot for the existing tuple */ + Assert(mtstate->mt_existing == NULL); + mtstate->mt_existing = + ExecInitExtraTupleSlot(mtstate->ps.state, + mtstate->mt_partition_tuple_routing ? + NULL : relationDesc); + + /* initialise slot for merge actions */ + Assert(mtstate->mt_mergeproj == NULL); + mtstate->mt_mergeproj = + ExecInitExtraTupleSlot(mtstate->ps.state, + mtstate->mt_partition_tuple_routing ? + NULL : relationDesc); + + /* + * Create a MergeActionState for each action on the mergeActionList + * and add it to either a list of matched actions or not-matched + * actions. + */ + foreach(l, node->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + MergeActionState *action_state = makeNode(MergeActionState); + TupleDesc tupDesc; + + action_state->matched = action->matched; + action_state->commandType = action->commandType; + action_state->whenqual = ExecInitQual((List *) action->qual, + &mtstate->ps); + + /* create target slot for this action's projection */ + tupDesc = ExecTypeFromTL((List *) action->targetList, + resultRelInfo->ri_RelationDesc->rd_rel->relhasoids); + action_state->tupDesc = tupDesc; + + /* build action projection state */ + action_state->proj = + ExecBuildProjectionInfo(action->targetList, econtext, + mtstate->mt_mergeproj, &mtstate->ps, + resultRelInfo->ri_RelationDesc->rd_att); + + /* + * We create two lists - one for WHEN MATCHED actions and one + * for WHEN NOT MATCHED actions - and stick the + * MergeActionState into the appropriate list. + */ + if (action_state->matched) + mergeMatchedActionStates = + lappend(mergeMatchedActionStates, action_state); + else + mergeNotMatchedActionStates = + lappend(mergeNotMatchedActionStates, action_state); + + switch (action->commandType) + { + case CMD_INSERT: + mtstate->mt_merge_subcommands |= MERGE_INSERT; + break; + case CMD_UPDATE: + mtstate->mt_merge_subcommands |= MERGE_UPDATE; + break; + case CMD_DELETE: + mtstate->mt_merge_subcommands |= MERGE_DELETE; + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown operation"); + break; + } + + resultRelInfo->ri_mergeState->matchedActionStates = + mergeMatchedActionStates; + resultRelInfo->ri_mergeState->notMatchedActionStates = + mergeNotMatchedActionStates; + + } + } + /* select first subplan */ mtstate->mt_whichplan = 0; subplan = (Plan *) linitial(node->plans); @@ -2491,7 +2769,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) * --- no need to look first. Typically, this will be a 'ctid' or * 'wholerow' attribute, but in the case of a foreign data wrapper it * might be a set of junk attributes sufficient to identify the remote - * row. + * row. We follow this logic for MERGE, so it always has a junk 'ctid'. * * If there are multiple result relations, each one needs its own junk * filter. Note multiple rels are only possible for UPDATE/DELETE, so we @@ -2519,6 +2797,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) break; case CMD_UPDATE: case CMD_DELETE: + case CMD_MERGE: junk_filter_needed = true; break; default: @@ -2534,6 +2813,11 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) JunkFilter *j; subplan = mtstate->mt_plans[i]->plan; + + /* + * XXX we probably need to check plan output for CMD_MERGE + * also + */ if (operation == CMD_INSERT || operation == CMD_UPDATE) ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, subplan->targetlist); @@ -2542,7 +2826,9 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) resultRelInfo->ri_RelationDesc->rd_att->tdhasoid, ExecInitExtraTupleSlot(estate, NULL)); - if (operation == CMD_UPDATE || operation == CMD_DELETE) + if (operation == CMD_UPDATE || + operation == CMD_DELETE || + operation == CMD_MERGE) { /* For UPDATE/DELETE, find the appropriate junk attr now */ char relkind; @@ -2555,6 +2841,14 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) j->jf_junkAttNo = ExecFindJunkAttribute(j, "ctid"); if (!AttributeNumberIsValid(j->jf_junkAttNo)) elog(ERROR, "could not find junk ctid column"); + + if (operation == CMD_MERGE) + { + j->jf_otherJunkAttNo = ExecFindJunkAttribute(j, "tableoid"); + if (!AttributeNumberIsValid(j->jf_otherJunkAttNo)) + elog(ERROR, "could not find junk tableoid column"); + + } } else if (relkind == RELKIND_FOREIGN_TABLE) { diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c index 9fc4431b80..e050a317c0 100644 --- a/src/backend/executor/spi.c +++ b/src/backend/executor/spi.c @@ -2404,6 +2404,9 @@ _SPI_pquery(QueryDesc *queryDesc, bool fire_triggers, uint64 tcount) else res = SPI_OK_UPDATE; break; + case CMD_MERGE: + res = SPI_OK_MERGE; + break; default: return SPI_ERROR_OPUNKNOWN; } diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index c7293a60d7..770ed3b1a8 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -207,6 +207,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(partitioned_rels); COPY_SCALAR_FIELD(partColsUpdated); COPY_NODE_FIELD(resultRelations); + COPY_SCALAR_FIELD(mergeTargetRelation); COPY_SCALAR_FIELD(resultRelIndex); COPY_SCALAR_FIELD(rootResultRelIndex); COPY_NODE_FIELD(plans); @@ -222,6 +223,8 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(onConflictWhere); COPY_SCALAR_FIELD(exclRelRTI); COPY_NODE_FIELD(exclRelTlist); + COPY_NODE_FIELD(mergeSourceTargetList); + COPY_NODE_FIELD(mergeActionList); return newnode; } @@ -2977,6 +2980,9 @@ _copyQuery(const Query *from) COPY_NODE_FIELD(setOperations); COPY_NODE_FIELD(constraintDeps); COPY_NODE_FIELD(withCheckOptions); + COPY_SCALAR_FIELD(mergeTarget_relation); + COPY_NODE_FIELD(mergeSourceTargetList); + COPY_NODE_FIELD(mergeActionList); COPY_LOCATION_FIELD(stmt_location); COPY_LOCATION_FIELD(stmt_len); @@ -3040,6 +3046,34 @@ _copyUpdateStmt(const UpdateStmt *from) return newnode; } +static MergeStmt * +_copyMergeStmt(const MergeStmt *from) +{ + MergeStmt *newnode = makeNode(MergeStmt); + + COPY_NODE_FIELD(relation); + COPY_NODE_FIELD(source_relation); + COPY_NODE_FIELD(join_condition); + COPY_NODE_FIELD(mergeActionList); + + return newnode; +} + +static MergeAction * +_copyMergeAction(const MergeAction *from) +{ + MergeAction *newnode = makeNode(MergeAction); + + COPY_SCALAR_FIELD(matched); + COPY_SCALAR_FIELD(commandType); + COPY_NODE_FIELD(condition); + COPY_NODE_FIELD(qual); + COPY_NODE_FIELD(stmt); + COPY_NODE_FIELD(targetList); + + return newnode; +} + static SelectStmt * _copySelectStmt(const SelectStmt *from) { @@ -5102,6 +5136,12 @@ copyObjectImpl(const void *from) case T_UpdateStmt: retval = _copyUpdateStmt(from); break; + case T_MergeStmt: + retval = _copyMergeStmt(from); + break; + case T_MergeAction: + retval = _copyMergeAction(from); + break; case T_SelectStmt: retval = _copySelectStmt(from); break; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 765b1be74b..5a0151eece 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -987,6 +987,8 @@ _equalQuery(const Query *a, const Query *b) COMPARE_NODE_FIELD(setOperations); COMPARE_NODE_FIELD(constraintDeps); COMPARE_NODE_FIELD(withCheckOptions); + COMPARE_NODE_FIELD(mergeSourceTargetList); + COMPARE_NODE_FIELD(mergeActionList); COMPARE_LOCATION_FIELD(stmt_location); COMPARE_LOCATION_FIELD(stmt_len); @@ -1042,6 +1044,30 @@ _equalUpdateStmt(const UpdateStmt *a, const UpdateStmt *b) return true; } +static bool +_equalMergeStmt(const MergeStmt *a, const MergeStmt *b) +{ + COMPARE_NODE_FIELD(relation); + COMPARE_NODE_FIELD(source_relation); + COMPARE_NODE_FIELD(join_condition); + COMPARE_NODE_FIELD(mergeActionList); + + return true; +} + +static bool +_equalMergeAction(const MergeAction *a, const MergeAction *b) +{ + COMPARE_SCALAR_FIELD(matched); + COMPARE_SCALAR_FIELD(commandType); + COMPARE_NODE_FIELD(condition); + COMPARE_NODE_FIELD(qual); + COMPARE_NODE_FIELD(stmt); + COMPARE_NODE_FIELD(targetList); + + return true; +} + static bool _equalSelectStmt(const SelectStmt *a, const SelectStmt *b) { @@ -3233,6 +3259,12 @@ equal(const void *a, const void *b) case T_UpdateStmt: retval = _equalUpdateStmt(a, b); break; + case T_MergeStmt: + retval = _equalMergeStmt(a, b); + break; + case T_MergeAction: + retval = _equalMergeAction(a, b); + break; case T_SelectStmt: retval = _equalSelectStmt(a, b); break; diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 6c76c41ebe..68e2cec66e 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -2146,6 +2146,16 @@ expression_tree_walker(Node *node, return true; } break; + case T_MergeAction: + { + MergeAction *action = (MergeAction *) node; + + if (walker(action->targetList, context)) + return true; + if (walker(action->qual, context)) + return true; + } + break; case T_JoinExpr: { JoinExpr *join = (JoinExpr *) node; @@ -2255,6 +2265,10 @@ query_tree_walker(Query *query, return true; if (walker((Node *) query->onConflict, context)) return true; + if (walker((Node *) query->mergeSourceTargetList, context)) + return true; + if (walker((Node *) query->mergeActionList, context)) + return true; if (walker((Node *) query->returningList, context)) return true; if (walker((Node *) query->jointree, context)) @@ -2932,6 +2946,18 @@ expression_tree_mutator(Node *node, return (Node *) newnode; } break; + case T_MergeAction: + { + MergeAction *action = (MergeAction *) node; + MergeAction *newnode; + + FLATCOPY(newnode, action, MergeAction); + MUTATE(newnode->qual, action->qual, Node *); + MUTATE(newnode->targetList, action->targetList, List *); + + return (Node *) newnode; + } + break; case T_JoinExpr: { JoinExpr *join = (JoinExpr *) node; @@ -3083,6 +3109,8 @@ query_tree_mutator(Query *query, MUTATE(query->targetList, query->targetList, List *); MUTATE(query->withCheckOptions, query->withCheckOptions, List *); MUTATE(query->onConflict, query->onConflict, OnConflictExpr *); + MUTATE(query->mergeSourceTargetList, query->mergeSourceTargetList, List *); + MUTATE(query->mergeActionList, query->mergeActionList, List *); MUTATE(query->returningList, query->returningList, List *); MUTATE(query->jointree, query->jointree, FromExpr *); MUTATE(query->setOperations, query->setOperations, Node *); @@ -3224,9 +3252,9 @@ query_or_expression_tree_mutator(Node *node, * boundaries: we descend to everything that's possibly interesting. * * Currently, the node type coverage here extends only to DML statements - * (SELECT/INSERT/UPDATE/DELETE) and nodes that can appear in them, because - * this is used mainly during analysis of CTEs, and only DML statements can - * appear in CTEs. + * (SELECT/INSERT/UPDATE/DELETE/MERGE) and nodes that can appear in them, + * because this is used mainly during analysis of CTEs, and only DML + * statements can appear in CTEs. */ bool raw_expression_tree_walker(Node *node, @@ -3406,6 +3434,20 @@ raw_expression_tree_walker(Node *node, return true; } break; + case T_MergeStmt: + { + MergeStmt *stmt = (MergeStmt *) node; + + if (walker(stmt->relation, context)) + return true; + if (walker(stmt->source_relation, context)) + return true; + if (walker(stmt->join_condition, context)) + return true; + if (walker(stmt->mergeActionList, context)) + return true; + } + break; case T_SelectStmt: { SelectStmt *stmt = (SelectStmt *) node; diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f61ae03ac5..9ebea55048 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -375,6 +375,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(partitioned_rels); WRITE_BOOL_FIELD(partColsUpdated); WRITE_NODE_FIELD(resultRelations); + WRITE_INT_FIELD(mergeTargetRelation); WRITE_INT_FIELD(resultRelIndex); WRITE_INT_FIELD(rootResultRelIndex); WRITE_NODE_FIELD(plans); @@ -390,6 +391,21 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(onConflictWhere); WRITE_UINT_FIELD(exclRelRTI); WRITE_NODE_FIELD(exclRelTlist); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); +} + +static void +_outMergeAction(StringInfo str, const MergeAction *node) +{ + WRITE_NODE_TYPE("MERGEACTION"); + + WRITE_BOOL_FIELD(matched); + WRITE_ENUM_FIELD(commandType, CmdType); + WRITE_NODE_FIELD(condition); + WRITE_NODE_FIELD(qual); + /* We don't dump the stmt node */ + WRITE_NODE_FIELD(targetList); } static void @@ -2114,6 +2130,7 @@ _outModifyTablePath(StringInfo str, const ModifyTablePath *node) WRITE_NODE_FIELD(partitioned_rels); WRITE_BOOL_FIELD(partColsUpdated); WRITE_NODE_FIELD(resultRelations); + WRITE_INT_FIELD(mergeTargetRelation); WRITE_NODE_FIELD(subpaths); WRITE_NODE_FIELD(subroots); WRITE_NODE_FIELD(withCheckOptionLists); @@ -2121,6 +2138,8 @@ _outModifyTablePath(StringInfo str, const ModifyTablePath *node) WRITE_NODE_FIELD(rowMarks); WRITE_NODE_FIELD(onconflict); WRITE_INT_FIELD(epqParam); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); } static void @@ -2942,6 +2961,9 @@ _outQuery(StringInfo str, const Query *node) WRITE_NODE_FIELD(setOperations); WRITE_NODE_FIELD(constraintDeps); /* withCheckOptions intentionally omitted, see comment in parsenodes.h */ + WRITE_INT_FIELD(mergeTarget_relation); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); WRITE_LOCATION_FIELD(stmt_location); WRITE_LOCATION_FIELD(stmt_len); } @@ -3657,6 +3679,9 @@ outNode(StringInfo str, const void *obj) case T_ModifyTable: _outModifyTable(str, obj); break; + case T_MergeAction: + _outMergeAction(str, obj); + break; case T_Append: _outAppend(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index fd4586e73d..3b8071d056 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -270,6 +270,9 @@ _readQuery(void) READ_NODE_FIELD(setOperations); READ_NODE_FIELD(constraintDeps); /* withCheckOptions intentionally omitted, see comment in parsenodes.h */ + READ_INT_FIELD(mergeTarget_relation); + READ_NODE_FIELD(mergeSourceTargetList); + READ_NODE_FIELD(mergeActionList); READ_LOCATION_FIELD(stmt_location); READ_LOCATION_FIELD(stmt_len); @@ -1576,6 +1579,7 @@ _readModifyTable(void) READ_NODE_FIELD(partitioned_rels); READ_BOOL_FIELD(partColsUpdated); READ_NODE_FIELD(resultRelations); + READ_INT_FIELD(mergeTargetRelation); READ_INT_FIELD(resultRelIndex); READ_INT_FIELD(rootResultRelIndex); READ_NODE_FIELD(plans); @@ -1591,6 +1595,8 @@ _readModifyTable(void) READ_NODE_FIELD(onConflictWhere); READ_UINT_FIELD(exclRelRTI); READ_NODE_FIELD(exclRelTlist); + READ_NODE_FIELD(mergeSourceTargetList); + READ_NODE_FIELD(mergeActionList); READ_DONE(); } diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 8b4f031d96..b4d376b4ba 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -282,9 +282,13 @@ static ModifyTable *make_modifytable(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subplans, + List *resultRelations, + Index mergeTargetRelation, + List *subplans, List *withCheckOptionLists, List *returningLists, - List *rowMarks, OnConflictExpr *onconflict, int epqParam); + List *rowMarks, OnConflictExpr *onconflict, + List *mergeSourceTargetList, + List *mergeActionList, int epqParam); static GatherMerge *create_gather_merge_plan(PlannerInfo *root, GatherMergePath *best_path); @@ -2394,11 +2398,14 @@ create_modifytable_plan(PlannerInfo *root, ModifyTablePath *best_path) best_path->partitioned_rels, best_path->partColsUpdated, best_path->resultRelations, + best_path->mergeTargetRelation, subplans, best_path->withCheckOptionLists, best_path->returningLists, best_path->rowMarks, best_path->onconflict, + best_path->mergeSourceTargetList, + best_path->mergeActionList, best_path->epqParam); copy_generic_path_info(&plan->plan, &best_path->path); @@ -6465,9 +6472,13 @@ make_modifytable(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subplans, + List *resultRelations, + Index mergeTargetRelation, + List *subplans, List *withCheckOptionLists, List *returningLists, - List *rowMarks, OnConflictExpr *onconflict, int epqParam) + List *rowMarks, OnConflictExpr *onconflict, + List *mergeSourceTargetList, + List *mergeActionList, int epqParam) { ModifyTable *node = makeNode(ModifyTable); List *fdw_private_list; @@ -6493,6 +6504,7 @@ make_modifytable(PlannerInfo *root, node->partitioned_rels = partitioned_rels; node->partColsUpdated = partColsUpdated; node->resultRelations = resultRelations; + node->mergeTargetRelation = mergeTargetRelation; node->resultRelIndex = -1; /* will be set correctly in setrefs.c */ node->rootResultRelIndex = -1; /* will be set correctly in setrefs.c */ node->plans = subplans; @@ -6525,6 +6537,8 @@ make_modifytable(PlannerInfo *root, node->withCheckOptionLists = withCheckOptionLists; node->returningLists = returningLists; node->rowMarks = rowMarks; + node->mergeSourceTargetList = mergeSourceTargetList; + node->mergeActionList = mergeActionList; node->epqParam = epqParam; /* diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 52c21e6870..14972bc9b2 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -790,6 +790,24 @@ subquery_planner(PlannerGlobal *glob, Query *parse, /* exclRelTlist contains only Vars, so no preprocessing needed */ } + foreach(l, parse->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + action->targetList = (List *) + preprocess_expression(root, + (Node *) action->targetList, + EXPRKIND_TARGET); + action->qual = + preprocess_expression(root, + (Node *) action->qual, + EXPRKIND_QUAL); + } + + parse->mergeSourceTargetList = (List *) + preprocess_expression(root, (Node *) parse->mergeSourceTargetList, + EXPRKIND_TARGET); + root->append_rel_list = (List *) preprocess_expression(root, (Node *) root->append_rel_list, EXPRKIND_APPINFO); @@ -1531,6 +1549,7 @@ inheritance_planner(PlannerInfo *root) subroot->parse->returningList); Assert(!parse->onConflict); + Assert(parse->mergeActionList == NIL); } /* Result path must go into outer query's FINAL upperrel */ @@ -1589,12 +1608,15 @@ inheritance_planner(PlannerInfo *root) partitioned_rels, partColsUpdated, resultRelations, + 0, subpaths, subroots, withCheckOptionLists, returningLists, rowMarks, NULL, + NULL, + NULL, SS_assign_special_param(root))); } @@ -2128,8 +2150,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, } /* - * If this is an INSERT/UPDATE/DELETE, and we're not being called from - * inheritance_planner, add the ModifyTable node. + * If this is an INSERT/UPDATE/DELETE/MERGE, and we're not being + * called from inheritance_planner, add the ModifyTable node. */ if (parse->commandType != CMD_SELECT && !inheritance_update) { @@ -2169,12 +2191,15 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, NIL, false, list_make1_int(parse->resultRelation), + parse->mergeTarget_relation, list_make1(path), list_make1(root), withCheckOptionLists, returningLists, rowMarks, parse->onConflict, + parse->mergeSourceTargetList, + parse->mergeActionList, SS_assign_special_param(root)); } diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index 69dd327f0c..d30889ec7c 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -851,9 +851,68 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) fix_scan_list(root, splan->exclRelTlist, rtoffset); } + /* + * The MERGE produces the target rows by performing a right + * join between the target relation and the source relation + * (which could be a plain relation or a subquery). The INSERT + * and UPDATE actions of the MERGE requires access to the + * columns from the source relation. We arrange things so that + * the source relation attributes are available as INNER_VAR + * and the target relation attributes are available from the + * scan tuple. + */ + if (splan->mergeActionList != NIL) + { + /* + * mergeSourceTargetList is already setup correctly to + * include all Vars coming from the source relation. So we + * fix the targetList of individual action nodes by + * ensuring that the source relation Vars are referenced + * as INNER_VAR. Note that for this to work correctly, + * during execution, the ecxt_innertuple must be set to + * the tuple obtained from the source relation. + * + * We leave the Vars from the result relation (i.e. the + * target relation) unchanged i.e. those Vars would be + * picked from the scan slot. So during execution, we must + * ensure that ecxt_scantuple is setup correctly to refer + * to the tuple from the target relation. + */ + + indexed_tlist *itlist; + + itlist = build_tlist_index(splan->mergeSourceTargetList); + + foreach(l, splan->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + /* Fix targetList of each action. */ + action->targetList = fix_join_expr(root, + action->targetList, + NULL, itlist, + linitial_int(splan->resultRelations), + rtoffset); + + /* Fix quals too. */ + action->qual = (Node *) fix_join_expr(root, + (List *) action->qual, + NULL, itlist, + linitial_int(splan->resultRelations), + rtoffset); + } + } + splan->nominalRelation += rtoffset; splan->exclRelRTI += rtoffset; + /* + * Don't mess-up with mergeTargetRelation if it's set to zero + * i.e. an invalid value. + */ + if (splan->mergeTargetRelation > 0) + splan->mergeTargetRelation += rtoffset; + foreach(l, splan->partitioned_rels) { lfirst_int(l) += rtoffset; diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8603feef2b..4a864b2340 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -105,9 +105,13 @@ preprocess_targetlist(PlannerInfo *root) * scribbles on parse->targetList, which is not very desirable, but we * keep it that way to avoid changing APIs used by FDWs. */ - if (command_type == CMD_UPDATE || command_type == CMD_DELETE) + if (command_type == CMD_UPDATE || + command_type == CMD_DELETE) rewriteTargetListUD(parse, target_rte, target_relation); + if (command_type == CMD_MERGE) + rewriteTargetListMerge(parse, target_relation); + /* * for heap_form_tuple to work, the targetlist must match the exact order * of the attributes. We also need to fill in any missing attributes. -ay @@ -118,6 +122,39 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE command, handle targetlist of each MergeAction separately. We + * give the same treatment to MergeAction->targetList as we would have + * given to a regular INSERT/UPDATE/DELETE. + */ + if (command_type == CMD_MERGE) + { + ListCell *l; + + foreach(l, parse->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + switch (action->commandType) + { + case CMD_INSERT: + case CMD_UPDATE: + action->targetList = expand_targetlist(action->targetList, + action->commandType, + result_relation, + target_relation); + break; + case CMD_DELETE: + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + + } + } + } + /* * Add necessary junk columns for rowmarked rels. These values are needed * for locking of rels selected FOR UPDATE/SHARE, and to do EvalPlanQual @@ -348,6 +385,7 @@ expand_targetlist(List *tlist, int command_type, true /* byval */ ); } break; + case CMD_MERGE: case CMD_UPDATE: if (!att_tup->attisdropped) { diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 22133fcf12..416b3f9578 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -3284,17 +3284,21 @@ create_lockrows_path(PlannerInfo *root, RelOptInfo *rel, * 'rowMarks' is a list of PlanRowMarks (non-locking only) * 'onconflict' is the ON CONFLICT clause, or NULL * 'epqParam' is the ID of Param for EvalPlanQual re-eval + * 'mergeActionList' is a list of MERGE actions */ ModifyTablePath * create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subpaths, + List *resultRelations, + Index mergeTargetRelation, + List *subpaths, List *subroots, List *withCheckOptionLists, List *returningLists, List *rowMarks, OnConflictExpr *onconflict, - int epqParam) + List *mergeSourceTargetList, + List *mergeActionList, int epqParam) { ModifyTablePath *pathnode = makeNode(ModifyTablePath); double total_size; @@ -3359,6 +3363,7 @@ create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, pathnode->partitioned_rels = list_copy(partitioned_rels); pathnode->partColsUpdated = partColsUpdated; pathnode->resultRelations = resultRelations; + pathnode->mergeTargetRelation = mergeTargetRelation; pathnode->subpaths = subpaths; pathnode->subroots = subroots; pathnode->withCheckOptionLists = withCheckOptionLists; @@ -3366,6 +3371,8 @@ create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, pathnode->rowMarks = rowMarks; pathnode->onconflict = onconflict; pathnode->epqParam = epqParam; + pathnode->mergeSourceTargetList = mergeSourceTargetList; + pathnode->mergeActionList = mergeActionList; return pathnode; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index bd3a0c4a0a..8626cb2904 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1835,6 +1835,10 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) trigDesc->trig_delete_before_row)) result = true; break; + /* There is no separate event for MERGE, only INSERT/UPDATE/DELETE */ + case CMD_MERGE: + result = false; + break; default: elog(ERROR, "unrecognized CmdType: %d", (int) event); break; diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index da8f0f93fc..901cf24e20 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -1237,7 +1237,6 @@ find_childrel_parents(PlannerInfo *root, RelOptInfo *rel) return result; } - /* * get_baserel_parampathinfo * Get the ParamPathInfo for a parameterized path for a base relation, diff --git a/src/backend/parser/Makefile b/src/backend/parser/Makefile index f14febdbda..95fdf0b973 100644 --- a/src/backend/parser/Makefile +++ b/src/backend/parser/Makefile @@ -14,7 +14,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS) OBJS= analyze.o gram.o scan.o parser.o \ parse_agg.o parse_clause.o parse_coerce.o parse_collate.o parse_cte.o \ - parse_enr.o parse_expr.o parse_func.o parse_node.o parse_oper.o \ + parse_enr.o parse_expr.o parse_func.o parse_merge.o parse_node.o parse_oper.o \ parse_param.o parse_relation.o parse_target.o parse_type.o \ parse_utilcmd.o scansup.o diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index a4b5aaef44..7eb9544efe 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -38,6 +38,7 @@ #include "parser/parse_cte.h" #include "parser/parse_expr.h" #include "parser/parse_func.h" +#include "parser/parse_merge.h" #include "parser/parse_oper.h" #include "parser/parse_param.h" #include "parser/parse_relation.h" @@ -53,9 +54,6 @@ post_parse_analyze_hook_type post_parse_analyze_hook = NULL; static Query *transformOptionalSelectInto(ParseState *pstate, Node *parseTree); static Query *transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt); static Query *transformInsertStmt(ParseState *pstate, InsertStmt *stmt); -static List *transformInsertRow(ParseState *pstate, List *exprlist, - List *stmtcols, List *icolumns, List *attrnos, - bool strip_indirection); static OnConflictExpr *transformOnConflictClause(ParseState *pstate, OnConflictClause *onConflictClause); static int count_rowexpr_columns(ParseState *pstate, Node *expr); @@ -68,8 +66,6 @@ static void determineRecursiveColTypes(ParseState *pstate, Node *larg, List *nrtargetlist); static Query *transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt); static List *transformReturningList(ParseState *pstate, List *returningList); -static List *transformUpdateTargetList(ParseState *pstate, - List *targetList); static Query *transformDeclareCursorStmt(ParseState *pstate, DeclareCursorStmt *stmt); static Query *transformExplainStmt(ParseState *pstate, @@ -267,6 +263,7 @@ transformStmt(ParseState *pstate, Node *parseTree) case T_InsertStmt: case T_UpdateStmt: case T_DeleteStmt: + case T_MergeStmt: (void) test_raw_expression_coverage(parseTree, NULL); break; default: @@ -291,6 +288,10 @@ transformStmt(ParseState *pstate, Node *parseTree) result = transformUpdateStmt(pstate, (UpdateStmt *) parseTree); break; + case T_MergeStmt: + result = transformMergeStmt(pstate, (MergeStmt *) parseTree); + break; + case T_SelectStmt: { SelectStmt *n = (SelectStmt *) parseTree; @@ -366,6 +367,7 @@ analyze_requires_snapshot(RawStmt *parseTree) case T_InsertStmt: case T_DeleteStmt: case T_UpdateStmt: + case T_MergeStmt: case T_SelectStmt: result = true; break; @@ -896,7 +898,7 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) * attrnos: integer column numbers (must be same length as icolumns) * strip_indirection: if true, remove any field/array assignment nodes */ -static List * +List * transformInsertRow(ParseState *pstate, List *exprlist, List *stmtcols, List *icolumns, List *attrnos, bool strip_indirection) @@ -2260,9 +2262,9 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt) /* * transformUpdateTargetList - - * handle SET clause in UPDATE/INSERT ... ON CONFLICT UPDATE + * handle SET clause in UPDATE/MERGE/INSERT ... ON CONFLICT UPDATE */ -static List * +List * transformUpdateTargetList(ParseState *pstate, List *origTlist) { List *tlist = NIL; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index cd5ba2d4d8..ebca5f3eb7 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -282,6 +282,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CreateMatViewStmt RefreshMatViewStmt CreateAmStmt CreatePublicationStmt AlterPublicationStmt CreateSubscriptionStmt AlterSubscriptionStmt DropSubscriptionStmt + MergeStmt %type <node> select_no_parens select_with_parens select_clause simple_select values_clause @@ -584,6 +585,10 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound partbound_datum_list range_datum_list %type <defelt> hash_partbound_elem +%type <node> merge_when_clause opt_and_condition +%type <list> merge_when_list +%type <node> merge_update merge_delete merge_insert + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -651,7 +656,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED - MAPPING MATCH MATERIALIZED MAXVALUE METHOD MINUTE_P MINVALUE MODE MONTH_P MOVE + MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MERGE METHOD + MINUTE_P MINVALUE MODE MONTH_P MOVE NAME_P NAMES NATIONAL NATURAL NCHAR NEW NEXT NO NONE NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF @@ -920,6 +926,7 @@ stmt : | RefreshMatViewStmt | LoadStmt | LockStmt + | MergeStmt | NotifyStmt | PrepareStmt | ReassignOwnedStmt @@ -10660,6 +10667,7 @@ ExplainableStmt: | InsertStmt | UpdateStmt | DeleteStmt + | MergeStmt | DeclareCursorStmt | CreateAsStmt | CreateMatViewStmt @@ -10722,6 +10730,7 @@ PreparableStmt: | InsertStmt | UpdateStmt | DeleteStmt /* by default all are $$=$1 */ + | MergeStmt ; /***************************************************************************** @@ -11088,6 +11097,151 @@ set_target_list: ; +/***************************************************************************** + * + * QUERY: + * MERGE STATEMENT + * + *****************************************************************************/ + +MergeStmt: + MERGE INTO relation_expr_opt_alias + USING table_ref + ON a_expr + merge_when_list + { + MergeStmt *m = makeNode(MergeStmt); + + m->relation = $3; + m->source_relation = $5; + m->join_condition = $7; + m->mergeActionList = $8; + + $$ = (Node *)m; + } + ; + + +merge_when_list: + merge_when_clause { $$ = list_make1($1); } + | merge_when_list merge_when_clause { $$ = lappend($1,$2); } + ; + +merge_when_clause: + WHEN MATCHED opt_and_condition THEN merge_update + { + MergeAction *m = makeNode(MergeAction); + + m->matched = true; + m->commandType = CMD_UPDATE; + m->condition = $3; + m->stmt = $5; + + $$ = (Node *)m; + } + | WHEN MATCHED opt_and_condition THEN merge_delete + { + MergeAction *m = makeNode(MergeAction); + + m->matched = true; + m->commandType = CMD_DELETE; + m->condition = $3; + m->stmt = $5; + + $$ = (Node *)m; + } + | WHEN NOT MATCHED opt_and_condition THEN merge_insert + { + MergeAction *m = makeNode(MergeAction); + + m->matched = false; + m->commandType = CMD_INSERT; + m->condition = $4; + m->stmt = $6; + + $$ = (Node *)m; + } + | WHEN NOT MATCHED opt_and_condition THEN DO NOTHING + { + MergeAction *m = makeNode(MergeAction); + + m->matched = false; + m->commandType = CMD_NOTHING; + m->condition = $4; + m->stmt = NULL; + + $$ = (Node *)m; + } + ; + +opt_and_condition: + AND a_expr { $$ = $2; } + | { $$ = NULL; } + ; + +merge_delete: + DELETE_P + { + DeleteStmt *n = makeNode(DeleteStmt); + $$ = (Node *)n; + } + ; + +merge_update: + UPDATE SET set_clause_list + { + UpdateStmt *n = makeNode(UpdateStmt); + n->targetList = $3; + + $$ = (Node *)n; + } + ; + +merge_insert: + INSERT values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->selectStmt = $2; + + $$ = (Node *)n; + } + | INSERT OVERRIDING override_kind VALUE_P values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->override = $3; + n->selectStmt = $5; + + $$ = (Node *)n; + } + | INSERT '(' insert_column_list ')' values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = $3; + n->selectStmt = $5; + + $$ = (Node *)n; + } + | INSERT '(' insert_column_list ')' OVERRIDING override_kind VALUE_P values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = $3; + n->override = $6; + n->selectStmt = $8; + + $$ = (Node *)n; + } + | INSERT DEFAULT VALUES + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->selectStmt = NULL; + + $$ = (Node *)n; + } + ; + /***************************************************************************** * * QUERY: @@ -15088,8 +15242,10 @@ unreserved_keyword: | LOGGED | MAPPING | MATCH + | MATCHED | MATERIALIZED | MAXVALUE + | MERGE | METHOD | MINUTE_P | MINVALUE diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 377a7ed6d0..544e7300b8 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -455,6 +455,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) case EXPR_KIND_VALUES_SINGLE: errkind = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + if (isAgg) + err = _("aggregate functions are not allowed in WHEN AND conditions"); + else + err = _("grouping operations are not allowed in WHEN AND conditions"); + + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: if (isAgg) @@ -873,6 +880,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_VALUES_SINGLE: errkind = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + err = _("window functions are not allowed in WHEN AND conditions"); + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: err = _("window functions are not allowed in check constraints"); diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 3a02307bd9..9df9828df8 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -76,9 +76,6 @@ static RangeTblEntry *transformRangeTableFunc(ParseState *pstate, RangeTableFunc *t); static TableSampleClause *transformRangeTableSample(ParseState *pstate, RangeTableSample *rts); -static Node *transformFromClauseItem(ParseState *pstate, Node *n, - RangeTblEntry **top_rte, int *top_rti, - List **namespace); static Node *buildMergedJoinVar(ParseState *pstate, JoinType jointype, Var *l_colvar, Var *r_colvar); static ParseNamespaceItem *makeNamespaceItem(RangeTblEntry *rte, @@ -139,6 +136,7 @@ transformFromClause(ParseState *pstate, List *frmList) n = transformFromClauseItem(pstate, n, &rte, &rtindex, + NULL, NULL, &namespace); checkNameSpaceConflicts(pstate, pstate->p_namespace, namespace); @@ -1100,9 +1098,10 @@ getRTEForSpecialRelationTypes(ParseState *pstate, RangeVar *rv) * as table/column names by this item. (The lateral_only flags in these items * are indeterminate and should be explicitly set by the caller before use.) */ -static Node * +Node * transformFromClauseItem(ParseState *pstate, Node *n, RangeTblEntry **top_rte, int *top_rti, + RangeTblEntry **right_rte, int *right_rti, List **namespace) { if (IsA(n, RangeVar)) @@ -1194,7 +1193,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, /* Recursively transform the contained relation */ rel = transformFromClauseItem(pstate, rts->relation, - top_rte, top_rti, namespace); + top_rte, top_rti, NULL, NULL, namespace); /* Currently, grammar could only return a RangeVar as contained rel */ rtr = castNode(RangeTblRef, rel); rte = rt_fetch(rtr->rtindex, pstate->p_rtable); @@ -1222,6 +1221,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, List *l_namespace, *r_namespace, *my_namespace, + *save_namespace, *l_colnames, *r_colnames, *res_colnames, @@ -1240,6 +1240,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, j->larg = transformFromClauseItem(pstate, j->larg, &l_rte, &l_rtindex, + NULL, NULL, &l_namespace); /* @@ -1263,12 +1264,34 @@ transformFromClauseItem(ParseState *pstate, Node *n, sv_namespace_length = list_length(pstate->p_namespace); pstate->p_namespace = list_concat(pstate->p_namespace, l_namespace); + /* + * If we are running MERGE, don't make the other RTEs visible while + * parsing the source relation. It mustn't see them. + * + * XXX Currently, only MERGE passes non-NULL value for right_rte, so we + * can safely deduce if we're running MERGE or not by just looking at + * the right_rte. If that ever changes, we should look at other means + * to find that. + */ + if (right_rte) + { + save_namespace = pstate->p_namespace; + pstate->p_namespace = NIL; + } + /* And now we can process the RHS */ j->rarg = transformFromClauseItem(pstate, j->rarg, &r_rte, &r_rtindex, + NULL, NULL, &r_namespace); + /* + * And now restore the namespace again so that join-quals can see it. + */ + if (right_rte) + pstate->p_namespace = save_namespace; + /* Remove the left-side RTEs from the namespace list again */ pstate->p_namespace = list_truncate(pstate->p_namespace, sv_namespace_length); @@ -1295,6 +1318,12 @@ transformFromClauseItem(ParseState *pstate, Node *n, expandRTE(r_rte, r_rtindex, 0, -1, false, &r_colnames, &r_colvars); + if (right_rte) + *right_rte = r_rte; + + if (right_rti) + *right_rti = r_rtindex; + /* * Natural join does not explicitly specify columns; must generate * columns to join. Need to run through the list of columns from each diff --git a/src/backend/parser/parse_collate.c b/src/backend/parser/parse_collate.c index 6d34245083..51c73c4018 100644 --- a/src/backend/parser/parse_collate.c +++ b/src/backend/parser/parse_collate.c @@ -485,6 +485,7 @@ assign_collations_walker(Node *node, assign_collations_context *context) case T_FromExpr: case T_OnConflictExpr: case T_SortGroupClause: + case T_MergeAction: (void) expression_tree_walker(node, assign_collations_walker, (void *) &loccontext); diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 385e54a9b6..38fbe3366f 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -1818,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_RETURNING: case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: + case EXPR_KIND_MERGE_WHEN_AND: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3475,6 +3476,8 @@ ParseExprKindName(ParseExprKind exprKind) return "PARTITION BY"; case EXPR_KIND_CALL_ARGUMENT: return "CALL"; + case EXPR_KIND_MERGE_WHEN_AND: + return "MERGE WHEN AND"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index ea5d5212b4..615aee6d15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2277,6 +2277,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) /* okay, since we process this like a SELECT tlist */ pstate->p_hasTargetSRFs = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + err = _("set-returning functions are not allowed in WHEN AND conditions"); + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: err = _("set-returning functions are not allowed in check constraints"); diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c new file mode 100644 index 0000000000..b6e0c46656 --- /dev/null +++ b/src/backend/parser/parse_merge.c @@ -0,0 +1,670 @@ +/*------------------------------------------------------------------------- + * + * parse_merge.c + * handle merge-statement in parser + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/parser/parse_merge.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "miscadmin.h" + +#include "access/sysattr.h" +#include "nodes/makefuncs.h" +#include "parser/analyze.h" +#include "parser/parse_collate.h" +#include "parser/parsetree.h" +#include "parser/parser.h" +#include "parser/parse_clause.h" +#include "parser/parse_merge.h" +#include "parser/parse_relation.h" +#include "parser/parse_target.h" +#include "utils/rel.h" +#include "utils/relcache.h" + +static int transformMergeJoinClause(ParseState *pstate, Node *merge, + List **mergeSourceTargetList); +static void setNamespaceForMergeAction(ParseState *pstate, + MergeAction *action); +static void setNamespaceVisibilityForRTE(List *namespace, RangeTblEntry *rte, + bool rel_visible, + bool cols_visible); +static List *expandSourceTL(ParseState *pstate, RangeTblEntry *rte, + int rtindex); + +/* + * Special handling for MERGE statement is required because we assemble + * the query manually. This is similar to setTargetTable() followed + * by transformFromClause() but with a few less steps. + * + * Process the FROM clause and add items to the query's range table, + * joinlist, and namespace. + * + * A special targetlist comprising of the columns from the right-subtree of + * the join is populated and returned. Note that when the JoinExpr is + * setup by transformMergeStmt, the left subtree has the target result + * relation and the right subtree has the source relation. + * + * Returns the rangetable index of the target relation. + */ +static int +transformMergeJoinClause(ParseState *pstate, Node *merge, + List **mergeSourceTargetList) +{ + RangeTblEntry *rte, + *rt_rte; + List *namespace; + int rtindex, + rt_rtindex; + Node *n; + int mergeTarget_relation = list_length(pstate->p_rtable) + 1; + Var *var; + TargetEntry *te; + + n = transformFromClauseItem(pstate, merge, + &rte, + &rtindex, + &rt_rte, + &rt_rtindex, + &namespace); + + pstate->p_joinlist = list_make1(n); + + /* + * We created an internal join between the target and the source relation + * to carry out the MERGE actions. Normally such an unaliased join hides + * the joining relations, unless the column references are qualified. + * Also, any unqualified column refernces are resolved to the Join RTE, if + * there is a matching entry in the targetlist. But the way MERGE + * execution is later setup, we expect all column references to resolve to + * either the source or the target relation. Hence we must not add the + * Join RTE to the namespace. + * + * The last entry must be for the top-level Join RTE. We don't want to + * resolve any references to the Join RTE. So discard that. + * + * We also do not want to resolve any references from the leftside of the + * Join since that corresponds to the target relation. References to the + * columns of the target relation must be resolved from the result + * relation and not the one that is used in the join. So the + * mergeTarget_relation is marked invisible to both qualified as well as + * unqualified references. + */ + Assert(list_length(namespace) > 1); + namespace = list_truncate(namespace, list_length(namespace) - 1); + pstate->p_namespace = list_concat(pstate->p_namespace, namespace); + + setNamespaceVisibilityForRTE(pstate->p_namespace, + rt_fetch(mergeTarget_relation, pstate->p_rtable), false, false); + + /* + * Expand the right relation and add its columns to the + * mergeSourceTargetList. Note that the right relation can either be a + * plain relation or a subquery or anything that can have a + * RangeTableEntry. + */ + *mergeSourceTargetList = expandSourceTL(pstate, rt_rte, rt_rtindex); + + /* + * Add a whole-row-Var entry to support references to "source.*". + */ + var = makeWholeRowVar(rt_rte, rt_rtindex, 0, false); + te = makeTargetEntry((Expr *) var, list_length(*mergeSourceTargetList) + 1, + NULL, true); + *mergeSourceTargetList = lappend(*mergeSourceTargetList, te); + + return mergeTarget_relation; +} + +/* + * Make appropriate changes to the namespace visibility while transforming + * individual action's quals and targetlist expressions. In particular, for + * INSERT actions we must only see the source relation (since INSERT action is + * invoked for NOT MATCHED tuples and hence there is no target tuple to deal + * with). On the other hand, UPDATE and DELETE actions can see both source and + * target relations. + * + * Also, since the internal Join node can hide the source and target + * relations, we must explicitly make the respective relation as visible so + * that columns can be referenced unqualified from these relations. + */ +static void +setNamespaceForMergeAction(ParseState *pstate, MergeAction *action) +{ + RangeTblEntry *targetRelRTE, + *sourceRelRTE; + + /* Assume target relation is at index 1 */ + targetRelRTE = rt_fetch(1, pstate->p_rtable); + + /* + * Assume that the top-level join RTE is at the end. The source relation + * is just before that. + */ + sourceRelRTE = rt_fetch(list_length(pstate->p_rtable) - 1, pstate->p_rtable); + + switch (action->commandType) + { + case CMD_INSERT: + + /* + * Inserts can't see target relation, but they can see source + * relation. + */ + setNamespaceVisibilityForRTE(pstate->p_namespace, + targetRelRTE, false, false); + setNamespaceVisibilityForRTE(pstate->p_namespace, + sourceRelRTE, true, true); + break; + + case CMD_UPDATE: + case CMD_DELETE: + + /* + * Updates and deletes can see both target and source relations. + */ + setNamespaceVisibilityForRTE(pstate->p_namespace, + targetRelRTE, true, true); + setNamespaceVisibilityForRTE(pstate->p_namespace, + sourceRelRTE, true, true); + break; + + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } +} + +/* + * transformMergeStmt - + * transforms a MERGE statement + */ +Query * +transformMergeStmt(ParseState *pstate, MergeStmt *stmt) +{ + Query *qry = makeNode(Query); + ListCell *l; + AclMode targetPerms = ACL_NO_RIGHTS; + bool is_terminal[2]; + JoinExpr *joinexpr; + RangeTblEntry *resultRelRTE, *mergeRelRTE; + + /* There can't be any outer WITH to worry about */ + Assert(pstate->p_ctenamespace == NIL); + + qry->commandType = CMD_MERGE; + + /* + * Check WHEN clauses for permissions and sanity + */ + is_terminal[0] = false; + is_terminal[1] = false; + foreach(l, stmt->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + uint when_type = (action->matched ? 0 : 1); + + /* + * Collect action types so we can check Target permissions + */ + switch (action->commandType) + { + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + SelectStmt *selectStmt = (SelectStmt *) istmt->selectStmt; + + /* + * The grammar allows attaching ORDER BY, LIMIT, FOR + * UPDATE, or WITH to a VALUES clause and also multiple + * VALUES clauses. If we have any of those, ERROR. + */ + if (selectStmt && (selectStmt->valuesLists == NIL || + selectStmt->sortClause != NIL || + selectStmt->limitOffset != NULL || + selectStmt->limitCount != NULL || + selectStmt->lockingClause != NIL || + selectStmt->withClause != NULL)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("SELECT not allowed in MERGE INSERT statement"))); + + if (selectStmt && list_length(selectStmt->valuesLists) > 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("Multiple VALUES clauses not allowed in MERGE INSERT statement"))); + + targetPerms |= ACL_INSERT; + } + break; + case CMD_UPDATE: + targetPerms |= ACL_UPDATE; + break; + case CMD_DELETE: + targetPerms |= ACL_DELETE; + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } + + /* + * Check for unreachable WHEN clauses + */ + if (action->condition == NULL) + is_terminal[when_type] = true; + else if (is_terminal[when_type]) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unreachable WHEN clause specified after unconditional WHEN clause"))); + } + + /* + * Construct a query of the form SELECT relation.ctid --junk attribute + * ,relation.tableoid --junk attribute ,source_relation.<somecols> + * ,relation.<somecols> FROM relation RIGHT JOIN source_relation ON + * join_condition -- no WHERE clause - all conditions are applied in + * executor + * + * stmt->relation is the target relation, given as a RangeVar + * stmt->source_relation is a RangeVar or subquery + * + * We specify the join as a RIGHT JOIN as a simple way of forcing the + * first (larg) RTE to refer to the target table. + * + * The MERGE query's join can be tuned in some cases, see below for these + * special case tweaks. + * + * We set QSRC_PARSER to show query constructed in parse analysis + * + * Note that we have only one Query for a MERGE statement and the planner + * is called only once. That query is executed once to produce our stream + * of candidate change rows, so the query must contain all of the columns + * required by each of the targetlist or conditions for each action. + * + * As top-level statements INSERT, UPDATE and DELETE have a Query, whereas + * with MERGE the individual actions do not require separate planning, + * only different handling in the executor. See nodeModifyTable handling + * of commandType CMD_MERGE. + * + * A sub-query can include the Target, but otherwise the sub-query cannot + * reference the outermost Target table at all. + */ + qry->querySource = QSRC_PARSER; + + /* + * Setup the target table. Unlike regular UPDATE/DELETE, we don't expand + * inheritance for the target relation in case of MERGE. + * + * This special arrangement is required for handling partitioned tables + * because we perform an JOIN between the target and the source relation to + * identify the matching and not-matching rows. If we take the usual path + * of expanding the target table's inheritance and create one subplan per + * partition, then we we won't be able to correctly identify the matching + * and not-matching rows since for a given source row, there may not be a + * matching row in one partition, but it may exists in some other + * partition. So we must first append all the qualifying rows from all the + * partitions and then do the matching. + * + * Once a target row is returned by the underlying join, we find the + * correct partition and setup required state to carry out UPDATE/DELETE. + * All of this happens during execution. + */ + qry->resultRelation = setTargetTable(pstate, stmt->relation, + false, /* do not expand inheritance */ + true, targetPerms); + + /* + * Create a JOIN between the target and the source relation. + */ + joinexpr = makeNode(JoinExpr); + joinexpr->isNatural = false; + joinexpr->alias = NULL; + joinexpr->usingClause = NIL; + joinexpr->quals = stmt->join_condition; + joinexpr->larg = (Node *) stmt->relation; + joinexpr->rarg = (Node *) stmt->source_relation; + + /* + * Simplify the MERGE query as much as possible + * + * These seem like things that could go into Optimizer, but they are + * semantic simplications rather than optimizations, per se. + * + * If there are no INSERT actions we won't be using the non-matching + * candidate rows for anything, so no need for an outer join. We do still + * need an inner join for UPDATE and DELETE actions. + * + * Possible additional simplifications... + * + * XXX if we have a constant ON clause, we can skip join altogether + * + * XXX if we have a constant subquery, we can also skip join + * + * XXX if we were really keen we could look through the actionList and + * pull out common conditions, if there were no terminal clauses and put + * them into the main query as an early row filter but that seems like an + * atypical case and so checking for it would be likely to just be wasted + * effort. + */ + if (targetPerms & ACL_INSERT) + joinexpr->jointype = JOIN_RIGHT; + else + joinexpr->jointype = JOIN_INNER; + + /* + * We use a special purpose transformation here because the normal + * routines don't quite work right for the MERGE case. + * + * A special mergeSourceTargetList is setup by transformMergeJoinClause(). + * It refers to all the attributes provided by the source relation. This + * is later used by set_plan_refs() to fix the UPDATE/INSERT target lists + * to so that they can correctly fetch the attributes from the source + * relation. + * + * The target relation when used in the underlying join, gets a new RTE + * with rte->inh set to true. We remember this RTE (and later pass on to + * the planner and executor) for two main reasons: + * + * 1. If we ever need to run EvalPlanQual while performing MERGE, we must + * make the modified tuple available to the underlying join query, which is + * using a different RTE from the resultRelation RTE. + * + * 2. rewriteTargetListMerge() requires the RTE of the underlying join in + * order to add junk CTID and TABLEOID attributes. + */ + qry->mergeTarget_relation = transformMergeJoinClause(pstate, (Node *) joinexpr, + &qry->mergeSourceTargetList); + + /* + * The target table referenced in the MERGE is looked up twice; once while + * setting it up as the result relation and again when it's used in the + * underlying the join query. In some rare situations, it may happen that + * these lookups return different results, for example, if a new relation + * with the same name gets created in a schema which is ahead in the + * search_path, in between the two lookups. + * + * It's a very narrow case, but nevertheless we guard against it by simply + * checking if the OIDs returned by the two lookups is the same. If not, we + * just throw an error. + */ + Assert(qry->resultRelation > 0); + Assert(qry->mergeTarget_relation > 0); + + /* Fetch both the RTEs */ + resultRelRTE = rt_fetch(qry->resultRelation, pstate->p_rtable); + mergeRelRTE = rt_fetch(qry->mergeTarget_relation, pstate->p_rtable); + + if (resultRelRTE->relid != mergeRelRTE->relid) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("relation referenced by MERGE statement has changed"))); + + /* + * This query should just provide the source relation columns. Later, in + * preprocess_targetlist(), we shall also add "ctid" attribute of the + * target relation to ensure that the target tuple can be fetched + * correctly. + */ + qry->targetList = qry->mergeSourceTargetList; + + /* qry has no WHERE clause so absent quals are shown as NULL */ + qry->jointree = makeFromExpr(pstate->p_joinlist, NULL); + qry->rtable = pstate->p_rtable; + + /* + * XXX MERGE is unsupported in various cases + */ + if (!(pstate->p_target_relation->rd_rel->relkind == RELKIND_RELATION || + pstate->p_target_relation->rd_rel->relkind == RELKIND_MATVIEW || + pstate->p_target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for this relation type"))); + + if (pstate->p_target_relation->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && + pstate->p_target_relation->rd_rel->relhassubclass) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for relations with inheritance"))); + + if (pstate->p_target_relation->rd_rel->relhasrules) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for relations with rules"))); + + /* + * We now have a good query shape, so now look at the when conditions and + * action targetlists. + * + * Overall, the MERGE Query's targetlist is NIL. + * + * Each individual action has its own targetlist that needs separate + * transformation. These transforms don't do anything to the overall + * targetlist, since that is only used for resjunk columns. + * + * We can reference any column in Target or Source, which is OK because + * both of those already have RTEs. There is nothing like the EXCLUDED + * pseudo-relation for INSERT ON CONFLICT. + */ + foreach(l, stmt->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + /* + * Set namespace for the specific action. This must be done before + * analysing the WHEN quals and the action targetlisst. + */ + setNamespaceForMergeAction(pstate, action); + + /* + * Transform the when condition. + * + * Note that these quals are NOT added to the join quals; instead they + * are evaluated sepaartely during execution to decide which of the + * WHEN MATCHED or WHEN NOT MATCHED actions to execute. + */ + action->qual = transformWhereClause(pstate, action->condition, + EXPR_KIND_MERGE_WHEN_AND, "WHEN"); + + /* + * Transform target lists for each INSERT and UPDATE action stmt + */ + switch (action->commandType) + { + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + SelectStmt *selectStmt = (SelectStmt *) istmt->selectStmt; + List *exprList = NIL; + ListCell *lc; + RangeTblEntry *rte; + ListCell *icols; + ListCell *attnos; + List *icolumns; + List *attrnos; + + pstate->p_is_insert = true; + + icolumns = checkInsertTargets(pstate, istmt->cols, &attrnos); + Assert(list_length(icolumns) == list_length(attrnos)); + + /* + * Handle INSERT much like in transformInsertStmt + */ + if (selectStmt == NULL) + { + /* + * We have INSERT ... DEFAULT VALUES. We can handle + * this case by emitting an empty targetlist --- all + * columns will be defaulted when the planner expands + * the targetlist. + */ + exprList = NIL; + } + else + { + /* + * Process INSERT ... VALUES with a single VALUES + * sublist. We treat this case separately for + * efficiency. The sublist is just computed directly + * as the Query's targetlist, with no VALUES RTE. So + * it works just like a SELECT without any FROM. + */ + List *valuesLists = selectStmt->valuesLists; + + Assert(list_length(valuesLists) == 1); + Assert(selectStmt->intoClause == NULL); + + /* + * Do basic expression transformation (same as a ROW() + * expr, but allow SetToDefault at top level) + */ + exprList = transformExpressionList(pstate, + (List *) linitial(valuesLists), + EXPR_KIND_VALUES_SINGLE, + true); + + /* Prepare row for assignment to target table */ + exprList = transformInsertRow(pstate, exprList, + istmt->cols, + icolumns, attrnos, + false); + } + + /* + * Generate action's target list using the computed list + * of expressions. Also, mark all the target columns as + * needing insert permissions. + */ + rte = pstate->p_target_rangetblentry; + icols = list_head(icolumns); + attnos = list_head(attrnos); + foreach(lc, exprList) + { + Expr *expr = (Expr *) lfirst(lc); + ResTarget *col; + AttrNumber attr_num; + TargetEntry *tle; + + col = lfirst_node(ResTarget, icols); + attr_num = (AttrNumber) lfirst_int(attnos); + + tle = makeTargetEntry(expr, + attr_num, + col->name, + false); + action->targetList = lappend(action->targetList, tle); + + rte->insertedCols = bms_add_member(rte->insertedCols, + attr_num - FirstLowInvalidHeapAttributeNumber); + + icols = lnext(icols); + attnos = lnext(attnos); + } + } + break; + case CMD_UPDATE: + { + UpdateStmt *ustmt = (UpdateStmt *) action->stmt; + + pstate->p_is_insert = false; + action->targetList = transformUpdateTargetList(pstate, ustmt->targetList); + } + break; + case CMD_DELETE: + break; + + case CMD_NOTHING: + action->targetList = NIL; + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } + } + + qry->mergeActionList = stmt->mergeActionList; + + /* XXX maybe later */ + qry->returningList = NULL; + + qry->hasTargetSRFs = false; + qry->hasSubLinks = pstate->p_hasSubLinks; + + assign_query_collations(pstate, qry); + + return qry; +} + +static void +setNamespaceVisibilityForRTE(List *namespace, RangeTblEntry *rte, + bool rel_visible, + bool cols_visible) +{ + ListCell *lc; + + foreach(lc, namespace) + { + ParseNamespaceItem *nsitem = (ParseNamespaceItem *) lfirst(lc); + + if (nsitem->p_rte == rte) + { + nsitem->p_rel_visible = rel_visible; + nsitem->p_cols_visible = cols_visible; + break; + } + } + +} + +/* + * Expand the source relation to include all attributes of this RTE. + * + * This function is very similar to expandRelAttrs except that we don't mark + * columns for SELECT privileges. That will be decided later when we transform + * the action targetlists and the WHEN quals for actual references to the + * source relation. + */ +static List * +expandSourceTL(ParseState *pstate, RangeTblEntry *rte, int rtindex) +{ + List *names, + *vars; + ListCell *name, + *var; + List *te_list = NIL; + + expandRTE(rte, rtindex, 0, -1, false, &names, &vars); + + /* + * Require read access to the table. + */ + rte->requiredPerms |= ACL_SELECT; + + forboth(name, names, var, vars) + { + char *label = strVal(lfirst(name)); + Var *varnode = (Var *) lfirst(var); + TargetEntry *te; + + te = makeTargetEntry((Expr *) varnode, + (AttrNumber) pstate->p_next_resno++, + label, + false); + te_list = lappend(te_list, te); + } + + Assert(name == NULL && var == NULL); /* lists not the same length? */ + + return te_list; +} diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c index 053ae02c9f..5583404e1b 100644 --- a/src/backend/parser/parse_relation.c +++ b/src/backend/parser/parse_relation.c @@ -728,6 +728,16 @@ scanRTEForColumn(ParseState *pstate, RangeTblEntry *rte, const char *colname, colname), parser_errposition(pstate, location))); + /* In MERGE when and condition, no system column is allowed */ + if (pstate->p_expr_kind == EXPR_KIND_MERGE_WHEN_AND && + attnum < InvalidAttrNumber && + !(attnum == TableOidAttributeNumber || attnum == ObjectIdAttributeNumber)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_COLUMN_REFERENCE), + errmsg("system column \"%s\" reference in WHEN AND condition is invalid", + colname), + parser_errposition(pstate, location))); + if (attnum != InvalidAttrNumber) { /* now check to see if column actually is defined */ diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index 66253fc3d3..45875ec289 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1376,6 +1376,53 @@ rewriteTargetListUD(Query *parsetree, RangeTblEntry *target_rte, } } +void +rewriteTargetListMerge(Query *parsetree, Relation target_relation) +{ + Var *var = NULL; + const char *attrname; + TargetEntry *tle; + + Assert(target_relation->rd_rel->relkind == RELKIND_RELATION || + target_relation->rd_rel->relkind == RELKIND_MATVIEW || + target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE); + + /* + * Emit CTID so that executor can find the row to update or delete. + */ + var = makeVar(parsetree->mergeTarget_relation, + SelfItemPointerAttributeNumber, + TIDOID, + -1, + InvalidOid, + 0); + + attrname = "ctid"; + tle = makeTargetEntry((Expr *) var, + list_length(parsetree->targetList) + 1, + pstrdup(attrname), + true); + + parsetree->targetList = lappend(parsetree->targetList, tle); + + /* + * Emit TABLEOID so that executor can find the row to update or delete. + */ + var = makeVar(parsetree->mergeTarget_relation, + TableOidAttributeNumber, + OIDOID, + -1, + InvalidOid, + 0); + + attrname = "tableoid"; + tle = makeTargetEntry((Expr *) var, + list_length(parsetree->targetList) + 1, + pstrdup(attrname), + true); + + parsetree->targetList = lappend(parsetree->targetList, tle); +} /* * matchLocks - @@ -3330,6 +3377,7 @@ RewriteQuery(Query *parsetree, List *rewrite_events) } else if (event == CMD_UPDATE) { + Assert(parsetree->override == OVERRIDING_NOT_SET); parsetree->targetList = rewriteTargetListIU(parsetree->targetList, parsetree->commandType, @@ -3337,6 +3385,50 @@ RewriteQuery(Query *parsetree, List *rewrite_events) rt_entry_relation, parsetree->resultRelation, NULL); } + else if (event == CMD_MERGE) + { + Assert(parsetree->override == OVERRIDING_NOT_SET); + + /* + * Rewrite each action targetlist separately + */ + foreach(lc1, parsetree->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(lc1); + + switch (action->commandType) + { + case CMD_NOTHING: + case CMD_DELETE: /* Nothing to do here */ + break; + case CMD_UPDATE: + action->targetList = + rewriteTargetListIU(action->targetList, + action->commandType, + parsetree->override, + rt_entry_relation, + parsetree->resultRelation, + NULL); + break; + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + + action->targetList = + rewriteTargetListIU(action->targetList, + action->commandType, + istmt->override, + rt_entry_relation, + parsetree->resultRelation, + NULL); + } + break; + default: + elog(ERROR, "unrecognized commandType: %d", action->commandType); + break; + } + } + } else if (event == CMD_DELETE) { /* Nothing to do here */ @@ -3350,13 +3442,19 @@ RewriteQuery(Query *parsetree, List *rewrite_events) locks = matchLocks(event, rt_entry_relation->rd_rules, result_relation, parsetree, &hasUpdate); - product_queries = fireRules(parsetree, - result_relation, - event, - locks, - &instead, - &returning, - &qual_product); + /* + * MERGE doesn't support rules as it's unclear how that could work. + */ + if (event == CMD_MERGE) + product_queries = NIL; + else + product_queries = fireRules(parsetree, + result_relation, + event, + locks, + &instead, + &returning, + &qual_product); /* * If there were no INSTEAD rules, and the target relation is a view diff --git a/src/backend/rewrite/rowsecurity.c b/src/backend/rewrite/rowsecurity.c index ce77a18bc9..6e85886e64 100644 --- a/src/backend/rewrite/rowsecurity.c +++ b/src/backend/rewrite/rowsecurity.c @@ -379,6 +379,95 @@ get_row_security_policies(Query *root, RangeTblEntry *rte, int rt_index, } } + /* + * FOR MERGE, we fetch policies for UPDATE, DELETE and INSERT (and ALL) + * and set them up so that we can enforce the appropriate policy depending + * on the final action we take. + * + * We don't fetch the SELECT policies since they are correctly applied to + * the root->mergeTarget_relation. The target rows are selected after + * joining the mergeTarget_relation and the source relation and hence it's + * enough to apply SELECT policies to the mergeTarget_relation. + * + * We don't push the UPDATE/DELETE USING quals to the RTE because we don't + * really want to apply them while scanning the relation since we don't + * know whether we will be doing a UPDATE or a DELETE at the end. We apply + * the respective policy once we decide the final action on the target + * tuple. + * + * XXX We are setting up USING quals as WITH CHECK. If RLS prohibits + * UPDATE/DELETE on the target row, we shall throw an error instead of + * silently ignoring the row. This is different than how normal + * UPDATE/DELETE works and more in line with INSERT ON CONFLICT DO UPDATE + * handling. + */ + if (commandType == CMD_MERGE) + { + List *merge_permissive_policies; + List *merge_restrictive_policies; + + /* + * Fetch the UPDATE policies and set them up to execute on the + * existing target row before doing UPDATE. + */ + get_policies_for_relation(rel, CMD_UPDATE, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + /* + * WCO_RLS_MERGE_UPDATE_CHECK is used to check UPDATE USING quals on + * the existing target row. + */ + add_with_check_options(rel, rt_index, + WCO_RLS_MERGE_UPDATE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + true); + + /* + * Same with DELETE policies. + */ + get_policies_for_relation(rel, CMD_DELETE, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + add_with_check_options(rel, rt_index, + WCO_RLS_MERGE_DELETE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + true); + + /* + * No special handling is required for INSERT policies. They will be + * checked and enforced during ExecInsert(). But we must add them to + * withCheckOptions. + */ + get_policies_for_relation(rel, CMD_INSERT, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + add_with_check_options(rel, rt_index, + WCO_RLS_INSERT_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + false); + + /* Enforce the WITH CHECK clauses of the UPDATE policies */ + add_with_check_options(rel, rt_index, + WCO_RLS_UPDATE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + false); + } + heap_close(rel, NoLock); /* @@ -438,6 +527,14 @@ get_policies_for_relation(Relation relation, CmdType cmd, Oid user_id, if (policy->polcmd == ACL_DELETE_CHR) cmd_matches = true; break; + case CMD_MERGE: + + /* + * We do not support a separate policy for MERGE command. + * Instead it derives from the policies defined for other + * commands. + */ + break; default: elog(ERROR, "unrecognized policy command type %d", (int) cmd); diff --git a/src/backend/tcop/pquery.c b/src/backend/tcop/pquery.c index 66cc5c35c6..50f852a4aa 100644 --- a/src/backend/tcop/pquery.c +++ b/src/backend/tcop/pquery.c @@ -193,6 +193,11 @@ ProcessQuery(PlannedStmt *plan, "DELETE " UINT64_FORMAT, queryDesc->estate->es_processed); break; + case CMD_MERGE: + snprintf(completionTag, COMPLETION_TAG_BUFSIZE, + "MERGE " UINT64_FORMAT, + queryDesc->estate->es_processed); + break; default: strcpy(completionTag, "???"); break; diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index e144583bd1..2521944b9d 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -110,6 +110,7 @@ CommandIsReadOnly(PlannedStmt *pstmt) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: return false; case CMD_UTILITY: /* For now, treat all utility commands as read/write */ @@ -1832,6 +1833,8 @@ QueryReturnsTuples(Query *parsetree) case CMD_SELECT: /* returns tuples */ return true; + case CMD_MERGE: + return false; case CMD_INSERT: case CMD_UPDATE: case CMD_DELETE: @@ -2076,6 +2079,10 @@ CreateCommandTag(Node *parsetree) tag = "UPDATE"; break; + case T_MergeStmt: + tag = "MERGE"; + break; + case T_SelectStmt: tag = "SELECT"; break; @@ -2819,6 +2826,9 @@ CreateCommandTag(Node *parsetree) case CMD_DELETE: tag = "DELETE"; break; + case CMD_MERGE: + tag = "MERGE"; + break; case CMD_UTILITY: tag = CreateCommandTag(stmt->utilityStmt); break; @@ -2879,6 +2889,9 @@ CreateCommandTag(Node *parsetree) case CMD_DELETE: tag = "DELETE"; break; + case CMD_MERGE: + tag = "MERGE"; + break; case CMD_UTILITY: tag = CreateCommandTag(stmt->utilityStmt); break; @@ -2927,6 +2940,7 @@ GetCommandLogLevel(Node *parsetree) case T_InsertStmt: case T_DeleteStmt: case T_UpdateStmt: + case T_MergeStmt: lev = LOGSTMT_MOD; break; @@ -3366,6 +3380,7 @@ GetCommandLogLevel(Node *parsetree) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: lev = LOGSTMT_MOD; break; @@ -3396,6 +3411,7 @@ GetCommandLogLevel(Node *parsetree) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: lev = LOGSTMT_MOD; break; diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 4c0256b18a..100174138d 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -67,6 +67,7 @@ typedef enum LockTupleMode */ typedef struct HeapUpdateFailureData { + HTSU_Result result; ItemPointerData ctid; TransactionId xmax; CommandId cmax; diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h index a5b8610fa2..1b79a80310 100644 --- a/src/include/commands/trigger.h +++ b/src/include/commands/trigger.h @@ -206,7 +206,8 @@ extern bool ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, - HeapTuple fdw_trigtuple); + HeapTuple fdw_trigtuple, + HeapUpdateFailureData *hufdp); extern void ExecARDeleteTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, @@ -225,7 +226,8 @@ extern TupleTableSlot *ExecBRUpdateTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, HeapTuple fdw_trigtuple, - TupleTableSlot *slot); + TupleTableSlot *slot, + HeapUpdateFailureData *hufdp); extern void ExecARUpdateTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, diff --git a/src/include/executor/execPartition.h b/src/include/executor/execPartition.h index 03a599ad57..9f55f6409e 100644 --- a/src/include/executor/execPartition.h +++ b/src/include/executor/execPartition.h @@ -114,6 +114,7 @@ extern int ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, TupleTableSlot *slot, EState *estate); +extern int ExecFindPartitionByOid(PartitionTupleRouting *proute, Oid partoid); extern ResultRelInfo *ExecInitPartitionInfo(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, PartitionTupleRouting *proute, diff --git a/src/include/executor/nodeMerge.h b/src/include/executor/nodeMerge.h new file mode 100644 index 0000000000..c222e9ee65 --- /dev/null +++ b/src/include/executor/nodeMerge.h @@ -0,0 +1,22 @@ +/*------------------------------------------------------------------------- + * + * nodeMerge.h + * + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/executor/nodeMerge.h + * + *------------------------------------------------------------------------- + */ +#ifndef NODEMERGE_H +#define NODEMERGE_H + +#include "nodes/execnodes.h" + +extern void +ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, + JunkFilter *junkfilter, ResultRelInfo *resultRelInfo); + +#endif /* NODEMERGE_H */ diff --git a/src/include/executor/nodeModifyTable.h b/src/include/executor/nodeModifyTable.h index 0d7e579e1c..686cfa6171 100644 --- a/src/include/executor/nodeModifyTable.h +++ b/src/include/executor/nodeModifyTable.h @@ -18,5 +18,26 @@ extern ModifyTableState *ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags); extern void ExecEndModifyTable(ModifyTableState *node); extern void ExecReScanModifyTable(ModifyTableState *node); +extern TupleTableSlot *ExecPrepareTupleRouting(ModifyTableState *mtstate, + EState *estate, + struct PartitionTupleRouting *proute, + ResultRelInfo *targetRelInfo, + TupleTableSlot *slot); +extern TupleTableSlot *ExecDelete(ModifyTableState *mtstate, + ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *planSlot, + EPQState *epqstate, EState *estate, bool *tupleDeleted, + bool processReturning, HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag); +extern TupleTableSlot *ExecUpdate(ModifyTableState *mtstate, + ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *slot, + TupleTableSlot *planSlot, EPQState *epqstate, EState *estate, + bool *tuple_updated, HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag); +extern TupleTableSlot *ExecInsert(ModifyTableState *mtstate, + TupleTableSlot *slot, + TupleTableSlot *planSlot, + EState *estate, + MergeActionState *actionState, + bool canSetTag); #endif /* NODEMODIFYTABLE_H */ diff --git a/src/include/executor/spi.h b/src/include/executor/spi.h index e5bdaecc4e..78410b9f77 100644 --- a/src/include/executor/spi.h +++ b/src/include/executor/spi.h @@ -64,6 +64,7 @@ typedef struct _SPI_plan *SPIPlanPtr; #define SPI_OK_REL_REGISTER 15 #define SPI_OK_REL_UNREGISTER 16 #define SPI_OK_TD_REGISTER 17 +#define SPI_OK_MERGE 18 #define SPI_OPT_NONATOMIC (1 << 0) diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 6070a42b6f..0bf1d7aeb6 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -360,8 +360,17 @@ typedef struct JunkFilter AttrNumber *jf_cleanMap; TupleTableSlot *jf_resultSlot; AttrNumber jf_junkAttNo; + AttrNumber jf_otherJunkAttNo; } JunkFilter; +typedef struct MergeState +{ + /* List of MERGE MATCHED action states */ + List *matchedActionStates; + /* List of MERGE NOT MATCHED action states */ + List *notMatchedActionStates; +} MergeState; + /* * OnConflictSetState * @@ -452,8 +461,38 @@ typedef struct ResultRelInfo /* relation descriptor for root partitioned table */ Relation ri_PartitionRoot; + + int ri_PartitionLeafIndex; + /* for running MERGE on this result relation */ + MergeState *ri_mergeState; + + /* + * While executing MERGE, the target relation is processed twice; once to + * as a result relation and to run a join between the target and the + * source. We generate two different RTEs for these two purposes, one with + * rte->inh set to false and other with rte->inh set to true. + * + * Since the plan re-evaluated by EvalPlanQual uses the second RTE, we must + * install the updated tuple in the scan corresponding to that RTE. The + * following member tracks the index of the second RTE for EvalPlanQual + * purposes. ri_mergeTargetRTI is non-zero only when MERGE is in-progress. + * We use ri_mergeTargetRTI to run EvalPlanQual for MERGE and + * ri_RangeTableIndex elsewhere. + */ + Index ri_mergeTargetRTI; } ResultRelInfo; +/* + * Get the Range table index for EvalPlanQual. + * + * We use the ri_mergeTargetRTI if set, otherwise use ri_RangeTableIndex. + * ri_mergeTargetRTI should really be ever set iff we're running MERGE. + */ +#define GetEPQRangeTableIndex(r) \ + (((r)->ri_mergeTargetRTI > 0) \ + ? (r)->ri_mergeTargetRTI \ + : (r)->ri_RangeTableIndex) + /* ---------------- * EState information * @@ -1012,6 +1051,24 @@ typedef struct ProjectSetState MemoryContext argcontext; /* context for SRF arguments */ } ProjectSetState; +/* ---------------- + * MergeActionState information + * ---------------- + */ +typedef struct MergeActionState +{ + NodeTag type; + bool matched; /* true if a WHEN MATCHED action, + * false if a NOT MATCHED action. */ + ExprState *whenqual; /* additional conditions attached to + * WHEN [NOT] MATCHED clause */ + CmdType commandType; /* INSERT/UPDATE/DELETE/DO NOTHING */ + ProjectionInfo *proj; /* projection information for the tuple + * produced by this action */ + TupleDesc tupDesc; /* tuple descriptor associated with the + * projection */ +} MergeActionState; + /* ---------------- * ModifyTableState information * ---------------- @@ -1019,7 +1076,7 @@ typedef struct ProjectSetState typedef struct ModifyTableState { PlanState ps; /* its first field is NodeTag */ - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ bool mt_done; /* are we done? */ PlanState **mt_plans; /* subplans (one per target rel) */ @@ -1035,6 +1092,8 @@ typedef struct ModifyTableState List *mt_excludedtlist; /* the excluded pseudo relation's tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection target */ + TupleTableSlot *mt_mergeproj; /* MERGE action projection target */ + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; @@ -1046,6 +1105,9 @@ typedef struct ModifyTableState /* Per plan map for tuple conversion from child to root */ TupleConversionMap **mt_per_subplan_tupconv_maps; + + int mt_merge_subcommands; /* Flags show which cmd types are + * present */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 443de22704..fce48026b6 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -97,6 +97,7 @@ typedef enum NodeTag T_PlanState, T_ResultState, T_ProjectSetState, + T_MergeActionState, T_ModifyTableState, T_AppendState, T_MergeAppendState, @@ -308,6 +309,8 @@ typedef enum NodeTag T_InsertStmt, T_DeleteStmt, T_UpdateStmt, + T_MergeStmt, + T_MergeAction, T_SelectStmt, T_AlterTableStmt, T_AlterTableCmd, @@ -657,7 +660,8 @@ typedef enum CmdType CMD_SELECT, /* select stmt */ CMD_UPDATE, /* update stmt */ CMD_INSERT, /* insert stmt */ - CMD_DELETE, + CMD_DELETE, /* delete stmt */ + CMD_MERGE, /* merge stmt */ CMD_UTILITY, /* cmds like create, destroy, copy, vacuum, * etc. */ CMD_NOTHING /* dummy command for instead nothing rules diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 92082b3a7a..0c904f4d7f 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -38,7 +38,7 @@ typedef enum OverridingKind typedef enum QuerySource { QSRC_ORIGINAL, /* original parsetree (explicit query) */ - QSRC_PARSER, /* added by parse analysis (now unused) */ + QSRC_PARSER, /* added by parse analysis in MERGE */ QSRC_INSTEAD_RULE, /* added by unconditional INSTEAD rule */ QSRC_QUAL_INSTEAD_RULE, /* added by conditional INSTEAD rule */ QSRC_NON_INSTEAD_RULE /* added by non-INSTEAD rule */ @@ -107,7 +107,7 @@ typedef struct Query { NodeTag type; - CmdType commandType; /* select|insert|update|delete|utility */ + CmdType commandType; /* select|insert|update|delete|merge|utility */ QuerySource querySource; /* where did I come from? */ @@ -118,7 +118,7 @@ typedef struct Query Node *utilityStmt; /* non-null if commandType == CMD_UTILITY */ int resultRelation; /* rtable index of target relation for - * INSERT/UPDATE/DELETE; 0 for SELECT */ + * INSERT/UPDATE/DELETE/MERGE; 0 for SELECT */ bool hasAggs; /* has aggregates in tlist or havingQual */ bool hasWindowFuncs; /* has window functions in tlist */ @@ -169,6 +169,9 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's, which are * only added during rewrite and therefore * are not written out as part of Query. */ + int mergeTarget_relation; + List *mergeSourceTargetList; + List *mergeActionList; /* list of actions for MERGE (only) */ /* * The following two fields identify the portion of the source text string @@ -1128,7 +1131,9 @@ typedef enum WCOKind WCO_VIEW_CHECK, /* WCO on an auto-updatable view */ WCO_RLS_INSERT_CHECK, /* RLS INSERT WITH CHECK policy */ WCO_RLS_UPDATE_CHECK, /* RLS UPDATE WITH CHECK policy */ - WCO_RLS_CONFLICT_CHECK /* RLS ON CONFLICT DO UPDATE USING policy */ + WCO_RLS_CONFLICT_CHECK, /* RLS ON CONFLICT DO UPDATE USING policy */ + WCO_RLS_MERGE_UPDATE_CHECK, /* RLS MERGE UPDATE USING policy */ + WCO_RLS_MERGE_DELETE_CHECK /* RLS MERGE DELETE USING policy */ } WCOKind; typedef struct WithCheckOption @@ -1503,6 +1508,32 @@ typedef struct UpdateStmt WithClause *withClause; /* WITH clause */ } UpdateStmt; +/* ---------------------- + * Merge Statement + * ---------------------- + */ +typedef struct MergeStmt +{ + NodeTag type; + RangeVar *relation; /* target relation to merge */ + Node *source_relation; /* source relation */ + Node *join_condition; /* join condition between source and target */ + List *mergeActionList; /* list of MergeAction(s) */ +} MergeStmt; + +typedef struct MergeAction +{ + NodeTag type; + bool matched; /* true if a WHEN MATCHED action, + * false if a WHEN NOT MATCHED action */ + Node *condition; /* WHEN AND conditions (raw parser) */ + Node *qual; /* transformed WHEN AND conditions */ + CmdType commandType; /* type of action - INSERT/UPDATE/DELETE/DO + * NOTHING */ + Node *stmt; /* T_UpdateStmt etc */ + List *targetList; /* the target list (of ResTarget) */ +} MergeAction; + /* ---------------------- * Select Statement * diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index c922216b7d..0a797f0a05 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -18,6 +18,7 @@ #include "lib/stringinfo.h" #include "nodes/bitmapset.h" #include "nodes/lockoptions.h" +#include "nodes/parsenodes.h" #include "nodes/primnodes.h" @@ -42,7 +43,7 @@ typedef struct PlannedStmt { NodeTag type; - CmdType commandType; /* select|insert|update|delete|utility */ + CmdType commandType; /* select|insert|update|delete|merge|utility */ uint64 queryId; /* query identifier (copied from Query) */ @@ -216,13 +217,14 @@ typedef struct ProjectSet typedef struct ModifyTable { Plan plan; - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ Index nominalRelation; /* Parent RT index for use of EXPLAIN */ /* RT indexes of non-leaf tables in a partition tree */ List *partitioned_rels; bool partColsUpdated; /* some part key in hierarchy updated */ List *resultRelations; /* integer list of RT indexes */ + Index mergeTargetRelation; /* RT index of the merge target */ int resultRelIndex; /* index of first resultRel in plan's list */ int rootResultRelIndex; /* index of the partitioned table root */ List *plans; /* plan(s) producing source data */ @@ -238,6 +240,8 @@ typedef struct ModifyTable Node *onConflictWhere; /* WHERE for ON CONFLICT UPDATE */ Index exclRelRTI; /* RTI of the EXCLUDED pseudo relation */ List *exclRelTlist; /* tlist of the EXCLUDED pseudo relation */ + List *mergeSourceTargetList; + List *mergeActionList; /* actions for MERGE */ } ModifyTable; /* ---------------- diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h index abbbda9e91..91dfff4cb5 100644 --- a/src/include/nodes/relation.h +++ b/src/include/nodes/relation.h @@ -1670,7 +1670,7 @@ typedef struct LockRowsPath } LockRowsPath; /* - * ModifyTablePath represents performing INSERT/UPDATE/DELETE modifications + * ModifyTablePath represents performing INSERT/UPDATE/DELETE/MERGE * * We represent most things that will be in the ModifyTable plan node * literally, except we have child Path(s) not Plan(s). But analysis of the @@ -1679,13 +1679,14 @@ typedef struct LockRowsPath typedef struct ModifyTablePath { Path path; - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ Index nominalRelation; /* Parent RT index for use of EXPLAIN */ /* RT indexes of non-leaf tables in a partition tree */ List *partitioned_rels; bool partColsUpdated; /* some part key in hierarchy updated */ List *resultRelations; /* integer list of RT indexes */ + Index mergeTargetRelation;/* RT index of merge target relation */ List *subpaths; /* Path(s) producing source data */ List *subroots; /* per-target-table PlannerInfos */ List *withCheckOptionLists; /* per-target-table WCO lists */ @@ -1693,6 +1694,8 @@ typedef struct ModifyTablePath List *rowMarks; /* PlanRowMarks (non-locking only) */ OnConflictExpr *onconflict; /* ON CONFLICT clause, or NULL */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ + List *mergeSourceTargetList; + List *mergeActionList; /* actions for MERGE */ } ModifyTablePath; /* diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index 381bc30813..895bf6959d 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -241,11 +241,14 @@ extern ModifyTablePath *create_modifytable_path(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subpaths, + List *resultRelations, + Index mergeTargetRelation, + List *subpaths, List *subroots, List *withCheckOptionLists, List *returningLists, List *rowMarks, OnConflictExpr *onconflict, - int epqParam); + List *mergeSourceTargetList, + List *mergeActionList, int epqParam); extern LimitPath *create_limit_path(PlannerInfo *root, RelOptInfo *rel, Path *subpath, Node *limitOffset, Node *limitCount, diff --git a/src/include/parser/analyze.h b/src/include/parser/analyze.h index 687ae1b5b7..41fb10666e 100644 --- a/src/include/parser/analyze.h +++ b/src/include/parser/analyze.h @@ -32,6 +32,11 @@ extern Query *parse_sub_analyze(Node *parseTree, ParseState *parentParseState, bool locked_from_parent, bool resolve_unknowns); +extern List *transformInsertRow(ParseState *pstate, List *exprlist, + List *stmtcols, List *icolumns, List *attrnos, + bool strip_indirection); +extern List *transformUpdateTargetList(ParseState *pstate, + List *targetList); extern Query *transformTopLevelStmt(ParseState *pstate, RawStmt *parseTree); extern Query *transformStmt(ParseState *pstate, Node *parseTree); diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index cf32197bc3..4dff55a8e9 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -244,8 +244,10 @@ PG_KEYWORD("locked", LOCKED, UNRESERVED_KEYWORD) PG_KEYWORD("logged", LOGGED, UNRESERVED_KEYWORD) PG_KEYWORD("mapping", MAPPING, UNRESERVED_KEYWORD) PG_KEYWORD("match", MATCH, UNRESERVED_KEYWORD) +PG_KEYWORD("matched", MATCHED, UNRESERVED_KEYWORD) PG_KEYWORD("materialized", MATERIALIZED, UNRESERVED_KEYWORD) PG_KEYWORD("maxvalue", MAXVALUE, UNRESERVED_KEYWORD) +PG_KEYWORD("merge", MERGE, UNRESERVED_KEYWORD) PG_KEYWORD("method", METHOD, UNRESERVED_KEYWORD) PG_KEYWORD("minute", MINUTE_P, UNRESERVED_KEYWORD) PG_KEYWORD("minvalue", MINVALUE, UNRESERVED_KEYWORD) diff --git a/src/include/parser/parse_clause.h b/src/include/parser/parse_clause.h index 2c0e092862..30121c98ed 100644 --- a/src/include/parser/parse_clause.h +++ b/src/include/parser/parse_clause.h @@ -20,7 +20,10 @@ extern void transformFromClause(ParseState *pstate, List *frmList); extern int setTargetTable(ParseState *pstate, RangeVar *relation, bool inh, bool alsoSource, AclMode requiredPerms); extern bool interpretOidsOption(List *defList, bool allowOids); - +extern Node *transformFromClauseItem(ParseState *pstate, Node *n, + RangeTblEntry **top_rte, int *top_rti, + RangeTblEntry **right_rte, int *right_rti, + List **namespace); extern Node *transformWhereClause(ParseState *pstate, Node *clause, ParseExprKind exprKind, const char *constructName); extern Node *transformLimitClause(ParseState *pstate, Node *clause, diff --git a/src/include/parser/parse_merge.h b/src/include/parser/parse_merge.h new file mode 100644 index 0000000000..0151809e09 --- /dev/null +++ b/src/include/parser/parse_merge.h @@ -0,0 +1,19 @@ +/*------------------------------------------------------------------------- + * + * parse_merge.h + * handle merge-stmt in parser + * + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/parser/parse_merge.h + * + *------------------------------------------------------------------------- + */ +#ifndef PARSE_MERGE_H +#define PARSE_MERGE_H + +#include "parser/parse_node.h" +extern Query *transformMergeStmt(ParseState *pstate, MergeStmt *stmt); +#endif diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index 0230543810..8b80e79fd2 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -50,6 +50,7 @@ typedef enum ParseExprKind EXPR_KIND_INSERT_TARGET, /* INSERT target list item */ EXPR_KIND_UPDATE_SOURCE, /* UPDATE assignment source item */ EXPR_KIND_UPDATE_TARGET, /* UPDATE assignment target item */ + EXPR_KIND_MERGE_WHEN_AND, /* MERGE WHEN ... AND condition */ EXPR_KIND_GROUP_BY, /* GROUP BY */ EXPR_KIND_ORDER_BY, /* ORDER BY */ EXPR_KIND_DISTINCT_ON, /* DISTINCT ON */ @@ -127,7 +128,8 @@ typedef Node *(*CoerceParamHook) (ParseState *pstate, Param *param, * p_parent_cte: CommonTableExpr that immediately contains the current query, * if any. * - * p_target_relation: target relation, if query is INSERT, UPDATE, or DELETE. + * p_target_relation: target relation, if query is INSERT, UPDATE, DELETE + * or MERGE. * * p_target_rangetblentry: target relation's entry in the rtable list. * @@ -181,7 +183,7 @@ struct ParseState List *p_ctenamespace; /* current namespace for common table exprs */ List *p_future_ctes; /* common table exprs not yet in namespace */ CommonTableExpr *p_parent_cte; /* this query's containing CTE */ - Relation p_target_relation; /* INSERT/UPDATE/DELETE target rel */ + Relation p_target_relation; /* INSERT/UPDATE/DELETE/MERGE target rel */ RangeTblEntry *p_target_rangetblentry; /* target rel's RTE */ bool p_is_insert; /* process assignment like INSERT not UPDATE */ List *p_windowdefs; /* raw representations of window clauses */ diff --git a/src/include/rewrite/rewriteHandler.h b/src/include/rewrite/rewriteHandler.h index 8128199fc3..1ab5de3942 100644 --- a/src/include/rewrite/rewriteHandler.h +++ b/src/include/rewrite/rewriteHandler.h @@ -25,6 +25,7 @@ extern void AcquireRewriteLocks(Query *parsetree, extern Node *build_column_default(Relation rel, int attrno); extern void rewriteTargetListUD(Query *parsetree, RangeTblEntry *target_rte, Relation target_relation); +extern void rewriteTargetListMerge(Query *parsetree, Relation target_relation); extern Query *get_view_query(Relation view); extern const char *view_query_is_auto_updatable(Query *viewquery, diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c index 4c0114c514..a432636322 100644 --- a/src/interfaces/libpq/fe-exec.c +++ b/src/interfaces/libpq/fe-exec.c @@ -3055,9 +3055,9 @@ PQoidValue(const PGresult *res) /* * PQcmdTuples - - * If the last command was INSERT/UPDATE/DELETE/MOVE/FETCH/COPY, return - * a string containing the number of inserted/affected tuples. If not, - * return "". + * If the last command was INSERT/UPDATE/DELETE/MERGE/MOVE/FETCH/COPY, + * return a string containing the number of inserted/affected tuples. + * If not, return "". * * XXX: this should probably return an int */ @@ -3084,7 +3084,8 @@ PQcmdTuples(PGresult *res) strncmp(res->cmdStatus, "DELETE ", 7) == 0 || strncmp(res->cmdStatus, "UPDATE ", 7) == 0) p = res->cmdStatus + 7; - else if (strncmp(res->cmdStatus, "FETCH ", 6) == 0) + else if (strncmp(res->cmdStatus, "FETCH ", 6) == 0 || + strncmp(res->cmdStatus, "MERGE ", 6) == 0) p = res->cmdStatus + 6; else if (strncmp(res->cmdStatus, "MOVE ", 5) == 0 || strncmp(res->cmdStatus, "COPY ", 5) == 0) diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c index 38ea7a091f..8a1d32a95c 100644 --- a/src/pl/plpgsql/src/pl_exec.c +++ b/src/pl/plpgsql/src/pl_exec.c @@ -3932,7 +3932,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, /* * On the first call for this statement generate the plan, and detect - * whether the statement is INSERT/UPDATE/DELETE + * whether the statement is INSERT/UPDATE/DELETE/MERGE */ if (expr->plan == NULL) { @@ -3953,6 +3953,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, { if (q->commandType == CMD_INSERT || q->commandType == CMD_UPDATE || + q->commandType == CMD_MERGE || q->commandType == CMD_DELETE) stmt->mod_stmt = true; } @@ -4010,6 +4011,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, case SPI_OK_INSERT_RETURNING: case SPI_OK_UPDATE_RETURNING: case SPI_OK_DELETE_RETURNING: + case SPI_OK_MERGE: Assert(stmt->mod_stmt); exec_set_found(estate, (SPI_processed != 0)); break; @@ -4187,6 +4189,7 @@ exec_stmt_dynexecute(PLpgSQL_execstate *estate, case SPI_OK_INSERT_RETURNING: case SPI_OK_UPDATE_RETURNING: case SPI_OK_DELETE_RETURNING: + case SPI_OK_MERGE: case SPI_OK_UTILITY: case SPI_OK_REWRITTEN: break; diff --git a/src/pl/plpgsql/src/pl_gram.y b/src/pl/plpgsql/src/pl_gram.y index 4c80936678..7d9fb2f039 100644 --- a/src/pl/plpgsql/src/pl_gram.y +++ b/src/pl/plpgsql/src/pl_gram.y @@ -303,6 +303,7 @@ static void check_raise_parameters(PLpgSQL_stmt_raise *stmt); %token <keyword> K_LAST %token <keyword> K_LOG %token <keyword> K_LOOP +%token <keyword> K_MERGE %token <keyword> K_MESSAGE %token <keyword> K_MESSAGE_TEXT %token <keyword> K_MOVE @@ -1930,6 +1931,10 @@ stmt_execsql : K_IMPORT { $$ = make_execsql_stmt(K_INSERT, @1); } + | K_MERGE + { + $$ = make_execsql_stmt(K_MERGE, @1); + } | T_WORD { int tok; @@ -2451,6 +2456,7 @@ unreserved_keyword : | K_IS | K_LAST | K_LOG + | K_MERGE | K_MESSAGE | K_MESSAGE_TEXT | K_MOVE @@ -2912,6 +2918,8 @@ make_execsql_stmt(int firsttoken, int location) { if (prev_tok == K_INSERT) continue; /* INSERT INTO is not an INTO-target */ + if (prev_tok == K_MERGE) + continue; /* MERGE INTO is not an INTO-target */ if (firsttoken == K_IMPORT) continue; /* IMPORT ... INTO is not an INTO-target */ if (have_into) diff --git a/src/pl/plpgsql/src/pl_scanner.c b/src/pl/plpgsql/src/pl_scanner.c index 65774f9902..85078ac15b 100644 --- a/src/pl/plpgsql/src/pl_scanner.c +++ b/src/pl/plpgsql/src/pl_scanner.c @@ -137,6 +137,7 @@ static const ScanKeyword unreserved_keywords[] = { PG_KEYWORD("is", K_IS, UNRESERVED_KEYWORD) PG_KEYWORD("last", K_LAST, UNRESERVED_KEYWORD) PG_KEYWORD("log", K_LOG, UNRESERVED_KEYWORD) + PG_KEYWORD("merge", K_MERGE, UNRESERVED_KEYWORD) PG_KEYWORD("message", K_MESSAGE, UNRESERVED_KEYWORD) PG_KEYWORD("message_text", K_MESSAGE_TEXT, UNRESERVED_KEYWORD) PG_KEYWORD("move", K_MOVE, UNRESERVED_KEYWORD) diff --git a/src/pl/plpgsql/src/plpgsql.h b/src/pl/plpgsql/src/plpgsql.h index f7619a63f9..8d6ef3326f 100644 --- a/src/pl/plpgsql/src/plpgsql.h +++ b/src/pl/plpgsql/src/plpgsql.h @@ -845,8 +845,8 @@ typedef struct PLpgSQL_stmt_execsql PLpgSQL_stmt_type cmd_type; int lineno; PLpgSQL_expr *sqlstmt; - bool mod_stmt; /* is the stmt INSERT/UPDATE/DELETE? Note: - * mod_stmt is set when we plan the query */ + bool mod_stmt; /* is the stmt INSERT/UPDATE/DELETE/MERGE? + * Note mod_stmt is set when we plan the query */ bool into; /* INTO supplied? */ bool strict; /* INTO STRICT flag */ PLpgSQL_variable *target; /* INTO target (record or row) */ diff --git a/src/test/isolation/expected/merge-delete.out b/src/test/isolation/expected/merge-delete.out new file mode 100644 index 0000000000..40e62901b7 --- /dev/null +++ b/src/test/isolation/expected/merge-delete.out @@ -0,0 +1,97 @@ +Parsed test spec with 2 sessions + +starting permutation: delete c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete c1 update1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete c1 update1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete c1 merge2 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: merge_delete c1 merge2 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: delete update1 c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; <waiting ...> +step c1: COMMIT; +step update1: <... completed> +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete update1 c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; <waiting ...> +step c1: COMMIT; +step update1: <... completed> +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete merge2 c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: merge_delete merge2 c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; diff --git a/src/test/isolation/expected/merge-insert-update.out b/src/test/isolation/expected/merge-insert-update.out new file mode 100644 index 0000000000..317fa16a3d --- /dev/null +++ b/src/test/isolation/expected/merge-insert-update.out @@ -0,0 +1,84 @@ +Parsed test spec with 2 sessions + +starting permutation: merge1 c1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +1 merge1 +step c2: COMMIT; + +starting permutation: merge1 c1 merge2 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step select2: SELECT * FROM target; +key val + +1 merge1 updated by merge2 +step c2: COMMIT; + +starting permutation: insert1 merge2 c1 select2 c2 +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: merge1 merge2 c1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: merge1 merge2 a1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step a1: ABORT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2 +step c2: COMMIT; + +starting permutation: delete1 insert1 c1 merge2 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step select2: SELECT * FROM target; +key val + +1 insert1 updated by merge2 +step c2: COMMIT; + +starting permutation: delete1 insert1 merge2 c1 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: delete1 insert1 merge2i c1 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2i: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +1 insert1 +step c2: COMMIT; diff --git a/src/test/isolation/expected/merge-match-recheck.out b/src/test/isolation/expected/merge-match-recheck.out new file mode 100644 index 0000000000..96a9f45ac8 --- /dev/null +++ b/src/test/isolation/expected/merge-match-recheck.out @@ -0,0 +1,106 @@ +Parsed test spec with 2 sessions + +starting permutation: update1 merge_status c2 select1 c1 +step update1: UPDATE target t SET balance = balance + 10, val = t.val || ' updated by update1' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 170 s2 setup updated by update1 when1 +step c1: COMMIT; + +starting permutation: update2 merge_status c2 select1 c1 +step update2: UPDATE target t SET status = 's2', val = t.val || ' updated by update2' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s3 setup updated by update2 when2 +step c1: COMMIT; + +starting permutation: update3 merge_status c2 select1 c1 +step update3: UPDATE target t SET status = 's3', val = t.val || ' updated by update3' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s4 setup updated by update3 when3 +step c1: COMMIT; + +starting permutation: update5 merge_status c2 select1 c1 +step update5: UPDATE target t SET status = 's5', val = t.val || ' updated by update5' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s5 setup updated by update5 +step c1: COMMIT; + +starting permutation: update_bal1 merge_bal c2 select1 c1 +step update_bal1: UPDATE target t SET balance = 50, val = t.val || ' updated by update_bal1' WHERE t.key = 1; +step merge_bal: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND balance < 100 THEN + UPDATE SET balance = balance * 2, val = t.val || ' when1' + WHEN MATCHED AND balance < 200 THEN + UPDATE SET balance = balance * 4, val = t.val || ' when2' + WHEN MATCHED AND balance < 300 THEN + UPDATE SET balance = balance * 8, val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_bal: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 100 s1 setup updated by update_bal1 when1 +step c1: COMMIT; diff --git a/src/test/isolation/expected/merge-update.out b/src/test/isolation/expected/merge-update.out new file mode 100644 index 0000000000..60ae42ebd0 --- /dev/null +++ b/src/test/isolation/expected/merge-update.out @@ -0,0 +1,213 @@ +Parsed test spec with 2 sessions + +starting permutation: merge1 c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +step c2: COMMIT; + +starting permutation: merge1 c1 merge2a select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step c1: COMMIT; +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2a c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2a: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2a a1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step a1: ABORT; +step merge2a: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2b c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2b: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2b' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED AND t.key < 2 THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2b: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2b +step c2: COMMIT; + +starting permutation: merge1 merge2c c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2c: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2c' as val) s + ON s.key = t.key AND t.key < 2 + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2c: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2c +step c2: COMMIT; + +starting permutation: pa_merge1 pa_merge2a c1 pa_select2 c2 +step pa_merge1: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set val = t.val || ' updated by ' || s.val; + +step pa_merge2a: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step pa_merge2a: <... completed> +step pa_select2: SELECT * FROM pa_target; +key val + +2 initial +2 initial updated by pa_merge2a +step c2: COMMIT; + +starting permutation: pa_merge2 pa_merge2a c1 pa_select2 c2 +step pa_merge2: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step pa_merge2a: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step pa_merge2a: <... completed> +step pa_select2: SELECT * FROM pa_target; +key val + +1 pa_merge2a +2 initial +2 initial updated by pa_merge1 +step c2: COMMIT; diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index 74d7d59546..644e071112 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -33,6 +33,10 @@ test: insert-conflict-do-update test: insert-conflict-do-update-2 test: insert-conflict-do-update-3 test: insert-conflict-toast +test: merge-insert-update +test: merge-delete +test: merge-update +test: merge-match-recheck test: delete-abort-savept test: delete-abort-savept-2 test: aborted-keyrevoke diff --git a/src/test/isolation/specs/merge-delete.spec b/src/test/isolation/specs/merge-delete.spec new file mode 100644 index 0000000000..656954f847 --- /dev/null +++ b/src/test/isolation/specs/merge-delete.spec @@ -0,0 +1,51 @@ +# MERGE DELETE +# +# This test looks at the interactions involving concurrent deletes +# comparing the behavior of MERGE, DELETE and UPDATE + +setup +{ + CREATE TABLE target (key int primary key, val text); + INSERT INTO target VALUES (1, 'setup1'); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "delete" { DELETE FROM target t WHERE t.key = 1; } +step "merge_delete" { MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "update1" { UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; } +step "merge2" { MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; } +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } + +# Basic effects +permutation "delete" "c1" "select2" "c2" +permutation "merge_delete" "c1" "select2" "c2" + +# One after the other, no concurrency +permutation "delete" "c1" "update1" "select2" "c2" +permutation "merge_delete" "c1" "update1" "select2" "c2" +permutation "delete" "c1" "merge2" "select2" "c2" +permutation "merge_delete" "c1" "merge2" "select2" "c2" + +# Now with concurrency +permutation "delete" "update1" "c1" "select2" "c2" +permutation "merge_delete" "update1" "c1" "select2" "c2" +permutation "delete" "merge2" "c1" "select2" "c2" +permutation "merge_delete" "merge2" "c1" "select2" "c2" diff --git a/src/test/isolation/specs/merge-insert-update.spec b/src/test/isolation/specs/merge-insert-update.spec new file mode 100644 index 0000000000..704492be1f --- /dev/null +++ b/src/test/isolation/specs/merge-insert-update.spec @@ -0,0 +1,52 @@ +# MERGE INSERT UPDATE +# +# This looks at how we handle concurrent INSERTs, illustrating how the +# behavior differs from INSERT ... ON CONFLICT + +setup +{ + CREATE TABLE target (key int primary key, val text); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge1" { MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; } +step "delete1" { DELETE FROM target WHERE key = 1; } +step "insert1" { INSERT INTO target VALUES (1, 'insert1'); } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge2" { MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; } + +step "merge2i" { MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; } + +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } +step "a2" { ABORT; } + +# Basic effects +permutation "merge1" "c1" "select2" "c2" +permutation "merge1" "c1" "merge2" "select2" "c2" + +# check concurrent inserts +permutation "insert1" "merge2" "c1" "select2" "c2" +permutation "merge1" "merge2" "c1" "select2" "c2" +permutation "merge1" "merge2" "a1" "select2" "c2" + +# check how we handle when visible row has been concurrently deleted, then same key re-inserted +permutation "delete1" "insert1" "c1" "merge2" "select2" "c2" +permutation "delete1" "insert1" "merge2" "c1" "select2" "c2" +permutation "delete1" "insert1" "merge2i" "c1" "select2" "c2" diff --git a/src/test/isolation/specs/merge-match-recheck.spec b/src/test/isolation/specs/merge-match-recheck.spec new file mode 100644 index 0000000000..193033da17 --- /dev/null +++ b/src/test/isolation/specs/merge-match-recheck.spec @@ -0,0 +1,79 @@ +# MERGE MATCHED RECHECK +# +# This test looks at what happens when we have complex +# WHEN MATCHED AND conditions and a concurrent UPDATE causes a +# recheck of the AND condition on the new row + +setup +{ + CREATE TABLE target (key int primary key, balance integer, status text, val text); + INSERT INTO target VALUES (1, 160, 's1', 'setup'); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge_status" +{ + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; +} + +step "merge_bal" +{ + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND balance < 100 THEN + UPDATE SET balance = balance * 2, val = t.val || ' when1' + WHEN MATCHED AND balance < 200 THEN + UPDATE SET balance = balance * 4, val = t.val || ' when2' + WHEN MATCHED AND balance < 300 THEN + UPDATE SET balance = balance * 8, val = t.val || ' when3'; +} + +step "select1" { SELECT * FROM target; } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "update1" { UPDATE target t SET balance = balance + 10, val = t.val || ' updated by update1' WHERE t.key = 1; } +step "update2" { UPDATE target t SET status = 's2', val = t.val || ' updated by update2' WHERE t.key = 1; } +step "update3" { UPDATE target t SET status = 's3', val = t.val || ' updated by update3' WHERE t.key = 1; } +step "update5" { UPDATE target t SET status = 's5', val = t.val || ' updated by update5' WHERE t.key = 1; } +step "update_bal1" { UPDATE target t SET balance = 50, val = t.val || ' updated by update_bal1' WHERE t.key = 1; } +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } + +# merge_status sees concurrently updated row and rechecks WHEN conditions, but recheck passes and final status = 's2' +permutation "update1" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final status = 's3' not 's2' +permutation "update2" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final status = 's4' not 's2' +permutation "update3" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, but we skip update and MERGE does nothing +permutation "update5" "merge_status" "c2" "select1" "c1" + +# merge_bal sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final balance = 100 not 640 +permutation "update_bal1" "merge_bal" "c2" "select1" "c1" diff --git a/src/test/isolation/specs/merge-update.spec b/src/test/isolation/specs/merge-update.spec new file mode 100644 index 0000000000..64e849966e --- /dev/null +++ b/src/test/isolation/specs/merge-update.spec @@ -0,0 +1,132 @@ +# MERGE UPDATE +# +# This test exercises atypical cases +# 1. UPDATEs of PKs that change the join in the ON clause +# 2. UPDATEs with WHEN AND conditions that would fail after concurrent update +# 3. UPDATEs with extra ON conditions that would fail after concurrent update + +setup +{ + CREATE TABLE target (key int primary key, val text); + INSERT INTO target VALUES (1, 'setup1'); + + CREATE TABLE pa_target (key integer, val text) + PARTITION BY LIST (key); + CREATE TABLE part1 (key integer, val text); + CREATE TABLE part2 (val text, key integer); + CREATE TABLE part3 (key integer, val text); + + ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); + ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); + ALTER TABLE pa_target ATTACH PARTITION part3 DEFAULT; + + INSERT INTO pa_target VALUES (1, 'initial'); + INSERT INTO pa_target VALUES (2, 'initial'); +} + +teardown +{ + DROP TABLE target; + DROP TABLE pa_target CASCADE; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge1" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "pa_merge1" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set val = t.val || ' updated by ' || s.val; +} +step "pa_merge2" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge2a" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "merge2b" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2b' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED AND t.key < 2 THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "merge2c" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2c' as val) s + ON s.key = t.key AND t.key < 2 + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "pa_merge2a" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "select2" { SELECT * FROM target; } +step "pa_select2" { SELECT * FROM pa_target; } +step "c2" { COMMIT; } + +# Basic effects +permutation "merge1" "c1" "select2" "c2" + +# One after the other, no concurrency +permutation "merge1" "c1" "merge2a" "select2" "c2" + +# Now with concurrency +permutation "merge1" "merge2a" "c1" "select2" "c2" +permutation "merge1" "merge2a" "a1" "select2" "c2" +permutation "merge1" "merge2b" "c1" "select2" "c2" +permutation "merge1" "merge2c" "c1" "select2" "c2" +permutation "pa_merge1" "pa_merge2a" "c1" "pa_select2" "c2" +permutation "pa_merge2" "pa_merge2a" "c1" "pa_select2" "c2" diff --git a/src/test/regress/expected/identity.out b/src/test/regress/expected/identity.out index d7d5178f5d..3a6016c80a 100644 --- a/src/test/regress/expected/identity.out +++ b/src/test/regress/expected/identity.out @@ -386,3 +386,58 @@ CREATE TABLE itest_child PARTITION OF itest_parent ( ) FOR VALUES FROM ('2016-07-01') TO ('2016-08-01'); -- error ERROR: identity columns are not supported on partitions DROP TABLE itest_parent; +-- MERGE tests +CREATE TABLE itest14 (a int GENERATED ALWAYS AS IDENTITY, b text); +CREATE TABLE itest15 (a int GENERATED BY DEFAULT AS IDENTITY, b text); +MERGE INTO itest14 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); +ERROR: cannot insert into column "a" +DETAIL: Column "a" is an identity column defined as GENERATED ALWAYS. +HINT: Use OVERRIDING SYSTEM VALUE to override. +MERGE INTO itest14 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); +ERROR: cannot insert into column "a" +DETAIL: Column "a" is an identity column defined as GENERATED ALWAYS. +HINT: Use OVERRIDING SYSTEM VALUE to override. +MERGE INTO itest14 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); +SELECT * FROM itest14; + a | b +----+------------------- + 30 | inserted by merge +(1 row) + +SELECT * FROM itest15; + a | b +----+------------------- + 10 | inserted by merge + 1 | inserted by merge + 30 | inserted by merge +(3 rows) + +DROP TABLE itest14; +DROP TABLE itest15; diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out new file mode 100644 index 0000000000..05c4287078 --- /dev/null +++ b/src/test/regress/expected/merge.out @@ -0,0 +1,1599 @@ +-- +-- MERGE +-- +--\set VERBOSITY verbose +--set debug_print_rewritten = true; +--set debug_print_parse = true; +--set debug_print_pretty = true; +CREATE USER merge_privs; +CREATE USER merge_no_privs; +DROP TABLE IF EXISTS target; +NOTICE: table "target" does not exist, skipping +DROP TABLE IF EXISTS source; +NOTICE: table "source" does not exist, skipping +CREATE TABLE target (tid integer, balance integer); +CREATE TABLE source (sid integer, delta integer); --no index +INSERT INTO target VALUES (1, 10); +INSERT INTO target VALUES (2, 20); +INSERT INTO target VALUES (3, 30); +SELECT t.ctid is not null as matched, t.*, s.* FROM source s FULL OUTER JOIN target t ON s.sid = t.tid ORDER BY t.tid, s.sid; + matched | tid | balance | sid | delta +---------+-----+---------+-----+------- + t | 1 | 10 | | + t | 2 | 20 | | + t | 3 | 30 | | +(3 rows) + +ALTER TABLE target OWNER TO merge_privs; +ALTER TABLE source OWNER TO merge_privs; +CREATE TABLE target2 (tid integer, balance integer); +CREATE TABLE source2 (sid integer, delta integer); +ALTER TABLE target2 OWNER TO merge_no_privs; +ALTER TABLE source2 OWNER TO merge_no_privs; +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_privs; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Merge Join + Merge Cond: (t_1.tid = s.sid) + -> Sort + Sort Key: t_1.tid + -> Seq Scan on target t_1 + -> Sort + Sort Key: s.sid + -> Seq Scan on source s +(9 rows) + +-- +-- Errors +-- +MERGE INTO target t RANDOMWORD +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: syntax error at or near "RANDOMWORD" +LINE 1: MERGE INTO target t RANDOMWORD + ^ +-- MATCHED/INSERT error +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: syntax error at or near "INSERT" +LINE 5: INSERT DEFAULT VALUES + ^ +-- incorrectly specifying INTO target +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT INTO target DEFAULT VALUES +; +ERROR: syntax error at or near "INTO" +LINE 5: INSERT INTO target DEFAULT VALUES + ^ +-- NOT MATCHED/UPDATE +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: syntax error at or near "UPDATE" +LINE 5: UPDATE SET balance = 0 + ^ +-- UPDATE tablename +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE target SET balance = 0 +; +ERROR: syntax error at or near "target" +LINE 5: UPDATE target SET balance = 0 + ^ +-- permissions +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: permission denied for table source2 +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_no_privs; +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: permission denied for table target +GRANT UPDATE ON target2 TO merge_privs; +SET SESSION AUTHORIZATION merge_privs; +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN MATCHED THEN + DELETE +; +ERROR: permission denied for table target2 +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: permission denied for table target2 +-- check if the target can be accessed from source relation subquery; we should +-- not be able to do so +MERGE INTO target t +USING (SELECT * FROM source WHERE t.tid > sid) s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 2: USING (SELECT * FROM source WHERE t.tid > sid) s + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +-- +-- initial tests +-- +-- zero rows in source has no effect +MERGE INTO target +USING source +ON target.tid = source.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ROLLBACK; +-- insert some non-matching source rows to work from +INSERT INTO source VALUES (4, 40); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 4 | 40 +(1 row) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + DO NOTHING +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + | +(4 rows) + +ROLLBACK; +-- index plans +INSERT INTO target SELECT generate_series(1000,2500), 0; +ALTER TABLE target ADD PRIMARY KEY (tid); +ANALYZE target; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL); + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Left Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +; +DELETE FROM target WHERE tid > 100; +ANALYZE target; +-- insert some matching source rows to work from +INSERT INTO source VALUES (2, 5); +INSERT INTO source VALUES (3, 20); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 3 | 20 + 4 | 40 +(3 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +-- equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 +(3 rows) + +ROLLBACK; +-- equivalent of a DELETE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 +(1 row) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | +(4 rows) + +ROLLBACK; +-- duplicate source row causes multiple target row update ERROR +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 2 | 5 + 3 | 20 + 4 | 40 +(4 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: MERGE command cannot affect row a second time +HINT: Ensure that not more than one source rows match any one target row +ROLLBACK; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +ERROR: MERGE command cannot affect row a second time +HINT: Ensure that not more than one source rows match any one target row +ROLLBACK; +-- correct source data +DELETE FROM source WHERE sid = 2; +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 3 | 20 + 4 | 40 +(3 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +-- remove constraints +alter table target drop CONSTRAINT target_pkey; +alter table target alter column tid drop not null; +-- multiple actions +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4) +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 + 4 | 4 +(4 rows) + +ROLLBACK; +-- should be equivalent +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4); +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 + 4 | 4 +(4 rows) + +ROLLBACK; +-- column references +-- do a simple equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 25 + 3 | 50 +(3 rows) + +ROLLBACK; +-- do a simple equivalent of an INSERT SELECT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- and again with explicitly identified column list +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- and again with a subtle error: referring to non-existent target row for NOT MATCHED +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 5: INSERT (tid, balance) VALUES (t.tid, s.delta) + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +-- and again with a constant ON clause +BEGIN; +MERGE INTO target t +USING source AS s +ON (SELECT true) +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 5: INSERT (tid, balance) VALUES (t.tid, s.delta) + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +SELECT * FROM target ORDER BY tid; +ERROR: current transaction is aborted, commands ignored until end of transaction block +ROLLBACK; +-- now the classic UPSERT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 25 + 3 | 50 + 4 | 40 +(4 rows) + +ROLLBACK; +-- unreachable WHEN clause should ERROR +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN /* Terminal WHEN clause for MATCHED */ + DELETE +WHEN MATCHED AND s.delta > 0 THEN + UPDATE SET balance = t.balance - s.delta +; +ERROR: unreachable WHEN clause specified after unconditional WHEN clause +ROLLBACK; +-- conditional WHEN clause +CREATE TABLE wq_target (tid integer not null, balance integer DEFAULT -1); +CREATE TABLE wq_source (balance integer, sid integer); +INSERT INTO wq_source (sid, balance) VALUES (1, 100); +BEGIN; +-- try a simple INSERT with default values first +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +ROLLBACK; +-- this time with a FALSE condition +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND FALSE THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- +(0 rows) + +-- this time with an actual condition which returns false +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance <> 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- +(0 rows) + +BEGIN; +-- and now with a condition which returns true +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +ROLLBACK; +-- conditions in the NOT MATCHED clause can only refer to source columns +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND t.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 3: WHEN NOT MATCHED AND t.balance = 100 THEN + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +SELECT * FROM wq_target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +ROLLBACK; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +-- conditions in MATCHED clause can refer to both source and target +SELECT * FROM wq_source; + balance | sid +---------+----- + 100 | 1 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +-- check if AND works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 199 +(1 row) + +-- check if OR works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 199 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 199 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 299 +(1 row) + +-- check if subqueries work in the conditions? +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance > (SELECT max(balance) FROM target) THEN + UPDATE SET balance = t.balance + s.balance; +-- check if we can access system columns in the conditions +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.xmin = t.xmax THEN + UPDATE SET balance = t.balance + s.balance; +ERROR: system column "xmin" reference in WHEN AND condition is invalid +LINE 3: WHEN MATCHED AND t.xmin = t.xmax THEN + ^ +ALTER TABLE wq_target SET WITH OIDS; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 399 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.oid >= 0 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 499 +(1 row) + +-- test preventing WHEN AND conditions from writing to the database +create or replace function merge_when_and_write() returns boolean +language plpgsql as +$$ +BEGIN + INSERT INTO target VALUES (100, 100); + RETURN TRUE; +END; +$$; +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND (merge_when_and_write()) THEN + UPDATE SET balance = t.balance + s.balance; +ROLLBACK; +drop function merge_when_and_write(); +DROP TABLE wq_target, wq_source; +-- test triggers +create or replace function merge_trigfunc () returns trigger +language plpgsql as +$$ +BEGIN + RAISE NOTICE '% % % trigger', TG_WHEN, TG_OP, TG_LEVEL; + IF (TG_WHEN = 'BEFORE' AND TG_LEVEL = 'ROW') THEN + IF (TG_OP = 'DELETE') THEN + RETURN OLD; + ELSE + RETURN NEW; + END IF; + ELSE + RETURN NULL; + END IF; +END; +$$; +CREATE TRIGGER merge_bsi BEFORE INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsu BEFORE UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsd BEFORE DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asi AFTER INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asu AFTER UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asd AFTER DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bri BEFORE INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bru BEFORE UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_brd BEFORE DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ari AFTER INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_aru AFTER UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ard AFTER DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +-- now the classic UPSERT, with a DELETE +BEGIN; +UPDATE target SET balance = 0 WHERE tid = 3; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE DELETE STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: BEFORE DELETE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER DELETE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER DELETE STATEMENT trigger +NOTICE: AFTER UPDATE STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger + QUERY PLAN +------------------------------------------------------------------ + Merge on target t (actual rows=0 loops=1) + Tuples Inserted: 1 + Tuples Updated: 1 + Tuples Deleted: 1 + -> Hash Left Join (actual rows=3 loops=1) + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s (actual rows=3 loops=1) + -> Hash (actual rows=3 loops=1) + Buckets: 1024 Batches: 1 Memory Usage: 9kB + -> Seq Scan on target t_1 (actual rows=3 loops=1) + Trigger merge_ard: calls=1 + Trigger merge_ari: calls=1 + Trigger merge_aru: calls=1 + Trigger merge_asd: calls=1 + Trigger merge_asi: calls=1 + Trigger merge_asu: calls=1 + Trigger merge_brd: calls=1 + Trigger merge_bri: calls=1 + Trigger merge_bru: calls=1 + Trigger merge_bsd: calls=1 + Trigger merge_bsi: calls=1 + Trigger merge_bsu: calls=1 +(22 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 15 + 4 | 40 +(3 rows) + +ROLLBACK; +-- test from PL/pgSQL +-- make sure MERGE INTO isn't interpreted to mean returning variables like SELECT INTO +BEGIN; +DO LANGUAGE plpgsql $$ +BEGIN +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +; +END; +$$; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +ROLLBACK; +--source constants +BEGIN; +MERGE INTO target t +USING (SELECT 9 AS sid, 57 AS delta) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 9 | 57 +(4 rows) + +ROLLBACK; +--source query +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta as newname FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.newname) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +--self-merge +BEGIN; +MERGE INTO target t1 +USING target t2 +ON t1.tid = t2.tid +WHEN MATCHED THEN + UPDATE SET balance = t1.balance + t2.balance +WHEN NOT MATCHED THEN + INSERT VALUES (t2.tid, t2.balance) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 20 + 2 | 40 + 3 | 60 +(3 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING (SELECT tid as sid, balance as delta FROM target WHERE balance > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING +(SELECT sid, max(delta) AS delta + FROM source + GROUP BY sid + HAVING count(*) = 1 + ORDER BY sid ASC) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- plpgsql parameters and results +BEGIN; +CREATE FUNCTION merge_func (p_id integer, p_bal integer) +RETURNS INTEGER +LANGUAGE plpgsql +AS $$ +DECLARE + result integer; +BEGIN +MERGE INTO target t +USING (SELECT p_id AS sid) AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance - p_bal +; +IF FOUND THEN + GET DIAGNOSTICS result := ROW_COUNT; +END IF; +RETURN result; +END; +$$; +SELECT merge_func(3, 4); +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger + merge_func +------------ + 1 +(1 row) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 26 +(3 rows) + +ROLLBACK; +-- PREPARE +BEGIN; +prepare foom as merge into target t using (select 1 as sid) s on (t.tid = s.sid) when matched then update set balance = 1; +execute foom; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 1 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +BEGIN; +PREPARE foom2 (integer, integer) AS +MERGE INTO target t +USING (SELECT 1) s +ON t.tid = $1 +WHEN MATCHED THEN +UPDATE SET balance = $2; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +execute foom2 (1, 1); +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger + QUERY PLAN +------------------------------------------------------ + Merge on target t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 1 + Tuples Deleted: 0 + -> Seq Scan on target t_1 (actual rows=1 loops=1) + Filter: (tid = 1) + Rows Removed by Filter: 2 + Trigger merge_aru: calls=1 + Trigger merge_asu: calls=1 + Trigger merge_bru: calls=1 + Trigger merge_bsu: calls=1 +(11 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 1 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +-- subqueries in source relation +CREATE TABLE sq_target (tid integer NOT NULL, balance integer); +CREATE TABLE sq_source (delta integer, sid integer, balance integer DEFAULT 0); +INSERT INTO sq_target(tid, balance) VALUES (1,100), (2,200), (3,300); +INSERT INTO sq_source(sid, delta) VALUES (1,10), (2,20), (4,40); +BEGIN; +MERGE INTO sq_target t +USING (SELECT * FROM sq_source) s +ON tid = sid +WHEN MATCHED AND t.balance > delta THEN + UPDATE SET balance = t.balance + delta; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 3 | 300 + 1 | 110 + 2 | 220 +(3 rows) + +ROLLBACK; +-- try a view +CREATE VIEW v AS SELECT * FROM sq_source WHERE sid < 2; +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = v.balance + delta; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 2 | 200 + 3 | 300 + 1 | 10 +(3 rows) + +ROLLBACK; +-- ambiguous reference to a column +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +ERROR: column reference "balance" is ambiguous +LINE 5: UPDATE SET balance = balance + delta + ^ +ROLLBACK; +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 2 | 200 + 3 | 300 + -1 | -11 +(3 rows) + +ROLLBACK; +-- CTEs +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +WITH targq AS ( + SELECT * FROM v +) +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +; +ERROR: syntax error at or near "MERGE" +LINE 4: MERGE INTO sq_target t + ^ +ROLLBACK; +-- RETURNING +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +RETURNING * +; +ERROR: syntax error at or near "RETURNING" +LINE 10: RETURNING * + ^ +ROLLBACK; +-- Subqueries +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = (SELECT count(*) FROM sq_target) +; +ROLLBACK; +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND (SELECT count(*) > 0 FROM sq_target) THEN + UPDATE SET balance = 42 +; +ROLLBACK; +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid AND (SELECT count(*) > 0 FROM sq_target) +WHEN MATCHED THEN + UPDATE SET balance = 42 +; +ROLLBACK; +DROP TABLE sq_target, sq_source CASCADE; +NOTICE: drop cascades to view v +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 PARTITION OF pa_target FOR VALUES IN (1,4); +CREATE TABLE part2 PARTITION OF pa_target FOR VALUES IN (2,5,6); +CREATE TABLE part3 PARTITION OF pa_target FOR VALUES IN (3,8,9); +CREATE TABLE part4 PARTITION OF pa_target DEFAULT; +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 330 | initial updated by merge + 4 | 40 | inserted by merge + 5 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 7 | 770 | initial updated by merge + 8 | 80 | inserted by merge + 9 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 11 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 13 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 30 | inserted by merge + 3 | 300 | initial + 4 | 40 | inserted by merge + 5 | 500 | initial + 5 | 50 | inserted by merge + 6 | 60 | inserted by merge + 7 | 700 | initial + 7 | 70 | inserted by merge + 8 | 80 | inserted by merge + 9 | 90 | inserted by merge + 9 | 900 | initial + 10 | 100 | inserted by merge + 11 | 1100 | initial + 11 | 110 | inserted by merge + 12 | 120 | inserted by merge + 13 | 1300 | initial + 13 | 130 | inserted by merge + 14 | 140 | inserted by merge +(20 rows) + +ROLLBACK; +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 2 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 4 | 40 | inserted by merge + 4 | 330 | initial updated by merge + 6 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 8 | 80 | inserted by merge + 8 | 770 | initial updated by merge + 10 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 12 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 14 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +DROP TABLE pa_target CASCADE; +-- The target table is partitioned in the same way, but this time by attaching +-- partitions which have columns in different order, dropped columns etc. +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 (tid integer, balance float, val text); +CREATE TABLE part2 (balance float, tid integer, val text); +CREATE TABLE part3 (tid integer, balance float, val text); +CREATE TABLE part4 (extraid text, tid integer, balance float, val text); +ALTER TABLE part4 DROP COLUMN extraid; +ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); +ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); +ALTER TABLE pa_target ATTACH PARTITION part3 FOR VALUES IN (3,8,9); +ALTER TABLE pa_target ATTACH PARTITION part4 DEFAULT; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 330 | initial updated by merge + 4 | 40 | inserted by merge + 5 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 7 | 770 | initial updated by merge + 8 | 80 | inserted by merge + 9 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 11 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 13 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 30 | inserted by merge + 3 | 300 | initial + 4 | 40 | inserted by merge + 5 | 500 | initial + 5 | 50 | inserted by merge + 6 | 60 | inserted by merge + 7 | 700 | initial + 7 | 70 | inserted by merge + 8 | 80 | inserted by merge + 9 | 90 | inserted by merge + 9 | 900 | initial + 10 | 100 | inserted by merge + 11 | 1100 | initial + 11 | 110 | inserted by merge + 12 | 120 | inserted by merge + 13 | 1300 | initial + 13 | 130 | inserted by merge + 14 | 140 | inserted by merge +(20 rows) + +ROLLBACK; +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 2 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 4 | 40 | inserted by merge + 4 | 330 | initial updated by merge + 6 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 8 | 80 | inserted by merge + 8 | 770 | initial updated by merge + 10 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 12 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 14 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; +-- Sub-partitionin +CREATE TABLE pa_target (logts timestamp, tid integer, balance float, val text) + PARTITION BY RANGE (logts); +CREATE TABLE part_m01 PARTITION OF pa_target + FOR VALUES FROM ('2017-01-01') TO ('2017-02-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m01_odd PARTITION OF part_m01 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m01_even PARTITION OF part_m01 + FOR VALUES IN (2,4,6,8); +CREATE TABLE part_m02 PARTITION OF pa_target + FOR VALUES FROM ('2017-02-01') TO ('2017-03-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m02_odd PARTITION OF part_m02 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m02_even PARTITION OF part_m02 + FOR VALUES IN (2,4,6,8); +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT '2017-01-31', id, id * 100, 'initial' FROM generate_series(1,9,3) AS id; +INSERT INTO pa_target SELECT '2017-02-28', id, id * 100, 'initial' FROM generate_series(2,9,3) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING (SELECT '2017-01-15' AS slogts, * FROM pa_source WHERE sid < 10) s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + logts | tid | balance | val +--------------------------+-----+---------+-------------------------- + Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge + Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge + Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge +(9 rows) + +ROLLBACK; +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; +-- some complex joins on the source side +CREATE TABLE cj_target (tid integer, balance float, val text); +CREATE TABLE cj_source1 (sid1 integer, scat integer, delta integer); +CREATE TABLE cj_source2 (sid2 integer, sval text); +INSERT INTO cj_source1 VALUES (1, 10, 100); +INSERT INTO cj_source1 VALUES (1, 20, 200); +INSERT INTO cj_source1 VALUES (2, 20, 300); +INSERT INTO cj_source1 VALUES (3, 10, 400); +INSERT INTO cj_source2 VALUES (1, 'initial source2'); +INSERT INTO cj_source2 VALUES (2, 'initial source2'); +INSERT INTO cj_source2 VALUES (3, 'initial source2'); +-- source relation is an unalised join +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid1, delta, sval); +-- try accessing columns from either side of the source join +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta, sval) +WHEN MATCHED THEN + DELETE; +-- some simple expressions in INSERT targetlist +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta + scat, sval) +WHEN MATCHED THEN + UPDATE SET val = val || ' updated by merge'; +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN MATCHED THEN + UPDATE SET val = val || ' ' || delta::text; +SELECT * FROM cj_target; + tid | balance | val +-----+---------+---------------------------------- + 3 | 400 | initial source2 updated by merge + 1 | 220 | initial source2 200 + 1 | 110 | initial source2 200 + 2 | 320 | initial source2 300 +(4 rows) + +ALTER TABLE cj_source1 RENAME COLUMN sid1 TO sid; +ALTER TABLE cj_source2 RENAME COLUMN sid2 TO sid; +TRUNCATE cj_target; +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON s1.sid = s2.sid +ON t.tid = s1.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s2.sid, delta, sval); +DROP TABLE cj_source2, cj_source1, cj_target; +-- Function scans +CREATE TABLE fs_target (a int, b int, c text); +MERGE INTO fs_target t +USING generate_series(1,100,1) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1); +MERGE INTO fs_target t +USING generate_series(1,100,2) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id, c = 'updated '|| id.*::text +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1, 'inserted ' || id.*::text); +SELECT count(*) FROM fs_target; + count +------- + 100 +(1 row) + +DROP TABLE fs_target; +-- SERIALIZABLE test +-- handled in isolation tests +-- prepare +RESET SESSION AUTHORIZATION; +DROP TABLE target, target2; +DROP TABLE source, source2; +DROP FUNCTION merge_trigfunc(); +DROP USER merge_privs; +DROP USER merge_no_privs; diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index ac8968d24f..864f2c1345 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -517,6 +517,104 @@ SELECT atest6 FROM atest6; -- ok (0 rows) COPY atest6 TO stdout; -- ok +-- test column privileges with MERGE +SET SESSION AUTHORIZATION regress_priv_user1; +CREATE TABLE mtarget (a int, b text); +CREATE TABLE msource (a int, b text); +INSERT INTO mtarget VALUES (1, 'init1'), (2, 'init2'); +INSERT INTO msource VALUES (1, 'source1'), (2, 'source2'), (3, 'source3'); +GRANT SELECT (a) ON msource TO regress_priv_user4; +GRANT SELECT (a) ON mtarget TO regress_priv_user4; +GRANT INSERT (a,b) ON mtarget TO regress_priv_user4; +GRANT UPDATE (b) ON mtarget TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; +-- +-- test source privileges +-- +-- fail (no SELECT priv on s.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table msource +-- fail (s.b used in the INSERTed values) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table msource +-- fail (s.b used in the WHEN quals) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND s.b = 'x' THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table msource +-- this should be ok since only s.a is accessed +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'ok' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ROLLBACK; +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT SELECT (b) ON msource TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; +-- should now be ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ROLLBACK; +-- +-- test target privileges +-- +-- fail (no SELECT priv on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table mtarget +-- fail (no UPDATE on t.a) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b, a = t.a + 1 +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table mtarget +-- fail (no SELECT on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table mtarget +-- ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b; +ROLLBACK; +-- fail (no DELETE) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ERROR: permission denied for table mtarget +-- grant delete privileges +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT DELETE ON mtarget TO regress_priv_user4; +-- should be ok now +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ROLLBACK; -- check error reporting with column privs SET SESSION AUTHORIZATION regress_priv_user1; CREATE TABLE t1 (c1 int, c2 int, c3 int check (c3 < 5), primary key (c1, c2)); diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index f1ae40df61..bf7af3ba82 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -2138,6 +2138,188 @@ ERROR: new row violates row-level security policy (USING expression) for table INSERT INTO document VALUES (1, (SELECT cid from category WHERE cname = 'novel'), 1, 'regress_rls_bob', 'my first novel') ON CONFLICT (did) DO UPDATE SET dauthor = 'regress_rls_carol'; ERROR: new row violates row-level security policy for table "document" +-- +-- MERGE +-- +RESET SESSION AUTHORIZATION; +DROP POLICY p3_with_all ON document; +ALTER TABLE document ADD COLUMN dnotes text DEFAULT ''; +-- all documents are readable +CREATE POLICY p1 ON document FOR SELECT USING (true); +-- one may insert documents only authored by them +CREATE POLICY p2 ON document FOR INSERT WITH CHECK (dauthor = current_user); +-- one may only update documents in 'novel' category +CREATE POLICY p3 ON document FOR UPDATE + USING (cid = (SELECT cid from category WHERE cname = 'novel')) + WITH CHECK (dauthor = current_user); +-- one may only delete documents in 'manga' category +CREATE POLICY p4 ON document FOR DELETE + USING (cid = (SELECT cid from category WHERE cname = 'manga')); +SELECT * FROM document; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-------------------+----------------------------------+-------- + 1 | 11 | 1 | regress_rls_bob | my first novel | + 3 | 22 | 2 | regress_rls_bob | my science fiction | + 4 | 44 | 1 | regress_rls_bob | my first manga | + 5 | 44 | 2 | regress_rls_bob | my second manga | + 6 | 22 | 1 | regress_rls_carol | great science fiction | + 7 | 33 | 2 | regress_rls_carol | great technology book | + 8 | 44 | 1 | regress_rls_carol | great manga | + 9 | 22 | 1 | regress_rls_dave | awesome science fiction | + 10 | 33 | 2 | regress_rls_dave | awesome technology book | + 11 | 33 | 1 | regress_rls_carol | hoge | + 33 | 22 | 1 | regress_rls_bob | okay science fiction | + 2 | 11 | 2 | regress_rls_bob | my first novel | + 78 | 33 | 1 | regress_rls_bob | some technology novel | + 79 | 33 | 1 | regress_rls_bob | technology book, can only insert | +(14 rows) + +SET SESSION AUTHORIZATION regress_rls_bob; +-- Fails, since update violates WITH CHECK qual on dauthor +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge1 ', dauthor = 'regress_rls_alice'; +ERROR: new row violates row-level security policy for table "document" +-- Should be OK since USING and WITH CHECK quals pass +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge2 '; +-- Even when dauthor is updated explicitly, but to the existing value +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge3 ', dauthor = 'regress_rls_bob'; +-- There is a MATCH for did = 3, but UPDATE's USING qual does not allow +-- updating an item in category 'science fiction' +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge '; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- The same thing with DELETE action, but fails again because no permissions +-- to delete items in 'science fiction' category that did 3 belongs to. +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- Document with did 4 belongs to 'manga' category which is allowed for +-- deletion. But this fails because the UPDATE action is matched first and +-- UPDATE policy does not allow updation in the category. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes = '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- UPDATE action is not matched this time because of the WHEN AND qual. +-- DELETE still fails because role regress_rls_bob does not have SELECT +-- privileges on 'manga' category row in the category table. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +SELECT * FROM document WHERE did = 4; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-----------------+----------------+-------- + 4 | 44 | 1 | regress_rls_bob | my first manga | +(1 row) + +-- Switch to regress_rls_carol role and try the DELETE again. It should succeed +-- this time +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_carol; +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +-- Switch back to regress_rls_bob role +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_bob; +-- Try INSERT action. This fails because we are trying to insert +-- dauthor = regress_rls_dave and INSERT's WITH CHECK does not allow +-- that +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_dave', 'another novel'); +ERROR: new row violates row-level security policy for table "document" +-- This should be fine +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +-- ok +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge4 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +-- drop and create a new SELECT policy which prevents us from reading +-- any document except with category 'magna' +RESET SESSION AUTHORIZATION; +DROP POLICY p1 ON document; +CREATE POLICY p1 ON document FOR SELECT + USING (cid = (SELECT cid from category WHERE cname = 'manga')); +SET SESSION AUTHORIZATION regress_rls_bob; +-- MERGE can no longer see the matching row and hence attempts the +-- NOT MATCHED action, which results in unique key violation +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge5 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +ERROR: duplicate key value violates unique constraint "document_pkey" +RESET SESSION AUTHORIZATION; +-- drop the restrictive SELECT policy so that we can look at the +-- final state of the table +DROP POLICY p1 ON document; +-- Just check everything went per plan +SELECT * FROM document; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-------------------+----------------------------------+----------------------------------------------------------------------- + 3 | 22 | 2 | regress_rls_bob | my science fiction | + 5 | 44 | 2 | regress_rls_bob | my second manga | + 6 | 22 | 1 | regress_rls_carol | great science fiction | + 7 | 33 | 2 | regress_rls_carol | great technology book | + 8 | 44 | 1 | regress_rls_carol | great manga | + 9 | 22 | 1 | regress_rls_dave | awesome science fiction | + 10 | 33 | 2 | regress_rls_dave | awesome technology book | + 11 | 33 | 1 | regress_rls_carol | hoge | + 33 | 22 | 1 | regress_rls_bob | okay science fiction | + 2 | 11 | 2 | regress_rls_bob | my first novel | + 78 | 33 | 1 | regress_rls_bob | some technology novel | + 79 | 33 | 1 | regress_rls_bob | technology book, can only insert | + 12 | 11 | 1 | regress_rls_bob | another novel | + 1 | 11 | 1 | regress_rls_bob | my first novel | notes added by merge2 notes added by merge3 notes added by merge4 +(14 rows) + -- -- ROLE/GROUP -- diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 5149b72fe9..d369a73173 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -3263,6 +3263,37 @@ CREATE RULE rules_parted_table_insert AS ON INSERT to rules_parted_table ALTER RULE rules_parted_table_insert ON rules_parted_table RENAME TO rules_parted_table_insert_redirect; DROP TABLE rules_parted_table; -- +-- test MERGE +-- +CREATE TABLE rule_merge1 (a int, b text); +CREATE TABLE rule_merge2 (a int, b text); +CREATE RULE rule1 AS ON INSERT TO rule_merge1 + DO INSTEAD INSERT INTO rule_merge2 VALUES (NEW.*); +CREATE RULE rule2 AS ON UPDATE TO rule_merge1 + DO INSTEAD UPDATE rule_merge2 SET a = NEW.a, b = NEW.b + WHERE a = OLD.a; +CREATE RULE rule3 AS ON DELETE TO rule_merge1 + DO INSTEAD DELETE FROM rule_merge2 WHERE a = OLD.a; +-- MERGE not supported for table with rules +MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); +ERROR: MERGE is not supported for relations with rules +-- should be ok with the other table though +MERGE INTO rule_merge2 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); +-- -- Test enabling/disabling -- CREATE TABLE ruletest1 (a int); diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out index 387e40d67d..7f4a94ef7d 100644 --- a/src/test/regress/expected/triggers.out +++ b/src/test/regress/expected/triggers.out @@ -2761,6 +2761,54 @@ delete from self_ref where a = 1; NOTICE: trigger_func(self_ref) called: action = DELETE, when = BEFORE, level = STATEMENT NOTICE: trigger = self_ref_s_trig, old table = (1,), (2,1), (3,2), (4,3) drop table self_ref; +-- +-- test transition tables with MERGE +-- +create table merge_target_table (a int primary key, b text); +create trigger merge_target_table_insert_trig + after insert on merge_target_table referencing new table as new_table + for each statement execute procedure dump_insert(); +create trigger merge_target_table_update_trig + after update on merge_target_table referencing old table as old_table new table as new_table + for each statement execute procedure dump_update(); +create trigger merge_target_table_delete_trig + after delete on merge_target_table referencing old table as old_table + for each statement execute procedure dump_delete(); +create table merge_source_table (a int, b text); +insert into merge_source_table + values (1, 'initial1'), (2, 'initial2'), + (3, 'initial3'), (4, 'initial4'); +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_insert_trig, new table = (1,initial1), (2,initial2), (3,initial3), (4,initial4) +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_delete_trig, old table = (3,initial3), (4,initial4) +NOTICE: trigger = merge_target_table_update_trig, old table = (1,initial1), (2,initial2), new table = (1,"initial1 updated by merge"), (2,"initial2 updated by merge") +NOTICE: trigger = merge_target_table_insert_trig, new table = <NULL> +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated again by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_delete_trig, old table = <NULL> +NOTICE: trigger = merge_target_table_update_trig, old table = (1,"initial1 updated by merge"), (2,"initial2 updated by merge"), new table = (1,"initial1 updated by merge updated again by merge"), (2,"initial2 updated by merge updated again by merge") +NOTICE: trigger = merge_target_table_insert_trig, new table = (3,initial3), (4,initial4) +drop table merge_source_table, merge_target_table; -- cleanup drop function dump_insert(); drop function dump_update(); diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index d308a05117..bfb9f11156 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -84,7 +84,7 @@ test: select_into select_distinct select_distinct_on select_implicit select_havi # ---------- # Another group of parallel tests # ---------- -test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password +test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password merge # ---------- # Another group of parallel tests diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 45147e9328..900814d33c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -122,6 +122,7 @@ test: tablesample test: groupingsets test: drop_operator test: password +test: merge test: alter_generic test: alter_operator test: misc diff --git a/src/test/regress/sql/identity.sql b/src/test/regress/sql/identity.sql index a35f331f4e..f8f34eaf18 100644 --- a/src/test/regress/sql/identity.sql +++ b/src/test/regress/sql/identity.sql @@ -246,3 +246,48 @@ CREATE TABLE itest_child PARTITION OF itest_parent ( f3 WITH OPTIONS GENERATED ALWAYS AS IDENTITY ) FOR VALUES FROM ('2016-07-01') TO ('2016-08-01'); -- error DROP TABLE itest_parent; + +-- MERGE tests +CREATE TABLE itest14 (a int GENERATED ALWAYS AS IDENTITY, b text); +CREATE TABLE itest15 (a int GENERATED BY DEFAULT AS IDENTITY, b text); + +MERGE INTO itest14 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); + +MERGE INTO itest14 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest14 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); + +SELECT * FROM itest14; +SELECT * FROM itest15; +DROP TABLE itest14; +DROP TABLE itest15; diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql new file mode 100644 index 0000000000..8b5244fc63 --- /dev/null +++ b/src/test/regress/sql/merge.sql @@ -0,0 +1,1068 @@ +-- +-- MERGE +-- +--\set VERBOSITY verbose + +--set debug_print_rewritten = true; +--set debug_print_parse = true; +--set debug_print_pretty = true; + + +CREATE USER merge_privs; +CREATE USER merge_no_privs; +DROP TABLE IF EXISTS target; +DROP TABLE IF EXISTS source; +CREATE TABLE target (tid integer, balance integer); +CREATE TABLE source (sid integer, delta integer); --no index +INSERT INTO target VALUES (1, 10); +INSERT INTO target VALUES (2, 20); +INSERT INTO target VALUES (3, 30); +SELECT t.ctid is not null as matched, t.*, s.* FROM source s FULL OUTER JOIN target t ON s.sid = t.tid ORDER BY t.tid, s.sid; + +ALTER TABLE target OWNER TO merge_privs; +ALTER TABLE source OWNER TO merge_privs; + +CREATE TABLE target2 (tid integer, balance integer); +CREATE TABLE source2 (sid integer, delta integer); + +ALTER TABLE target2 OWNER TO merge_no_privs; +ALTER TABLE source2 OWNER TO merge_no_privs; + +GRANT INSERT ON target TO merge_no_privs; + +SET SESSION AUTHORIZATION merge_privs; + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + +-- +-- Errors +-- +MERGE INTO target t RANDOMWORD +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +-- MATCHED/INSERT error +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + INSERT DEFAULT VALUES +; +-- incorrectly specifying INTO target +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT INTO target DEFAULT VALUES +; +-- NOT MATCHED/UPDATE +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + UPDATE SET balance = 0 +; +-- UPDATE tablename +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE target SET balance = 0 +; + +-- permissions + +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_no_privs; + +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +GRANT UPDATE ON target2 TO merge_privs; +SET SESSION AUTHORIZATION merge_privs; + +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN MATCHED THEN + DELETE +; + +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; + +-- check if the target can be accessed from source relation subquery; we should +-- not be able to do so +MERGE INTO target t +USING (SELECT * FROM source WHERE t.tid > sid) s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; + +-- +-- initial tests +-- +-- zero rows in source has no effect +MERGE INTO target +USING source +ON target.tid = source.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ROLLBACK; + +-- insert some non-matching source rows to work from +INSERT INTO source VALUES (4, 40); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + DO NOTHING +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- index plans +INSERT INTO target SELECT generate_series(1000,2500), 0; +ALTER TABLE target ADD PRIMARY KEY (tid); +ANALYZE target; + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL); +; +DELETE FROM target WHERE tid > 100; +ANALYZE target; + +-- insert some matching source rows to work from +INSERT INTO source VALUES (2, 5); +INSERT INTO source VALUES (3, 20); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +-- equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- equivalent of a DELETE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- duplicate source row causes multiple target row update ERROR +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +ROLLBACK; + +-- correct source data +DELETE FROM source WHERE sid = 2; +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +-- remove constraints +alter table target drop CONSTRAINT target_pkey; +alter table target alter column tid drop not null; + +-- multiple actions +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4) +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- should be equivalent +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4); +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- column references +-- do a simple equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- do a simple equivalent of an INSERT SELECT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- and again with explicitly identified column list +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- and again with a subtle error: referring to non-existent target row for NOT MATCHED +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; + +-- and again with a constant ON clause +BEGIN; +MERGE INTO target t +USING source AS s +ON (SELECT true) +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- now the classic UPSERT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- unreachable WHEN clause should ERROR +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN /* Terminal WHEN clause for MATCHED */ + DELETE +WHEN MATCHED AND s.delta > 0 THEN + UPDATE SET balance = t.balance - s.delta +; +ROLLBACK; + +-- conditional WHEN clause +CREATE TABLE wq_target (tid integer not null, balance integer DEFAULT -1); +CREATE TABLE wq_source (balance integer, sid integer); + +INSERT INTO wq_source (sid, balance) VALUES (1, 100); + +BEGIN; +-- try a simple INSERT with default values first +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +-- this time with a FALSE condition +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND FALSE THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +-- this time with an actual condition which returns false +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance <> 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +BEGIN; +-- and now with a condition which returns true +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +-- conditions in the NOT MATCHED clause can only refer to source columns +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND t.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +-- conditions in MATCHED clause can refer to both source and target +SELECT * FROM wq_source; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if AND works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if OR works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 199 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if subqueries work in the conditions? +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance > (SELECT max(balance) FROM target) THEN + UPDATE SET balance = t.balance + s.balance; + +-- check if we can access system columns in the conditions +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.xmin = t.xmax THEN + UPDATE SET balance = t.balance + s.balance; + +ALTER TABLE wq_target SET WITH OIDS; +SELECT * FROM wq_target; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.oid >= 0 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- test preventing WHEN AND conditions from writing to the database +create or replace function merge_when_and_write() returns boolean +language plpgsql as +$$ +BEGIN + INSERT INTO target VALUES (100, 100); + RETURN TRUE; +END; +$$; + +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND (merge_when_and_write()) THEN + UPDATE SET balance = t.balance + s.balance; +ROLLBACK; +drop function merge_when_and_write(); + +DROP TABLE wq_target, wq_source; + +-- test triggers +create or replace function merge_trigfunc () returns trigger +language plpgsql as +$$ +BEGIN + RAISE NOTICE '% % % trigger', TG_WHEN, TG_OP, TG_LEVEL; + IF (TG_WHEN = 'BEFORE' AND TG_LEVEL = 'ROW') THEN + IF (TG_OP = 'DELETE') THEN + RETURN OLD; + ELSE + RETURN NEW; + END IF; + ELSE + RETURN NULL; + END IF; +END; +$$; +CREATE TRIGGER merge_bsi BEFORE INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsu BEFORE UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsd BEFORE DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asi AFTER INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asu AFTER UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asd AFTER DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bri BEFORE INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bru BEFORE UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_brd BEFORE DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ari AFTER INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_aru AFTER UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ard AFTER DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); + +-- now the classic UPSERT, with a DELETE +BEGIN; +UPDATE target SET balance = 0 WHERE tid = 3; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- test from PL/pgSQL +-- make sure MERGE INTO isn't interpreted to mean returning variables like SELECT INTO +BEGIN; +DO LANGUAGE plpgsql $$ +BEGIN +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +; +END; +$$; +ROLLBACK; + +--source constants +BEGIN; +MERGE INTO target t +USING (SELECT 9 AS sid, 57 AS delta) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +--source query +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta as newname FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.newname) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +--self-merge +BEGIN; +MERGE INTO target t1 +USING target t2 +ON t1.tid = t2.tid +WHEN MATCHED THEN + UPDATE SET balance = t1.balance + t2.balance +WHEN NOT MATCHED THEN + INSERT VALUES (t2.tid, t2.balance) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING (SELECT tid as sid, balance as delta FROM target WHERE balance > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING +(SELECT sid, max(delta) AS delta + FROM source + GROUP BY sid + HAVING count(*) = 1 + ORDER BY sid ASC) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- plpgsql parameters and results +BEGIN; +CREATE FUNCTION merge_func (p_id integer, p_bal integer) +RETURNS INTEGER +LANGUAGE plpgsql +AS $$ +DECLARE + result integer; +BEGIN +MERGE INTO target t +USING (SELECT p_id AS sid) AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance - p_bal +; +IF FOUND THEN + GET DIAGNOSTICS result := ROW_COUNT; +END IF; +RETURN result; +END; +$$; +SELECT merge_func(3, 4); +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- PREPARE +BEGIN; +prepare foom as merge into target t using (select 1 as sid) s on (t.tid = s.sid) when matched then update set balance = 1; +execute foom; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +PREPARE foom2 (integer, integer) AS +MERGE INTO target t +USING (SELECT 1) s +ON t.tid = $1 +WHEN MATCHED THEN +UPDATE SET balance = $2; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +execute foom2 (1, 1); +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- subqueries in source relation + +CREATE TABLE sq_target (tid integer NOT NULL, balance integer); +CREATE TABLE sq_source (delta integer, sid integer, balance integer DEFAULT 0); + +INSERT INTO sq_target(tid, balance) VALUES (1,100), (2,200), (3,300); +INSERT INTO sq_source(sid, delta) VALUES (1,10), (2,20), (4,40); + +BEGIN; +MERGE INTO sq_target t +USING (SELECT * FROM sq_source) s +ON tid = sid +WHEN MATCHED AND t.balance > delta THEN + UPDATE SET balance = t.balance + delta; +SELECT * FROM sq_target; +ROLLBACK; + +-- try a view +CREATE VIEW v AS SELECT * FROM sq_source WHERE sid < 2; + +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = v.balance + delta; +SELECT * FROM sq_target; +ROLLBACK; + +-- ambiguous reference to a column +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +ROLLBACK; + +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +SELECT * FROM sq_target; +ROLLBACK; + +-- CTEs +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +WITH targq AS ( + SELECT * FROM v +) +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +; +ROLLBACK; + +-- RETURNING +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +RETURNING * +; +ROLLBACK; + +-- Subqueries +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = (SELECT count(*) FROM sq_target) +; +ROLLBACK; + +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND (SELECT count(*) > 0 FROM sq_target) THEN + UPDATE SET balance = 42 +; +ROLLBACK; + +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid AND (SELECT count(*) > 0 FROM sq_target) +WHEN MATCHED THEN + UPDATE SET balance = 42 +; +ROLLBACK; + +DROP TABLE sq_target, sq_source CASCADE; + +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); + +CREATE TABLE part1 PARTITION OF pa_target FOR VALUES IN (1,4); +CREATE TABLE part2 PARTITION OF pa_target FOR VALUES IN (2,5,6); +CREATE TABLE part3 PARTITION OF pa_target FOR VALUES IN (3,8,9); +CREATE TABLE part4 PARTITION OF pa_target DEFAULT; + +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_target CASCADE; + +-- The target table is partitioned in the same way, but this time by attaching +-- partitions which have columns in different order, dropped columns etc. +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 (tid integer, balance float, val text); +CREATE TABLE part2 (balance float, tid integer, val text); +CREATE TABLE part3 (tid integer, balance float, val text); +CREATE TABLE part4 (extraid text, tid integer, balance float, val text); +ALTER TABLE part4 DROP COLUMN extraid; + +ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); +ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); +ALTER TABLE pa_target ATTACH PARTITION part3 FOR VALUES IN (3,8,9); +ALTER TABLE pa_target ATTACH PARTITION part4 DEFAULT; + +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; + +-- Sub-partitionin +CREATE TABLE pa_target (logts timestamp, tid integer, balance float, val text) + PARTITION BY RANGE (logts); + +CREATE TABLE part_m01 PARTITION OF pa_target + FOR VALUES FROM ('2017-01-01') TO ('2017-02-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m01_odd PARTITION OF part_m01 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m01_even PARTITION OF part_m01 + FOR VALUES IN (2,4,6,8); +CREATE TABLE part_m02 PARTITION OF pa_target + FOR VALUES FROM ('2017-02-01') TO ('2017-03-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m02_odd PARTITION OF part_m02 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m02_even PARTITION OF part_m02 + FOR VALUES IN (2,4,6,8); + +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT '2017-01-31', id, id * 100, 'initial' FROM generate_series(1,9,3) AS id; +INSERT INTO pa_target SELECT '2017-02-28', id, id * 100, 'initial' FROM generate_series(2,9,3) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING (SELECT '2017-01-15' AS slogts, * FROM pa_source WHERE sid < 10) s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; + +-- some complex joins on the source side + +CREATE TABLE cj_target (tid integer, balance float, val text); +CREATE TABLE cj_source1 (sid1 integer, scat integer, delta integer); +CREATE TABLE cj_source2 (sid2 integer, sval text); +INSERT INTO cj_source1 VALUES (1, 10, 100); +INSERT INTO cj_source1 VALUES (1, 20, 200); +INSERT INTO cj_source1 VALUES (2, 20, 300); +INSERT INTO cj_source1 VALUES (3, 10, 400); +INSERT INTO cj_source2 VALUES (1, 'initial source2'); +INSERT INTO cj_source2 VALUES (2, 'initial source2'); +INSERT INTO cj_source2 VALUES (3, 'initial source2'); + +-- source relation is an unalised join +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid1, delta, sval); + +-- try accessing columns from either side of the source join +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta, sval) +WHEN MATCHED THEN + DELETE; + +-- some simple expressions in INSERT targetlist +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta + scat, sval) +WHEN MATCHED THEN + UPDATE SET val = val || ' updated by merge'; + +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN MATCHED THEN + UPDATE SET val = val || ' ' || delta::text; + +SELECT * FROM cj_target; + +ALTER TABLE cj_source1 RENAME COLUMN sid1 TO sid; +ALTER TABLE cj_source2 RENAME COLUMN sid2 TO sid; + +TRUNCATE cj_target; + +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON s1.sid = s2.sid +ON t.tid = s1.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s2.sid, delta, sval); + +DROP TABLE cj_source2, cj_source1, cj_target; + +-- Function scans +CREATE TABLE fs_target (a int, b int, c text); +MERGE INTO fs_target t +USING generate_series(1,100,1) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1); + +MERGE INTO fs_target t +USING generate_series(1,100,2) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id, c = 'updated '|| id.*::text +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1, 'inserted ' || id.*::text); + +SELECT count(*) FROM fs_target; +DROP TABLE fs_target; + +-- SERIALIZABLE test +-- handled in isolation tests + +-- prepare + +RESET SESSION AUTHORIZATION; +DROP TABLE target, target2; +DROP TABLE source, source2; +DROP FUNCTION merge_trigfunc(); +DROP USER merge_privs; +DROP USER merge_no_privs; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index f7f3bbbeeb..0a8abf2076 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -349,6 +349,114 @@ UPDATE atest5 SET one = 1; -- fail SELECT atest6 FROM atest6; -- ok COPY atest6 TO stdout; -- ok +-- test column privileges with MERGE +SET SESSION AUTHORIZATION regress_priv_user1; +CREATE TABLE mtarget (a int, b text); +CREATE TABLE msource (a int, b text); +INSERT INTO mtarget VALUES (1, 'init1'), (2, 'init2'); +INSERT INTO msource VALUES (1, 'source1'), (2, 'source2'), (3, 'source3'); + +GRANT SELECT (a) ON msource TO regress_priv_user4; +GRANT SELECT (a) ON mtarget TO regress_priv_user4; +GRANT INSERT (a,b) ON mtarget TO regress_priv_user4; +GRANT UPDATE (b) ON mtarget TO regress_priv_user4; + +SET SESSION AUTHORIZATION regress_priv_user4; + +-- +-- test source privileges +-- + +-- fail (no SELECT priv on s.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- fail (s.b used in the INSERTed values) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- fail (s.b used in the WHEN quals) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND s.b = 'x' THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- this should be ok since only s.a is accessed +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'ok' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ROLLBACK; + +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT SELECT (b) ON msource TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; + +-- should now be ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ROLLBACK; + +-- +-- test target privileges +-- + +-- fail (no SELECT priv on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- fail (no UPDATE on t.a) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b, a = t.a + 1 +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- fail (no SELECT on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b; +ROLLBACK; + +-- fail (no DELETE) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; + +-- grant delete privileges +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT DELETE ON mtarget TO regress_priv_user4; +-- should be ok now +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ROLLBACK; + -- check error reporting with column privs SET SESSION AUTHORIZATION regress_priv_user1; CREATE TABLE t1 (c1 int, c2 int, c3 int check (c3 < 5), primary key (c1, c2)); diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql index f3a31dbee0..6c75208998 100644 --- a/src/test/regress/sql/rowsecurity.sql +++ b/src/test/regress/sql/rowsecurity.sql @@ -812,6 +812,162 @@ INSERT INTO document VALUES (4, (SELECT cid from category WHERE cname = 'novel') INSERT INTO document VALUES (1, (SELECT cid from category WHERE cname = 'novel'), 1, 'regress_rls_bob', 'my first novel') ON CONFLICT (did) DO UPDATE SET dauthor = 'regress_rls_carol'; +-- +-- MERGE +-- +RESET SESSION AUTHORIZATION; +DROP POLICY p3_with_all ON document; + +ALTER TABLE document ADD COLUMN dnotes text DEFAULT ''; +-- all documents are readable +CREATE POLICY p1 ON document FOR SELECT USING (true); +-- one may insert documents only authored by them +CREATE POLICY p2 ON document FOR INSERT WITH CHECK (dauthor = current_user); +-- one may only update documents in 'novel' category +CREATE POLICY p3 ON document FOR UPDATE + USING (cid = (SELECT cid from category WHERE cname = 'novel')) + WITH CHECK (dauthor = current_user); +-- one may only delete documents in 'manga' category +CREATE POLICY p4 ON document FOR DELETE + USING (cid = (SELECT cid from category WHERE cname = 'manga')); + +SELECT * FROM document; + +SET SESSION AUTHORIZATION regress_rls_bob; + +-- Fails, since update violates WITH CHECK qual on dauthor +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge1 ', dauthor = 'regress_rls_alice'; + +-- Should be OK since USING and WITH CHECK quals pass +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge2 '; + +-- Even when dauthor is updated explicitly, but to the existing value +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge3 ', dauthor = 'regress_rls_bob'; + +-- There is a MATCH for did = 3, but UPDATE's USING qual does not allow +-- updating an item in category 'science fiction' +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge '; + +-- The same thing with DELETE action, but fails again because no permissions +-- to delete items in 'science fiction' category that did 3 belongs to. +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE; + +-- Document with did 4 belongs to 'manga' category which is allowed for +-- deletion. But this fails because the UPDATE action is matched first and +-- UPDATE policy does not allow updation in the category. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes = '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +-- UPDATE action is not matched this time because of the WHEN AND qual. +-- DELETE still fails because role regress_rls_bob does not have SELECT +-- privileges on 'manga' category row in the category table. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +SELECT * FROM document WHERE did = 4; + +-- Switch to regress_rls_carol role and try the DELETE again. It should succeed +-- this time +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_carol; + +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +-- Switch back to regress_rls_bob role +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_bob; + +-- Try INSERT action. This fails because we are trying to insert +-- dauthor = regress_rls_dave and INSERT's WITH CHECK does not allow +-- that +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_dave', 'another novel'); + +-- This should be fine +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +-- ok +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge4 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +-- drop and create a new SELECT policy which prevents us from reading +-- any document except with category 'magna' +RESET SESSION AUTHORIZATION; +DROP POLICY p1 ON document; +CREATE POLICY p1 ON document FOR SELECT + USING (cid = (SELECT cid from category WHERE cname = 'manga')); + +SET SESSION AUTHORIZATION regress_rls_bob; + +-- MERGE can no longer see the matching row and hence attempts the +-- NOT MATCHED action, which results in unique key violation +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge5 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +RESET SESSION AUTHORIZATION; +-- drop the restrictive SELECT policy so that we can look at the +-- final state of the table +DROP POLICY p1 ON document; +-- Just check everything went per plan +SELECT * FROM document; + -- -- ROLE/GROUP -- diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql index a82f52d154..b866268892 100644 --- a/src/test/regress/sql/rules.sql +++ b/src/test/regress/sql/rules.sql @@ -1191,6 +1191,39 @@ CREATE RULE rules_parted_table_insert AS ON INSERT to rules_parted_table ALTER RULE rules_parted_table_insert ON rules_parted_table RENAME TO rules_parted_table_insert_redirect; DROP TABLE rules_parted_table; +-- +-- test MERGE +-- +CREATE TABLE rule_merge1 (a int, b text); +CREATE TABLE rule_merge2 (a int, b text); +CREATE RULE rule1 AS ON INSERT TO rule_merge1 + DO INSTEAD INSERT INTO rule_merge2 VALUES (NEW.*); +CREATE RULE rule2 AS ON UPDATE TO rule_merge1 + DO INSTEAD UPDATE rule_merge2 SET a = NEW.a, b = NEW.b + WHERE a = OLD.a; +CREATE RULE rule3 AS ON DELETE TO rule_merge1 + DO INSTEAD DELETE FROM rule_merge2 WHERE a = OLD.a; + +-- MERGE not supported for table with rules +MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); + +-- should be ok with the other table though +MERGE INTO rule_merge2 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); + -- -- Test enabling/disabling -- diff --git a/src/test/regress/sql/triggers.sql b/src/test/regress/sql/triggers.sql index c6f31dd8c8..b51c884eee 100644 --- a/src/test/regress/sql/triggers.sql +++ b/src/test/regress/sql/triggers.sql @@ -2110,6 +2110,53 @@ delete from self_ref where a = 1; drop table self_ref; +-- +-- test transition tables with MERGE +-- +create table merge_target_table (a int primary key, b text); +create trigger merge_target_table_insert_trig + after insert on merge_target_table referencing new table as new_table + for each statement execute procedure dump_insert(); +create trigger merge_target_table_update_trig + after update on merge_target_table referencing old table as old_table new table as new_table + for each statement execute procedure dump_update(); +create trigger merge_target_table_delete_trig + after delete on merge_target_table referencing old table as old_table + for each statement execute procedure dump_delete(); + +create table merge_source_table (a int, b text); +insert into merge_source_table + values (1, 'initial1'), (2, 'initial2'), + (3, 'initial3'), (4, 'initial4'); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when not matched then + insert values (a, b); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated again by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); + +drop table merge_source_table, merge_target_table; + -- cleanup drop function dump_insert(); drop function dump_update(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 17bf55c1f5..a7eb3524cf 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1228,6 +1228,8 @@ MemoryContextCallbackFunction MemoryContextCounters MemoryContextData MemoryContextMethods +MergeAction +MergeActionState MergeAppend MergeAppendPath MergeAppendState @@ -1235,6 +1237,7 @@ MergeJoin MergeJoinClause MergeJoinState MergePath +MergeStmt MergeScanSelCache MetaCommand MinMaxAggInfo -- 2.14.3 (Apple Git-98) ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:28 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> @ 2018-03-27 09:40 ` Simon Riggs <[email protected]> 1 sibling, 0 replies; 81+ messages in thread From: Simon Riggs @ 2018-03-27 09:40 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Amit Langote <[email protected]>; Peter Geoghegan <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 27 March 2018 at 10:28, Pavan Deolasee <[email protected]> wrote: >> 1. In ExecMergeMatched() we have a line of code that does this... >> >> if (TransactionIdIsCurrentTransactionId(hufd.xmax)) >> then errcode(ERRCODE_CARDINALITY_VIOLATION) >> >> I notice this is correct, but too strong. It should be possible to run >> a sequence of MERGE commands inside a transaction, repeatedly updating >> the same set of rows, as is possible with UPDATE. >> >> We need to check whether the xid is the current subxid and the cid is >> the current commandid, rather than using >> TransactionIdIsCurrentTransactionId() > > AFAICS this is fine because we invoke that code only when > HeapTupleSatisfiesUpdate returns HeapTupleSelfUpdated i.e. for the case when > the tuple is updated by our transaction after the scan is started. > HeapTupleSatisfiesUpdate already checks for command id before returning > HeapTupleSelfUpdated. Cool. >> 5. I take it the special code for TransformMerge target relations is >> replaced by "right_rte"? Seems fragile to leave it like that. Can we >> add an Assert()? Do we care? > > I didn't get this point. Can you please explain? The code confused me at that point. More docs pls. >> 6. I didn't understand "Assume that the top-level join RTE is at the >> end. The source relation >> + * is just before that." >> What is there is no source relation? > > > Can that happen? I mean shouldn't there always be a source relation? It > could be a subquery or a function scan or just a plain relation, but > something, right? Yes, OK. So ordering is target, source, join. >> 10. Comment needs updating for changes in code below it - "In MERGE >> when and condition, no system column is allowed" >> > > Yeah, that's kinda half-true since the code below supports TABLEOID and OID > system columns. I am thinking about this in a larger context though. Peter > has expressed desire to support system columns in WHEN targetlist and quals. > I gave it a try and it seems if we remove that error block, all system > columns are supported readily. But only from the target side. There is a > problem if we try to refer a system column from the source side since the > mergrSourceTargetList only includes user columns and so set_plan_refs() > complains about a system column. > > I am not sure what's the best way to handle this. May be we can add system > columns to the mergrSourceTargetList. I haven't yet found a neat way to do > that. I was saying the comment needs changing, not the code. Cool, thanks -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:28 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> @ 2018-03-28 02:58 ` Peter Geoghegan <[email protected]> 2018-03-28 04:30 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-28 11:00 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 1 sibling, 2 replies; 81+ messages in thread From: Peter Geoghegan @ 2018-03-28 02:58 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Simon Riggs <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Tue, Mar 27, 2018 at 2:28 AM, Pavan Deolasee <[email protected]> wrote: > (Version 26) I have some feedback on this version: * ExecMergeMatched() needs to determine tuple lock mode for EvalPlanQual() in a way that's based on how everything else works; it's not okay to just use LockTupleExclusive in all cases. That will often lead to lock escalation, which can cause unprincipled deadlocks. You need to pass back the relevant info from routines like heap_update(), which means more state needs to come back to ExecMergeMatched() from routines like ExecUpdate(). * Doesn't ExecUpdateLockMode(), which is called from places like ExecBRUpdateTriggers(), also need to be taught about GetEPQRangeTableIndex() (i.e. the ri_mergeTargetRTI/ri_RangeTableIndex divide)? You should audit everything like that carefully. Maybe GetEPQRangeTableIndex() is not the best choke-point to do this kind of thing. Not that I have a clearly better idea. * Looks like there is a similar problem in ExecPartitionCheckEmitError(). I don't really understand how that works, so I might be wrong here. * More or less the same issue seems to exist within ExecConstraints(), including where GetInsertedColumns() is used. * Compiler warning: fwrapv -fexcess-precision=standard -Og -g3 -fno-omit-frame-pointer -I../../../src/include -I/code/postgresql/root/build/../source/src/include -D_FORTIFY_SOURCE=2 -D TRUST_STRXFRM -D LOCK_DEBUG -D WAL_DEBUG -D BTREE_BUILD_STATS -D MULTIXACT_DEBUG -D SELECTIVITY_DEBUG -D HJDEBUG -D_GNU_SOURCE -I/usr/include/libxml2 -c -o nodeModifyTable.o /code/postgresql/root/build/../source/src/backend/executor/nodeModifyTable.c /code/postgresql/root/build/../source/src/backend/executor/nodeModifyTable.c: In function ‘ExecInsert’: /code/postgresql/root/build/../source/src/backend/executor/nodeModifyTable.c:412:4: warning: ‘wco_kind’ may be used uninitialized in this function [-Wmaybe-uninitialized] ExecWithCheckOptions(wco_kind, resultRelInfo, slot, estate); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * The BufferIsValid() checks to decide if we need to ReleaseBuffer() within ExecMergeMatched() are unnecessary -- a buffer pin must be held throughout. This looks like it's leftover from before the ExecMergeNotMatched()/ExecMergeMatched() split was made. * There should be ResetExprContext() calls for your new MergeActionState projections. That's what we see for the RETURNING + ON CONFLICT projections within nodeModifyTable.c, which the new projections are very similar to, and clearly modeled on. -- Peter Geoghegan ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:28 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-28 02:58 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> @ 2018-03-28 04:30 ` Pavan Deolasee <[email protected]> 1 sibling, 0 replies; 81+ messages in thread From: Pavan Deolasee @ 2018-03-28 04:30 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Simon Riggs <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Wed, Mar 28, 2018 at 8:28 AM, Peter Geoghegan <[email protected]> wrote: > On Tue, Mar 27, 2018 at 2:28 AM, Pavan Deolasee > <[email protected]> wrote: > > (Version 26) > > I have some feedback on this version: > > * ExecMergeMatched() needs to determine tuple lock mode for > EvalPlanQual() in a way that's based on how everything else works; > it's not okay to just use LockTupleExclusive in all cases. That will > often lead to lock escalation, which can cause unprincipled deadlocks. > You need to pass back the relevant info from routines like > heap_update(), which means more state needs to come back to > ExecMergeMatched() from routines like ExecUpdate(). > You're right. I am thinking what would be a good way to pass that information back. Should we add a new out parameter to ExecUpdate() or a new member to HeapUpdateFailureData? It seems only ExecUpdate() would need the change, so may be it's fine to change the API but HeapUpdateFailureData doesn't look bad either since it deals with failure cases and we are indeed dealing with ExecUpdate() failure. Any preference? > > * Doesn't ExecUpdateLockMode(), which is called from places like > ExecBRUpdateTriggers(), also need to be taught about > GetEPQRangeTableIndex() (i.e. the ri_mergeTargetRTI/ri_RangeTableIndex > divide)? You should audit everything like that carefully. Maybe > GetEPQRangeTableIndex() is not the best choke-point to do this kind of > thing. Not that I have a clearly better idea. > > * Looks like there is a similar problem in > ExecPartitionCheckEmitError(). I don't really understand how that > works, so I might be wrong here. > > * More or less the same issue seems to exist within ExecConstraints(), > including where GetInsertedColumns() is used. > They all look fine to me. Remember that we always resolve column references in WHEN targetlist from the target relation and hence things like updatedCols/insertedCols are get set on the target RTE. All of these routines read from the target RTE as far as I can see. But I will check in more detail. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:28 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-28 02:58 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> @ 2018-03-28 11:00 ` Pavan Deolasee <[email protected]> 2018-03-29 09:50 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 1 sibling, 1 reply; 81+ messages in thread From: Pavan Deolasee @ 2018-03-28 11:00 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Simon Riggs <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Wed, Mar 28, 2018 at 8:28 AM, Peter Geoghegan <[email protected]> wrote: > > * ExecMergeMatched() needs to determine tuple lock mode for > EvalPlanQual() in a way that's based on how everything else works; > it's not okay to just use LockTupleExclusive in all cases. That will > often lead to lock escalation, which can cause unprincipled deadlocks. > You need to pass back the relevant info from routines like > heap_update(), which means more state needs to come back to > ExecMergeMatched() from routines like ExecUpdate(). > Fixed by adding a new member to HeapUpdateFailureData. That seemed more natural, but we can change if others disagree. > > * Doesn't ExecUpdateLockMode(), which is called from places like > ExecBRUpdateTriggers(), also need to be taught about > GetEPQRangeTableIndex() (i.e. the ri_mergeTargetRTI/ri_RangeTableIndex > divide)? You should audit everything like that carefully. Maybe > GetEPQRangeTableIndex() is not the best choke-point to do this kind of > thing. Not that I have a clearly better idea. > > * Looks like there is a similar problem in > ExecPartitionCheckEmitError(). I don't really understand how that > works, so I might be wrong here. > > * More or less the same issue seems to exist within ExecConstraints(), > including where GetInsertedColumns() is used. > As I said, I do not see a problem with this. The insertedCols/updatedCols etc are tracked in the target table's RTE and hence we should continue to use ri_RangeTableIndex for such purposes. The ri_mergeTargetRTI is only useful to running EvalPlanQual correctly. > > * Compiler warning: > > fwrapv -fexcess-precision=standard -Og -g3 -fno-omit-frame-pointer > -I../../../src/include > -I/code/postgresql/root/build/../source/src/include > -D_FORTIFY_SOURCE=2 -D TRUST_STRXFRM -D LOCK_DEBUG -D WAL_DEBUG -D > BTREE_BUILD_STATS -D MULTIXACT_DEBUG -D SELECTIVITY_DEBUG -D HJDEBUG > -D_GNU_SOURCE -I/usr/include/libxml2 -c -o nodeModifyTable.o > /code/postgresql/root/build/../source/src/backend/executor/ > nodeModifyTable.c > /code/postgresql/root/build/../source/src/backend/executor/ > nodeModifyTable.c: > In function ‘ExecInsert’: > /code/postgresql/root/build/../source/src/backend/executor/ > nodeModifyTable.c:412:4: > warning: ‘wco_kind’ may be used uninitialized in this function > [-Wmaybe-uninitialized] > ExecWithCheckOptions(wco_kind, resultRelInfo, slot, estate); > ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Fixed. > > * The BufferIsValid() checks to decide if we need to ReleaseBuffer() > within ExecMergeMatched() are unnecessary -- a buffer pin must be held > throughout. This looks like it's leftover from before the > ExecMergeNotMatched()/ExecMergeMatched() split was made. > Fixed. > > * There should be ResetExprContext() calls for your new > MergeActionState projections. That's what we see for the RETURNING + > ON CONFLICT projections within nodeModifyTable.c, which the new > projections are very similar to, and clearly modeled on. > Right. I thought about what would the right place to call ResetExprContext() and chose to do that in ExecMerge(). I am actually a bit surprised that ExecProcessReturning() and ExecOnConflictUpdate() call ResetExprContext(). Aren't they both using mtstate->ps.ps_ExprContext? If so, why is it safe to reset the expression context before we are completely done with the previous tuple? Clearly, the current code is working fine, so may be there is no bug, but it looks curious. On Tue, Mar 27, 2018 at 4:16 PM, Simon Riggs <[email protected]> wrote: > > More comments on v26 > > * Change errmsg “Ensure that not more than one source rows match any > one target row” > should be > “Ensure that not more than one source row matches any one target row” > Fixed. > > * I think we need an example in the docs showing a constant source > query, so people can understand how to use MERGE for OLTP as well as > large ELT > > * Long comment in ExecMerge() needs rewording, formatting and spell check > Tried. Please check and suggest improvements, if any. > I suggest not referring to an "order" since that concept doesn't exist > anywhere else > You mean when we say that the actions are evaluated "in an order"? > > * Need tests for coverage of these ERROR messages > Named security policy violation > Surprisingly none exists even for regular UPDATE/INSERT/DELETE. I will see what is needed to trigger that error message. > SELECT not allowed in MERGE INSERT... > Multiple VALUES clauses not... > MERGE is not supported for this... > MERGE is not supported for relations with inheritance > MERGE is not supported for relations with rules Added a test for each of those. v27 attached, though review changes are in the add-on 0005 patch. Apart from that - ran the patch through spell-checker and fixed a few typos - reorganised the code in 0004 a bit. - rebased on current master Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services Attachments: [application/octet-stream] v27-0005-Fixes-per-review-comments.patch (21.9K, ../../CABOikdMO4G3dBL9HSzP9DBT051uGnd0uejnYAEZk+iYn3AWY2g@mail.gmail.com/3-v27-0005-Fixes-per-review-comments.patch) download | inline diff: From 28c6a5aa7a3f1018738581489f7b03517850bf89 Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Wed, 28 Mar 2018 11:44:48 +0530 Subject: [PATCH v27 5/5] Fixes per review comments Use the correct lockmode while running EvalPlanQual Fix some typos matviews can't use MERGE, just like they can't use UPDATE/INSERTs Fix a compiler warning buffer can be released without checking BufferIsValid() Couple of new test cases to check ERROR conditions and some minor cleanup Call ResetExprContext() once per tuple processed --- src/backend/access/heap/heapam.c | 25 ++++++------- src/backend/executor/nodeMerge.c | 51 ++++++++++++++++----------- src/backend/executor/nodeModifyTable.c | 13 ++++--- src/backend/parser/parse_merge.c | 9 +++-- src/include/access/heapam.h | 12 ++++++- src/test/regress/expected/merge.out | 64 ++++++++++++++++++++++++++++++++-- src/test/regress/sql/merge.sql | 58 ++++++++++++++++++++++++++++++ 7 files changed, 184 insertions(+), 48 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index decc3d37c3..f96567f5d5 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -3508,7 +3508,7 @@ simple_heap_delete(Relation relation, ItemPointer tid) HTSU_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, - HeapUpdateFailureData *hufd, LockTupleMode *lockmode) + HeapUpdateFailureData *hufd) { HTSU_Result result; TransactionId xid = GetCurrentTransactionId(); @@ -3548,8 +3548,10 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, infomask2_old_tuple, infomask_new_tuple, infomask2_new_tuple; + LockTupleMode lockmode; Assert(ItemPointerIsValid(otid)); + Assert(hufd != NULL); /* * Forbid this during a parallel operation, lest it allocate a combocid. @@ -3665,7 +3667,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, */ if (!bms_overlap(modified_attrs, key_attrs)) { - *lockmode = LockTupleNoKeyExclusive; + lockmode = hufd->lockmode = LockTupleNoKeyExclusive; mxact_status = MultiXactStatusNoKeyUpdate; key_intact = true; @@ -3682,7 +3684,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, } else { - *lockmode = LockTupleExclusive; + lockmode = hufd->lockmode = LockTupleExclusive; mxact_status = MultiXactStatusUpdate; key_intact = false; } @@ -3760,12 +3762,12 @@ l2: int remain; if (DoesMultiXactIdConflict((MultiXactId) xwait, infomask, - *lockmode)) + lockmode)) { LockBuffer(buffer, BUFFER_LOCK_UNLOCK); /* acquire tuple lock, if necessary */ - heap_acquire_tuplock(relation, &(oldtup.t_self), *lockmode, + heap_acquire_tuplock(relation, &(oldtup.t_self), lockmode, LockWaitBlock, &have_tuple_lock); /* wait for multixact */ @@ -3849,7 +3851,7 @@ l2: * lock. */ LockBuffer(buffer, BUFFER_LOCK_UNLOCK); - heap_acquire_tuplock(relation, &(oldtup.t_self), *lockmode, + heap_acquire_tuplock(relation, &(oldtup.t_self), lockmode, LockWaitBlock, &have_tuple_lock); XactLockTableWait(xwait, relation, &oldtup.t_self, XLTW_Update); @@ -3897,7 +3899,7 @@ l2: hufd->cmax = InvalidCommandId; UnlockReleaseBuffer(buffer); if (have_tuple_lock) - UnlockTupleTuplock(relation, &(oldtup.t_self), *lockmode); + UnlockTupleTuplock(relation, &(oldtup.t_self), lockmode); if (vmbuffer != InvalidBuffer) ReleaseBuffer(vmbuffer); bms_free(hot_attrs); @@ -3935,7 +3937,7 @@ l2: compute_new_xmax_infomask(HeapTupleHeaderGetRawXmax(oldtup.t_data), oldtup.t_data->t_infomask, oldtup.t_data->t_infomask2, - xid, *lockmode, true, + xid, lockmode, true, &xmax_old_tuple, &infomask_old_tuple, &infomask2_old_tuple); @@ -4052,7 +4054,7 @@ l2: compute_new_xmax_infomask(HeapTupleHeaderGetRawXmax(oldtup.t_data), oldtup.t_data->t_infomask, oldtup.t_data->t_infomask2, - xid, *lockmode, false, + xid, lockmode, false, &xmax_lock_old_tuple, &infomask_lock_old_tuple, &infomask2_lock_old_tuple); @@ -4364,7 +4366,7 @@ l2: * Release the lmgr tuple lock, if we had it. */ if (have_tuple_lock) - UnlockTupleTuplock(relation, &(oldtup.t_self), *lockmode); + UnlockTupleTuplock(relation, &(oldtup.t_self), lockmode); pgstat_count_heap_update(relation, use_hot_update); @@ -4588,12 +4590,11 @@ simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup) { HTSU_Result result; HeapUpdateFailureData hufd; - LockTupleMode lockmode; result = heap_update(relation, otid, tup, GetCurrentCommandId(true), InvalidSnapshot, true /* wait for commit */ , - &hufd, &lockmode); + &hufd); switch (result) { case HeapTupleSelfUpdated: diff --git a/src/backend/executor/nodeMerge.c b/src/backend/executor/nodeMerge.c index ee41ed2eb2..67f1bfbb16 100644 --- a/src/backend/executor/nodeMerge.c +++ b/src/backend/executor/nodeMerge.c @@ -139,7 +139,6 @@ ExecMergeMatched(ModifyTableState *mtstate, EState *estate, lmerge_matched:; slot = saved_slot; - buffer = InvalidBuffer; /* * UPDATE/DELETE is only invoked for matched rows. And we must have found @@ -263,7 +262,7 @@ lmerge_matched:; ereport(ERROR, (errcode(ERRCODE_CARDINALITY_VIOLATION), errmsg("MERGE command cannot affect row a second time"), - errhint("Ensure that not more than one source rows match any one target row"))); + errhint("Ensure that not more than one source row match any one target row"))); /* This shouldn't happen */ elog(ERROR, "attempted to update or delete invisible tuple"); break; @@ -284,7 +283,7 @@ lmerge_matched:; * tuple. If EvalPlanQual() does not return a tuple (can * that happen?), then again we switch to NOT MATCHED * action. If it does return a tuple and the join qual is - * still satified, then we just need to recheck the + * still satisfied, then we just need to recheck the * MATCHED actions, starting from the top, and execute the * first qualifying action. */ @@ -335,8 +334,7 @@ lmerge_matched:; * newer tuple found in the update chain. */ *tupleid = hufd.ctid; - if (BufferIsValid(buffer)) - ReleaseBuffer(buffer); + ReleaseBuffer(buffer); goto lmerge_matched; } } @@ -348,8 +346,7 @@ lmerge_matched:; */ *tupleid = hufd.ctid; estate->es_result_relation_info = saved_resultRelInfo; - if (BufferIsValid(buffer)) - ReleaseBuffer(buffer); + ReleaseBuffer(buffer); return false; default: @@ -365,14 +362,13 @@ lmerge_matched:; /* * We've activated one of the WHEN clauses, so we don't search - * further. This is required behaviour, not an optimisation. + * further. This is required behaviour, not an optimization. */ estate->es_result_relation_info = saved_resultRelInfo; break; } - if (BufferIsValid(buffer)) - ReleaseBuffer(buffer); + ReleaseBuffer(buffer); /* * Successfully executed an action or no qualifying action was found. @@ -403,7 +399,7 @@ ExecMergeNotMatched(ModifyTableState *mtstate, EState *estate, resultRelInfo = mtstate->resultRelInfo; /* - * For INSERT actions, root relation's merge action is OK since the the + * For INSERT actions, root relation's merge action is OK since the * INSERT's targetlist and the WHEN conditions can only refer to the * source relation and hence it does not matter which result relation we * work with. @@ -484,6 +480,7 @@ void ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, JunkFilter *junkfilter, ResultRelInfo *resultRelInfo) { + ExprContext *econtext = mtstate->ps.ps_ExprContext; ItemPointer tupleid; ItemPointerData tuple_ctid; bool matched = false; @@ -493,9 +490,14 @@ ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, relkind = resultRelInfo->ri_RelationDesc->rd_rel->relkind; Assert(relkind == RELKIND_RELATION || - relkind == RELKIND_MATVIEW || relkind == RELKIND_PARTITIONED_TABLE); + /* + * Reset per-tuple memory context to free any expression evaluation + * storage allocated in the previous cycle. + */ + ResetExprContext(econtext); + /* * We run a JOIN between the target relation and the source relation to * find a set of candidate source rows that has matching row in the target @@ -523,12 +525,12 @@ ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, /* * If we are dealing with a WHEN MATCHED case, we look at the given WHEN - * MATCHED actions in an order and execute the first action which also - * satisfies the additional WHEN MATCHED AND quals. If an action without - * any additional quals is found, that action is executed. + * MATCHED actions in an order and execute the first action for which + * the additional WHEN MATCHED AND quals pass. If an action without + * quals is found, that action is executed. * * Similarly, if we are dealing with WHEN NOT MATCHED case, we look at the - * given WHEN NOT MATCHED actions in an ordr and execute the first + * given WHEN NOT MATCHED actions in an order and execute the first * qualifying action. * * Things get interesting in case of concurrent update/delete of the @@ -538,9 +540,9 @@ ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, * A concurrent update for example can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR 2. - * modify the target tuple so that the join quals no longer pass and hence - * the source tuple no longer has a match. + * additional quals attached to the current WHEN MATCHED action OR + * 2. modify the target tuple so that the join quals no longer pass and + * hence the source tuple no longer has a match. * * In the first case, we are still dealing with a WHEN MATCHED case, but * we should recheck the list of WHEN MATCHED actions and choose the first @@ -559,7 +561,14 @@ ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, * from ExecMergeNotMatched to ExecMergeMatched, there is no risk of a * livelock. */ - if (!matched || - !ExecMergeMatched(mtstate, estate, slot, junkfilter, tupleid)) + if (matched) + matched = ExecMergeMatched(mtstate, estate, slot, junkfilter, tupleid); + + /* + * Either we were dealing with a NOT MATCHED tuple or ExecMergeNotMatched() + * returned "false", indicating the previously MATCHED tuple is no longer a + * matching tuple. + */ + if (!matched) ExecMergeNotMatched(mtstate, estate, slot); } diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 08f812fc6d..5798eb12a5 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -398,11 +398,11 @@ ExecInsert(ModifyTableState *mtstate, */ if (mtstate->operation == CMD_UPDATE) wco_kind = WCO_RLS_UPDATE_CHECK; - else if (mtstate->operation == CMD_INSERT) - wco_kind = WCO_RLS_INSERT_CHECK; else if (mtstate->operation == CMD_MERGE) wco_kind = (actionState->commandType == CMD_UPDATE) ? WCO_RLS_UPDATE_CHECK : WCO_RLS_INSERT_CHECK; + else + wco_kind = WCO_RLS_INSERT_CHECK; /* * ExecWithCheckOptions() will skip any WCOs which are not of the kind @@ -1083,7 +1083,6 @@ ExecUpdate(ModifyTableState *mtstate, } else { - LockTupleMode lockmode; bool partition_constraint_failed; /* @@ -1204,7 +1203,7 @@ lreplace:; * retrieve the tuple conversion map for this resultRelInfo. * * If we're running MERGE then resultRelInfo is per-partition - * resultRelInfo as initialised in ExecInitPartitionInfo(). Note + * resultRelInfo as initialized in ExecInitPartitionInfo(). Note * that we don't expand inheritance for the resultRelation in case * of MERGE and hence there is just one subplan. Whereas for * regular UPDATE, resultRelInfo is one of the per-subplan @@ -1285,7 +1284,7 @@ lreplace:; estate->es_output_cid, estate->es_crosscheck_snapshot, true /* wait for commit */ , - &hufd, &lockmode); + &hufd); /* * Copy the necessary information, if the caller has asked for it. We @@ -1357,7 +1356,7 @@ lreplace:; epqstate, resultRelationDesc, GetEPQRangeTableIndex(resultRelInfo), - lockmode, + hufd.lockmode, &hufd.ctid, hufd.xmax); if (!TupIsNull(epqslot)) @@ -2684,7 +2683,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) mtstate->mt_partition_tuple_routing ? NULL : relationDesc); - /* initialise slot for merge actions */ + /* initialize slot for merge actions */ Assert(mtstate->mt_mergeproj == NULL); mtstate->mt_mergeproj = ExecInitExtraTupleSlot(mtstate->ps.state, diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c index 6df63439bb..25416a321c 100644 --- a/src/backend/parser/parse_merge.c +++ b/src/backend/parser/parse_merge.c @@ -83,7 +83,7 @@ transformMergeJoinClause(ParseState *pstate, Node *merge, * We created an internal join between the target and the source relation * to carry out the MERGE actions. Normally such an unaliased join hides * the joining relations, unless the column references are qualified. - * Also, any unqualified column refernces are resolved to the Join RTE, if + * Also, any unqualified column references are resolved to the Join RTE, if * there is a matching entry in the targetlist. But the way MERGE * execution is later setup, we expect all column references to resolve to * either the source or the target relation. Hence we must not add the @@ -351,7 +351,7 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) * Simplify the MERGE query as much as possible * * These seem like things that could go into Optimizer, but they are - * semantic simplications rather than optimizations, per se. + * semantic simplifications rather than optimizations, per se. * * If there are no INSERT actions we won't be using the non-matching * candidate rows for anything, so no need for an outer join. We do still @@ -426,7 +426,6 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) * XXX MERGE is unsupported in various cases */ if (!(pstate->p_target_relation->rd_rel->relkind == RELKIND_RELATION || - pstate->p_target_relation->rd_rel->relkind == RELKIND_MATVIEW || pstate->p_target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -463,7 +462,7 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) /* * Set namespace for the specific action. This must be done before - * analysing the WHEN quals and the action targetlisst. + * analyzing the WHEN quals and the action targetlisst. */ setNamespaceForMergeAction(pstate, action); @@ -471,7 +470,7 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) * Transform the when condition. * * Note that these quals are NOT added to the join quals; instead they - * are evaluated sepaartely during execution to decide which of the + * are evaluated separately during execution to decide which of the * WHEN MATCHED or WHEN NOT MATCHED actions to execute. */ action->qual = transformWhereClause(pstate, action->condition, diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 100174138d..608f50b061 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -53,17 +53,26 @@ typedef enum LockTupleMode * When heap_update, heap_delete, or heap_lock_tuple fail because the target * tuple is already outdated, they fill in this struct to provide information * to the caller about what happened. + * + * result is the result of HeapTupleSatisfiesUpdate, leading to the failure. + * It's set to HeapTupleMayBeUpdated when there is no failure. + * * ctid is the target's ctid link: it is the same as the target's TID if the * target was deleted, or the location of the replacement tuple if the target * was updated. + * * xmax is the outdating transaction's XID. If the caller wants to visit the * replacement tuple, it must check that this matches before believing the * replacement is really a match. + * * cmax is the outdating command's CID, but only when the failure code is * HeapTupleSelfUpdated (i.e., something in the current transaction outdated * the tuple); otherwise cmax is zero. (We make this restriction because * HeapTupleHeaderGetCmax doesn't work for tuples outdated in other * transactions.) + * + * lockmode is only relevant for callers of heap_update() and is the mode which + * the caller should use in case it needs to lock the updated tuple. */ typedef struct HeapUpdateFailureData { @@ -71,6 +80,7 @@ typedef struct HeapUpdateFailureData ItemPointerData ctid; TransactionId xmax; CommandId cmax; + LockTupleMode lockmode; } HeapUpdateFailureData; @@ -163,7 +173,7 @@ extern void heap_abort_speculative(Relation relation, HeapTuple tuple); extern HTSU_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, - HeapUpdateFailureData *hufd, LockTupleMode *lockmode); + HeapUpdateFailureData *hufd); extern HTSU_Result heap_lock_tuple(Relation relation, HeapTuple tuple, CommandId cid, LockTupleMode mode, LockWaitPolicy wait_policy, bool follow_update, diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out index 90f3177743..7546ed84f1 100644 --- a/src/test/regress/expected/merge.out +++ b/src/test/regress/expected/merge.out @@ -84,6 +84,24 @@ WHEN NOT MATCHED THEN ERROR: syntax error at or near "INTO" LINE 5: INSERT INTO target DEFAULT VALUES ^ +-- Multiple VALUES clause +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (1,1), (2,2); +ERROR: Multiple VALUES clauses not allowed in MERGE INSERT statement +; +-- SELECT query for INSERT +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT SELECT (1, 1); +ERROR: syntax error at or near "SELECT" +LINE 5: INSERT SELECT (1, 1); + ^ +; -- NOT MATCHED/UPDATE MERGE INTO target t USING source AS s @@ -104,6 +122,48 @@ WHEN MATCHED THEN ERROR: syntax error at or near "target" LINE 5: UPDATE target SET balance = 0 ^ +-- unsupported relation types +-- view +CREATE VIEW tv AS SELECT * FROM target; +MERGE INTO tv t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; +ERROR: MERGE is not supported for this relation type +DROP VIEW tv; +-- materialized view +CREATE MATERIALIZED VIEW mv AS SELECT * FROM target; +MERGE INTO mv t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; +ERROR: MERGE is not supported for this relation type +DROP MATERIALIZED VIEW mv; +-- inherited table +CREATE TABLE inhp (tid int, balance int); +CREATE TABLE child1() INHERITS (inhp); +CREATE TABLE child2() INHERITS (child1); +MERGE INTO inhp t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; +ERROR: MERGE is not supported for relations with inheritance +MERGE INTO child1 t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; +ERROR: MERGE is not supported for relations with inheritance +-- this should be ok +MERGE INTO child2 t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; +DROP TABLE inhp, child1, child2; -- permissions MERGE INTO target USING source2 @@ -382,7 +442,7 @@ WHEN MATCHED THEN UPDATE SET balance = 0 ; ERROR: MERGE command cannot affect row a second time -HINT: Ensure that not more than one source rows match any one target row +HINT: Ensure that not more than one source row match any one target row ROLLBACK; BEGIN; MERGE INTO target t @@ -392,7 +452,7 @@ WHEN MATCHED THEN DELETE ; ERROR: MERGE command cannot affect row a second time -HINT: Ensure that not more than one source rows match any one target row +HINT: Ensure that not more than one source row match any one target row ROLLBACK; -- correct source data DELETE FROM source WHERE sid = 2; diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql index cd6144bb5f..db2f0acb86 100644 --- a/src/test/regress/sql/merge.sql +++ b/src/test/regress/sql/merge.sql @@ -63,6 +63,20 @@ ON t.tid = s.sid WHEN NOT MATCHED THEN INSERT INTO target DEFAULT VALUES ; +-- Multiple VALUES clause +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (1,1), (2,2); +; +-- SELECT query for INSERT +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT SELECT (1, 1); +; -- NOT MATCHED/UPDATE MERGE INTO target t USING source AS s @@ -78,6 +92,50 @@ WHEN MATCHED THEN UPDATE target SET balance = 0 ; +-- unsupported relation types +-- view +CREATE VIEW tv AS SELECT * FROM target; +MERGE INTO tv t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; +DROP VIEW tv; + +-- materialized view +CREATE MATERIALIZED VIEW mv AS SELECT * FROM target; +MERGE INTO mv t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; +DROP MATERIALIZED VIEW mv; + +-- inherited table +CREATE TABLE inhp (tid int, balance int); +CREATE TABLE child1() INHERITS (inhp); +CREATE TABLE child2() INHERITS (child1); + +MERGE INTO inhp t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; + +MERGE INTO child1 t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; + +-- this should be ok +MERGE INTO child2 t +USING source s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES; +DROP TABLE inhp, child1, child2; + -- permissions MERGE INTO target -- 2.14.3 (Apple Git-98) [application/octet-stream] v27-0004-Basic-tab-completion-for-MERGE.patch (6.0K, ../../CABOikdMO4G3dBL9HSzP9DBT051uGnd0uejnYAEZk+iYn3AWY2g@mail.gmail.com/4-v27-0004-Basic-tab-completion-for-MERGE.patch) download | inline diff: From 99f89dd4ba38d6717631036df58f2ae16786a849 Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Tue, 27 Mar 2018 14:56:34 +0530 Subject: [PATCH v27 4/5] Basic tab-completion for MERGE --- src/bin/psql/tab-complete.c | 87 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 81 insertions(+), 6 deletions(-) diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 6926ca132e..db61fe11c4 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -659,6 +659,28 @@ static const SchemaQuery Query_for_list_of_updatables = { NULL }; +/* Relations supporting MERGE */ +static const SchemaQuery Query_for_list_of_mergetargets = { + /* min_server_version */ + 110000, + /* catname */ + "pg_catalog.pg_class c", + /* selcondition */ + "c.relkind IN (" CppAsString2(RELKIND_RELATION) ", " + CppAsString2(RELKIND_PARTITIONED_TABLE) ") AND " + "c.relhasrules = false AND " + "(c.relhassubclass = false OR " + " c.relkind = " CppAsString2(RELKIND_PARTITIONED_TABLE) ")", + /* viscondition */ + "pg_catalog.pg_table_is_visible(c.oid)", + /* namespace */ + "c.relnamespace", + /* result */ + "pg_catalog.quote_ident(c.relname)", + /* qualresult */ + NULL +}; + static const SchemaQuery Query_for_list_of_relations = { /* min_server_version */ 0, @@ -1605,7 +1627,7 @@ psql_completion(const char *text, int start, int end) "COMMENT", "COMMIT", "COPY", "CREATE", "DEALLOCATE", "DECLARE", "DELETE FROM", "DISCARD", "DO", "DROP", "END", "EXECUTE", "EXPLAIN", "FETCH", "GRANT", "IMPORT", "INSERT", "LISTEN", "LOAD", "LOCK", - "MOVE", "NOTIFY", "PREPARE", + "MERGE", "MOVE", "NOTIFY", "PREPARE", "REASSIGN", "REFRESH MATERIALIZED VIEW", "REINDEX", "RELEASE", "RESET", "REVOKE", "ROLLBACK", "SAVEPOINT", "SECURITY LABEL", "SELECT", "SET", "SHOW", "START", @@ -2999,14 +3021,15 @@ psql_completion(const char *text, int start, int end) * Complete EXPLAIN [ANALYZE] [VERBOSE] with list of EXPLAIN-able commands */ else if (Matches1("EXPLAIN")) - COMPLETE_WITH_LIST7("SELECT", "INSERT", "DELETE", "UPDATE", "DECLARE", - "ANALYZE", "VERBOSE"); + COMPLETE_WITH_LIST8("SELECT", "INSERT", "DELETE", "UPDATE", "MERGE", + "DECLARE", "ANALYZE", "VERBOSE"); else if (Matches2("EXPLAIN", "ANALYZE")) - COMPLETE_WITH_LIST6("SELECT", "INSERT", "DELETE", "UPDATE", "DECLARE", - "VERBOSE"); + COMPLETE_WITH_LIST7("SELECT", "INSERT", "DELETE", "UPDATE", "MERGE", + "DECLARE", "VERBOSE"); else if (Matches2("EXPLAIN", "VERBOSE") || Matches3("EXPLAIN", "ANALYZE", "VERBOSE")) - COMPLETE_WITH_LIST5("SELECT", "INSERT", "DELETE", "UPDATE", "DECLARE"); + COMPLETE_WITH_LIST6("SELECT", "INSERT", "DELETE", "UPDATE", "MERGE", + "DECLARE"); /* FETCH && MOVE */ /* Complete FETCH with one of FORWARD, BACKWARD, RELATIVE */ @@ -3229,6 +3252,9 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH_CONST("SCHEMA"); /* INSERT --- can be inside EXPLAIN, RULE, etc */ + /* Complete NOT MATCHED THEN INSERT */ + else if (TailMatches4("NOT", "MATCHED", "THEN", "INSERT")) + COMPLETE_WITH_LIST2("VALUES", "("); /* Complete INSERT with "INTO" */ else if (TailMatches1("INSERT")) COMPLETE_WITH_CONST("INTO"); @@ -3300,6 +3326,55 @@ psql_completion(const char *text, int start, int end) Matches5("LOCK", "TABLE", MatchAny, "IN", "SHARE")) COMPLETE_WITH_LIST3("MODE", "ROW EXCLUSIVE MODE", "UPDATE EXCLUSIVE MODE"); +/* MERGE --- can be inside EXPLAIN */ + else if (TailMatches1("MERGE")) + COMPLETE_WITH_CONST("INTO"); + else if (TailMatches2("MERGE", "INTO")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_mergetargets, NULL); + else if (TailMatches3("MERGE", "INTO", MatchAny)) + COMPLETE_WITH_LIST2("USING", "AS"); + else if (TailMatches4("MERGE", "INTO", MatchAny, "USING")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL); + /* with [AS] alias */ + else if (TailMatches5("MERGE", "INTO", MatchAny, "AS", MatchAny)) + COMPLETE_WITH_CONST("USING"); + else if (TailMatches4("MERGE", "INTO", MatchAny, MatchAny)) + COMPLETE_WITH_CONST("USING"); + else if (TailMatches6("MERGE", "INTO", MatchAny, "AS", MatchAny, "USING")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL); + else if (TailMatches5("MERGE", "INTO", MatchAny, MatchAny, "USING")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL); + /* ON */ + else if (TailMatches5("MERGE", "INTO", MatchAny, "USING", MatchAny)) + COMPLETE_WITH_CONST("ON"); + else if (TailMatches8("INTO", MatchAny, "AS", MatchAny, "USING", MatchAny, "AS", MatchAny)) + COMPLETE_WITH_CONST("ON"); + else if (TailMatches6("INTO", MatchAny, MatchAny, "USING", MatchAny, MatchAny)) + COMPLETE_WITH_CONST("ON"); + /* ON condition */ + else if (TailMatches5("INTO", MatchAny, "USING", MatchAny, "ON")) + COMPLETE_WITH_ATTR(prev4_wd, ""); + else if (TailMatches9("INTO", MatchAny, "AS", MatchAny, "USING", MatchAny, "AS", MatchAny, "ON")) + COMPLETE_WITH_ATTR(prev8_wd, ""); + else if (TailMatches7("INTO", MatchAny, MatchAny, "USING", MatchAny, MatchAny, "ON")) + COMPLETE_WITH_ATTR(prev6_wd, ""); + /* WHEN [NOT] MATCHED */ + else if (TailMatches4("USING", MatchAny, "ON", MatchAny)) + COMPLETE_WITH_LIST2("WHEN MATCHED", "WHEN NOT MATCHED"); + else if (TailMatches6("USING", MatchAny, "AS", MatchAny, "ON", MatchAny)) + COMPLETE_WITH_LIST2("WHEN MATCHED", "WHEN NOT MATCHED"); + else if (TailMatches5("USING", MatchAny, MatchAny, "ON", MatchAny)) + COMPLETE_WITH_LIST2("WHEN MATCHED", "WHEN NOT MATCHED"); + else if (TailMatches2("WHEN", "MATCHED")) + COMPLETE_WITH_LIST2("THEN", "AND"); + else if (TailMatches3("WHEN", "NOT", "MATCHED")) + COMPLETE_WITH_LIST2("THEN", "AND"); + else if (TailMatches3("WHEN", "MATCHED", "THEN")) + COMPLETE_WITH_LIST2("UPDATE", "DELETE"); + else if (TailMatches4("WHEN", "NOT", "MATCHED", "THEN")) + COMPLETE_WITH_LIST2("INSERT", "DO"); + else if (TailMatches5("WHEN", "NOT", "MATCHED", "THEN", "DO")) + COMPLETE_WITH_CONST("NOTHING"); /* NOTIFY --- can be inside EXPLAIN, RULE, etc */ else if (TailMatches1("NOTIFY")) -- 2.14.3 (Apple Git-98) [application/octet-stream] v27-0003-Fix-EXPLAIN-ANALYZE-output-to-report-counts-corr.patch (27.4K, ../../CABOikdMO4G3dBL9HSzP9DBT051uGnd0uejnYAEZk+iYn3AWY2g@mail.gmail.com/5-v27-0003-Fix-EXPLAIN-ANALYZE-output-to-report-counts-corr.patch) download | inline diff: From 77c426152e1fbcaf4bff2bd7a32765a3b568cbb4 Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Tue, 27 Mar 2018 09:20:33 +0530 Subject: [PATCH v27 3/5] Fix EXPLAIN ANALYZE output to report counts correctly. Fix some typos in passing Address Simon's review comments Run ExecCheckPlanOutput() for MERGE action's targetlists Fetch tableoid only for partitioned tables Some other misc fixes --- doc/src/sgml/ref/create_policy.sgml | 7 ++ doc/src/sgml/ref/merge.sgml | 7 ++ doc/src/sgml/trigger.sgml | 6 +- src/backend/commands/explain.c | 9 ++- src/backend/commands/trigger.c | 2 +- src/backend/executor/nodeMerge.c | 29 ++++--- src/backend/executor/nodeModifyTable.c | 11 +-- src/backend/optimizer/util/relnode.c | 1 + src/backend/parser/parse_clause.c | 6 ++ src/backend/parser/parse_merge.c | 23 ++---- src/backend/rewrite/rewriteHandler.c | 32 ++++---- src/include/executor/instrument.h | 7 +- src/include/nodes/execnodes.h | 11 ++- src/test/regress/expected/merge.out | 138 ++++++++++++++++++++++++++++++++- src/test/regress/expected/with.out | 16 ++-- src/test/regress/sql/merge.sql | 44 +++++++++++ 16 files changed, 277 insertions(+), 72 deletions(-) diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index 0e35b0ef43..32f39a48ba 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -94,6 +94,13 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable exist, a <quote>default deny</quote> policy is assumed, so that no rows will be visible or updatable. </para> + + <para> + No separate policy exists for <command>MERGE</command>. Instead policies + defined for <literal>SELECT</literal>, <literal>INSERT</literal>, + <literal>UPDATE</literal> and <literal>DELETE</literal> are applied + while executing MERGE, depending on the actions that are activated. + </para> </refsect1> <refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index 405a4cee29..539e512ced 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -115,6 +115,13 @@ DELETE of the <replaceable class="parameter">target_table_name</replaceable> referred to in a <literal>condition</literal>. </para> + + <para> + MERGE is not supported if the <replaceable + class="parameter">target_table_name</replaceable> has + <literal>RULES</literal> defined on it. + See <xref linkend="rules"/> for more information about <literal>RULES</literal>. + </para> </refsect1> <refsect1> diff --git a/doc/src/sgml/trigger.sgml b/doc/src/sgml/trigger.sgml index ac662bc64d..cce58fbf1d 100644 --- a/doc/src/sgml/trigger.sgml +++ b/doc/src/sgml/trigger.sgml @@ -186,15 +186,15 @@ No separate triggers are defined for <command>MERGE</command>. Instead, statement-level or row-level <command>UPDATE</command>, <command>DELETE</command> and <command>INSERT</command> triggers are fired - depedning on what actions are specified in the <command>MERGE</command> query + depending on what actions are specified in the <command>MERGE</command> query and what actions are activated. </para> <para> While running a <command>MERGE</command> command, statement-level <literal>BEFORE</literal> and <literal>AFTER</literal> triggers are fired for - specific actions specified in the <command>MERGE</command>, irrespective of - whether the action is finally activated or not. This is same as + events specified in the actions of the <command>MERGE</command> command, + irrespective of whether the action is finally activated or not. This is same as an <command>UPDATE</command> statement that updates no rows, yet statement-level triggers are fired. The row-level triggers are fired only when a row is actually updated, inserted or deleted. So it's perfectly legal diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index dc2f727d21..ca486872ac 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3086,18 +3086,21 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, double insert_path; double update_path; double delete_path; + double skipped_path; InstrEndLoop(mtstate->mt_plans[0]->instrument); /* count the number of source rows */ total = mtstate->mt_plans[0]->instrument->ntuples; - update_path = mtstate->ps.instrument->nfiltered1; - delete_path = mtstate->ps.instrument->nfiltered2; - insert_path = total - update_path - delete_path; + insert_path = mtstate->ps.instrument->nfiltered1; + update_path = mtstate->ps.instrument->nfiltered2; + delete_path = mtstate->ps.instrument->nfiltered3; + skipped_path = total - insert_path - update_path - delete_path; ExplainPropertyFloat("Tuples Inserted", NULL, insert_path, 0, es); ExplainPropertyFloat("Tuples Updated", NULL, update_path, 0, es); ExplainPropertyFloat("Tuples Deleted", NULL, delete_path, 0, es); + ExplainPropertyFloat("Tuples Skipped", NULL, skipped_path, 0, es); } } diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c index 331c37ad67..1617706376 100644 --- a/src/backend/commands/trigger.c +++ b/src/backend/commands/trigger.c @@ -3317,7 +3317,7 @@ ltrmark:; * If we're running MERGE then we must install the * new tuple in the slot of the underlying join query and * not the result relation itself. If the join does not - * yeild any tuple, the caller will take the necessary + * yield any tuple, the caller will take the necessary * action. */ epqslot = EvalPlanQual(estate, diff --git a/src/backend/executor/nodeMerge.c b/src/backend/executor/nodeMerge.c index e633af7704..ee41ed2eb2 100644 --- a/src/backend/executor/nodeMerge.c +++ b/src/backend/executor/nodeMerge.c @@ -59,8 +59,6 @@ ExecMergeMatched(ModifyTableState *mtstate, EState *estate, { ExprContext *econtext = mtstate->ps.ps_ExprContext; bool isNull; - Datum datum; - Oid tableoid = InvalidOid; List *mergeMatchedActionStates = NIL; HeapUpdateFailureData hufd; bool tuple_updated, @@ -73,22 +71,22 @@ ExecMergeMatched(ModifyTableState *mtstate, EState *estate, ListCell *l; TupleTableSlot *saved_slot = slot; - - /* - * We always fetch the tableoid while performing MATCHED MERGE action. - * This is strictly not required if the target table is not a partitioned - * table. But we are not yet optimising for that case. - */ - datum = ExecGetJunkAttribute(slot, junkfilter->jf_otherJunkAttNo, - &isNull); - Assert(!isNull); - tableoid = DatumGetObjectId(datum); - if (mtstate->mt_partition_tuple_routing) { + Datum datum; + Oid tableoid = InvalidOid; int leaf_part_index; PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing; + /* + * In case of partitioned table, we fetch the tableoid while performing + * MATCHED MERGE action. + */ + datum = ExecGetJunkAttribute(slot, junkfilter->jf_otherJunkAttNo, + &isNull); + Assert(!isNull); + tableoid = DatumGetObjectId(datum); + /* * If we're dealing with a MATCHED tuple, then tableoid must have been * set correctly. In case of partitioned table, we must now fetch the @@ -361,9 +359,9 @@ lmerge_matched:; } if (action->commandType == CMD_UPDATE && tuple_updated) - InstrCountFiltered1(&mtstate->ps, 1); - if (action->commandType == CMD_DELETE && tuple_deleted) InstrCountFiltered2(&mtstate->ps, 1); + if (action->commandType == CMD_DELETE && tuple_deleted) + InstrCountFiltered3(&mtstate->ps, 1); /* * We've activated one of the WHEN clauses, so we don't search @@ -466,6 +464,7 @@ ExecMergeNotMatched(ModifyTableState *mtstate, EState *estate, /* Revert ExecPrepareTupleRouting's state change. */ if (proute) estate->es_result_relation_info = resultRelInfo; + InstrCountFiltered1(&mtstate->ps, 1); break; case CMD_NOTHING: /* Do Nothing */ diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index def7eec223..08f812fc6d 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2733,9 +2733,13 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) switch (action->commandType) { case CMD_INSERT: + ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, + action->targetList); mtstate->mt_merge_subcommands |= MERGE_INSERT; break; case CMD_UPDATE: + ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, + action->targetList); mtstate->mt_merge_subcommands |= MERGE_UPDATE; break; case CMD_DELETE: @@ -2814,10 +2818,6 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) subplan = mtstate->mt_plans[i]->plan; - /* - * XXX we probably need to check plan output for CMD_MERGE - * also - */ if (operation == CMD_INSERT || operation == CMD_UPDATE) ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, subplan->targetlist); @@ -2842,7 +2842,8 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) if (!AttributeNumberIsValid(j->jf_junkAttNo)) elog(ERROR, "could not find junk ctid column"); - if (operation == CMD_MERGE) + if (operation == CMD_MERGE && + relkind == RELKIND_PARTITIONED_TABLE) { j->jf_otherJunkAttNo = ExecFindJunkAttribute(j, "tableoid"); if (!AttributeNumberIsValid(j->jf_otherJunkAttNo)) diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index 901cf24e20..da8f0f93fc 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -1237,6 +1237,7 @@ find_childrel_parents(PlannerInfo *root, RelOptInfo *rel) return result; } + /* * get_baserel_parampathinfo * Get the ParamPathInfo for a parameterized path for a base relation, diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 9df9828df8..857fe058c3 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -1094,6 +1094,12 @@ getRTEForSpecialRelationTypes(ParseState *pstate, RangeVar *rv) * * *top_rti: receives the rangetable index of top_rte. (Ditto.) * + * *right_rte: receives the RTE corresponding to the right side of the + * jointree. Only MERGE really needs to know about this and only MERGE passes a + * non-NULL pointer. + * + * *right_rti: receives the rangetable index of the right_rte. + * * *namespace: receives a List of ParseNamespaceItems for the RTEs exposed * as table/column names by this item. (The lateral_only flags in these items * are indeterminate and should be explicitly set by the caller before use.) diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c index 26a6692231..6df63439bb 100644 --- a/src/backend/parser/parse_merge.c +++ b/src/backend/parser/parse_merge.c @@ -279,10 +279,13 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) } /* - * Construct a query of the form SELECT relation.ctid --junk attribute - * ,relation.tableoid --junk attribute ,source_relation.<somecols> - * ,relation.<somecols> FROM relation RIGHT JOIN source_relation ON - * join_condition -- no WHERE clause - all conditions are applied in + * Construct a query of the form + * SELECT relation.ctid --junk attribute + * ,relation.tableoid --junk attribute + * ,source_relation.<somecols> + * ,relation.<somecols> + * FROM relation RIGHT JOIN source_relation + * ON join_condition; -- no WHERE clause - all conditions are applied in * executor * * stmt->relation is the target relation, given as a RangeVar @@ -353,18 +356,6 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) * If there are no INSERT actions we won't be using the non-matching * candidate rows for anything, so no need for an outer join. We do still * need an inner join for UPDATE and DELETE actions. - * - * Possible additional simplifications... - * - * XXX if we have a constant ON clause, we can skip join altogether - * - * XXX if we have a constant subquery, we can also skip join - * - * XXX if we were really keen we could look through the actionList and - * pull out common conditions, if there were no terminal clauses and put - * them into the main query as an early row filter but that seems like an - * atypical case and so checking for it would be likely to just be wasted - * effort. */ if (targetPerms & ACL_INSERT) joinexpr->jointype = JOIN_RIGHT; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index 8e5661df35..74b2829ffa 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1407,22 +1407,26 @@ rewriteTargetListMerge(Query *parsetree, Relation target_relation) parsetree->targetList = lappend(parsetree->targetList, tle); /* - * Emit TABLEOID so that executor can find the row to update or delete. + * If we are dealing with partitioned table, then emit TABLEOID so that + * executor can find the partition the row belongs to. */ - var = makeVar(parsetree->mergeTarget_relation, - TableOidAttributeNumber, - OIDOID, - -1, - InvalidOid, - 0); - - attrname = "tableoid"; - tle = makeTargetEntry((Expr *) var, - list_length(parsetree->targetList) + 1, - pstrdup(attrname), - true); + if (target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + var = makeVar(parsetree->mergeTarget_relation, + TableOidAttributeNumber, + OIDOID, + -1, + InvalidOid, + 0); + + attrname = "tableoid"; + tle = makeTargetEntry((Expr *) var, + list_length(parsetree->targetList) + 1, + pstrdup(attrname), + true); - parsetree->targetList = lappend(parsetree->targetList, tle); + parsetree->targetList = lappend(parsetree->targetList, tle); + } } /* diff --git a/src/include/executor/instrument.h b/src/include/executor/instrument.h index b72f91898a..28eb0093d4 100644 --- a/src/include/executor/instrument.h +++ b/src/include/executor/instrument.h @@ -58,8 +58,11 @@ typedef struct Instrumentation double total; /* Total total time (in seconds) */ double ntuples; /* Total tuples produced */ double nloops; /* # of run cycles for this node */ - double nfiltered1; /* # tuples removed by scanqual or joinqual */ - double nfiltered2; /* # tuples removed by "other" quals */ + double nfiltered1; /* # tuples removed by scanqual or joinqual OR + * # tuples inserted by MERGE */ + double nfiltered2; /* # tuples removed by "other" quals OR + * # tuples updated by MERGE */ + double nfiltered3; /* # tuples deleted by MERGE */ BufferUsage bufusage; /* Total buffer usage */ } Instrumentation; diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 0bf1d7aeb6..a839d53334 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -467,12 +467,12 @@ typedef struct ResultRelInfo MergeState *ri_mergeState; /* - * While executing MERGE, the target relation is processed twice; once to - * as a result relation and to run a join between the target and the + * While executing MERGE, the target relation is processed twice; once + * as a target relation and once to run a join between the target and the * source. We generate two different RTEs for these two purposes, one with * rte->inh set to false and other with rte->inh set to true. * - * Since the plan re-evaluated by EvalPlanQual uses the second RTE, we must + * Since the plan re-evaluated by EvalPlanQual uses the join RTE, we must * install the updated tuple in the scan corresponding to that RTE. The * following member tracks the index of the second RTE for EvalPlanQual * purposes. ri_mergeTargetRTI is non-zero only when MERGE is in-progress. @@ -1005,6 +1005,11 @@ typedef struct PlanState if (((PlanState *)(node))->instrument) \ ((PlanState *)(node))->instrument->nfiltered2 += (delta); \ } while(0) +#define InstrCountFiltered3(node, delta) \ + do { \ + if (((PlanState *)(node))->instrument) \ + ((PlanState *)(node))->instrument->nfiltered3 += (delta); \ + } while(0) /* * EPQState is state for executing an EvalPlanQual recheck on a candidate diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out index 411ccd0e66..90f3177743 100644 --- a/src/test/regress/expected/merge.out +++ b/src/test/regress/expected/merge.out @@ -822,6 +822,7 @@ NOTICE: AFTER INSERT STATEMENT trigger Tuples Inserted: 1 Tuples Updated: 1 Tuples Deleted: 1 + Tuples Skipped: 0 -> Hash Left Join (actual rows=3 loops=1) Hash Cond: (s.sid = t_1.tid) -> Seq Scan on source s (actual rows=3 loops=1) @@ -840,7 +841,7 @@ NOTICE: AFTER INSERT STATEMENT trigger Trigger merge_bsd: calls=1 Trigger merge_bsi: calls=1 Trigger merge_bsu: calls=1 -(22 rows) +(23 rows) SELECT * FROM target ORDER BY tid; tid | balance @@ -1085,6 +1086,7 @@ NOTICE: AFTER UPDATE STATEMENT trigger Tuples Inserted: 0 Tuples Updated: 1 Tuples Deleted: 0 + Tuples Skipped: 0 -> Seq Scan on target t_1 (actual rows=1 loops=1) Filter: (tid = 1) Rows Removed by Filter: 2 @@ -1092,7 +1094,7 @@ NOTICE: AFTER UPDATE STATEMENT trigger Trigger merge_asu: calls=1 Trigger merge_bru: calls=1 Trigger merge_bsu: calls=1 -(11 rows) +(12 rows) SELECT * FROM target ORDER BY tid; tid | balance @@ -1210,6 +1212,138 @@ ERROR: syntax error at or near "RETURNING" LINE 10: RETURNING * ^ ROLLBACK; +-- EXPLAIN +CREATE TABLE ex_mtarget (a int, b int); +CREATE TABLE ex_msource (a int, b int); +INSERT INTO ex_mtarget SELECT i, i*10 FROM generate_series(1,100,2) i; +INSERT INTO ex_msource SELECT i, i*10 FROM generate_series(1,100,1) i; +-- only updates +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b + 1; + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 50 + Tuples Deleted: 0 + Tuples Skipped: 0 + -> Merge Join (actual rows=50 loops=1) + Merge Cond: (t_1.a = s.a) + -> Sort (actual rows=50 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=50 loops=1) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) +(15 rows) + +-- only updates to selected tuples +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1; + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 5 + Tuples Deleted: 0 + Tuples Skipped: 45 + -> Merge Join (actual rows=50 loops=1) + Merge Cond: (t_1.a = s.a) + -> Sort (actual rows=50 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=50 loops=1) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) +(15 rows) + +-- updates + deletes +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1 +WHEN MATCHED AND t.a >= 10 AND t.a <= 20 THEN + DELETE; + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 5 + Tuples Deleted: 5 + Tuples Skipped: 40 + -> Merge Join (actual rows=50 loops=1) + Merge Cond: (t_1.a = s.a) + -> Sort (actual rows=50 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=50 loops=1) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) +(15 rows) + +-- only inserts +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN NOT MATCHED AND s.a < 10 THEN + INSERT VALUES (a, b); + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 4 + Tuples Updated: 0 + Tuples Deleted: 0 + Tuples Skipped: 96 + -> Merge Left Join (actual rows=100 loops=1) + Merge Cond: (s.a = t_1.a) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) + -> Sort (actual rows=45 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=45 loops=1) +(15 rows) + +-- all three +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1 +WHEN MATCHED AND t.a >= 30 AND t.a <= 40 THEN + DELETE +WHEN NOT MATCHED AND s.a < 20 THEN + INSERT VALUES (a, b); + QUERY PLAN +----------------------------------------------------------------------- + Merge on ex_mtarget t (actual rows=0 loops=1) + Tuples Inserted: 10 + Tuples Updated: 9 + Tuples Deleted: 5 + Tuples Skipped: 76 + -> Merge Left Join (actual rows=100 loops=1) + Merge Cond: (s.a = t_1.a) + -> Sort (actual rows=100 loops=1) + Sort Key: s.a + Sort Method: quicksort Memory: 33kB + -> Seq Scan on ex_msource s (actual rows=100 loops=1) + -> Sort (actual rows=49 loops=1) + Sort Key: t_1.a + Sort Method: quicksort Memory: 27kB + -> Seq Scan on ex_mtarget t_1 (actual rows=49 loops=1) +(15 rows) + +DROP TABLE ex_msource, ex_mtarget; -- Subqueries BEGIN; MERGE INTO sq_target t diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out index 543ca4f272..ba2c937bca 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -1932,10 +1932,10 @@ WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); -> Result Output: 1, 'cte_basic val'::text -> Hash Right Join - Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Output: o.k, o.v, o.*, m_1.ctid Hash Cond: (m_1.k = o.k) -> Seq Scan on public.m m_1 - Output: m_1.ctid, m_1.tableoid, m_1.k + Output: m_1.ctid, m_1.k -> Hash Output: o.k, o.v, o.* -> Subquery Scan on o @@ -1981,10 +1981,10 @@ WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); Output: (cte_init.b || ' merge update'::text) Filter: (cte_init.a = 1) -> Hash Right Join - Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Output: o.k, o.v, o.*, m_1.ctid Hash Cond: (m_1.k = o.k) -> Seq Scan on public.m m_1 - Output: m_1.ctid, m_1.tableoid, m_1.k + Output: m_1.ctid, m_1.k -> Hash Output: o.k, o.v, o.* -> Subquery Scan on o @@ -2011,8 +2011,8 @@ WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); - QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------ + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- Merge on public.m CTE merge_source_cte -> Result @@ -2025,10 +2025,10 @@ WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text -> CTE Scan on merge_source_cte merge_source_cte_2 Output: ((merge_source_cte_2.*)::text || ' merge insert'::text) -> Hash Right Join - Output: merge_source_cte.a, merge_source_cte.b, ROW(merge_source_cte.a, merge_source_cte.b), m_1.ctid, m_1.tableoid + Output: merge_source_cte.a, merge_source_cte.b, ROW(merge_source_cte.a, merge_source_cte.b), m_1.ctid Hash Cond: (m_1.k = merge_source_cte.a) -> Seq Scan on public.m m_1 - Output: m_1.ctid, m_1.tableoid, m_1.k + Output: m_1.ctid, m_1.k -> Hash Output: merge_source_cte.a, merge_source_cte.b -> CTE Scan on merge_source_cte diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql index 8b5244fc63..cd6144bb5f 100644 --- a/src/test/regress/sql/merge.sql +++ b/src/test/regress/sql/merge.sql @@ -790,6 +790,50 @@ RETURNING * ; ROLLBACK; +-- EXPLAIN +CREATE TABLE ex_mtarget (a int, b int); +CREATE TABLE ex_msource (a int, b int); +INSERT INTO ex_mtarget SELECT i, i*10 FROM generate_series(1,100,2) i; +INSERT INTO ex_msource SELECT i, i*10 FROM generate_series(1,100,1) i; + +-- only updates +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b + 1; + +-- only updates to selected tuples +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1; + +-- updates + deletes +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1 +WHEN MATCHED AND t.a >= 10 AND t.a <= 20 THEN + DELETE; + +-- only inserts +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN NOT MATCHED AND s.a < 10 THEN + INSERT VALUES (a, b); + +-- all three +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a +WHEN MATCHED AND t.a < 10 THEN + UPDATE SET b = t.b + 1 +WHEN MATCHED AND t.a >= 30 AND t.a <= 40 THEN + DELETE +WHEN NOT MATCHED AND s.a < 20 THEN + INSERT VALUES (a, b); + +DROP TABLE ex_msource, ex_mtarget; + -- Subqueries BEGIN; MERGE INTO sq_target t -- 2.14.3 (Apple Git-98) [application/octet-stream] v27-0002-Add-support-for-CTE.patch (13.5K, ../../CABOikdMO4G3dBL9HSzP9DBT051uGnd0uejnYAEZk+iYn3AWY2g@mail.gmail.com/6-v27-0002-Add-support-for-CTE.patch) download | inline diff: From 8a2bd24d5d1e75ef6a1e5da95c00deca73dcdfe4 Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Mon, 26 Mar 2018 18:56:50 +0530 Subject: [PATCH v27 2/5] Add support for CTE --- src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/parser/gram.y | 11 +-- src/backend/parser/parse_merge.c | 9 +++ src/include/nodes/parsenodes.h | 1 + src/test/regress/expected/merge.out | 3 - src/test/regress/expected/with.out | 132 ++++++++++++++++++++++++++++++++++++ src/test/regress/sql/with.sql | 51 ++++++++++++++ 9 files changed, 203 insertions(+), 8 deletions(-) diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 770ed3b1a8..c3efca3c45 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -3055,6 +3055,7 @@ _copyMergeStmt(const MergeStmt *from) COPY_NODE_FIELD(source_relation); COPY_NODE_FIELD(join_condition); COPY_NODE_FIELD(mergeActionList); + COPY_NODE_FIELD(withClause); return newnode; } diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 5a0151eece..45ceba2830 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -1051,6 +1051,7 @@ _equalMergeStmt(const MergeStmt *a, const MergeStmt *b) COMPARE_NODE_FIELD(source_relation); COMPARE_NODE_FIELD(join_condition); COMPARE_NODE_FIELD(mergeActionList); + COMPARE_NODE_FIELD(withClause); return true; } diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 68e2cec66e..7106765e2b 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3446,6 +3446,8 @@ raw_expression_tree_walker(Node *node, return true; if (walker(stmt->mergeActionList, context)) return true; + if (walker(stmt->withClause, context)) + return true; } break; case T_SelectStmt: diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index ebca5f3eb7..2f21571915 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -11105,17 +11105,18 @@ set_target_list: *****************************************************************************/ MergeStmt: - MERGE INTO relation_expr_opt_alias + opt_with_clause MERGE INTO relation_expr_opt_alias USING table_ref ON a_expr merge_when_list { MergeStmt *m = makeNode(MergeStmt); - m->relation = $3; - m->source_relation = $5; - m->join_condition = $7; - m->mergeActionList = $8; + m->withClause = $1; + m->relation = $4; + m->source_relation = $6; + m->join_condition = $8; + m->mergeActionList = $9; $$ = (Node *)m; } diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c index b6e0c46656..26a6692231 100644 --- a/src/backend/parser/parse_merge.c +++ b/src/backend/parser/parse_merge.c @@ -24,6 +24,7 @@ #include "parser/parsetree.h" #include "parser/parser.h" #include "parser/parse_clause.h" +#include "parser/parse_cte.h" #include "parser/parse_merge.h" #include "parser/parse_relation.h" #include "parser/parse_target.h" @@ -203,6 +204,14 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt) qry->commandType = CMD_MERGE; + /* process the WITH clause independently of all else */ + if (stmt->withClause) + { + qry->hasRecursive = stmt->withClause->recursive; + qry->cteList = transformWithClause(pstate, stmt->withClause); + qry->hasModifyingCTE = pstate->p_hasModifyingCTE; + } + /* * Check WHEN clauses for permissions and sanity */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 0c904f4d7f..36e6e2e976 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -1519,6 +1519,7 @@ typedef struct MergeStmt Node *source_relation; /* source relation */ Node *join_condition; /* join condition between source and target */ List *mergeActionList; /* list of MergeAction(s) */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeAction diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out index 05c4287078..411ccd0e66 100644 --- a/src/test/regress/expected/merge.out +++ b/src/test/regress/expected/merge.out @@ -1191,9 +1191,6 @@ WHEN NOT MATCHED THEN WHEN MATCHED AND tid < 2 THEN DELETE ; -ERROR: syntax error at or near "MERGE" -LINE 4: MERGE INTO sq_target t - ^ ROLLBACK; -- RETURNING BEGIN; diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out index 2a2085556b..543ca4f272 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -1904,6 +1904,138 @@ RETURNING k, v; (0 rows) DROP TABLE withz; +-- WITH referenced by MERGE statement +CREATE TABLE m AS SELECT i AS k, (i || ' v')::text v FROM generate_series(1, 16, 3) i; +ALTER TABLE m ADD UNIQUE (k); +-- Basic: +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 0; + k | v +---+---------------------- + 0 | merge source SubPlan +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + QUERY PLAN +------------------------------------------------------------------- + Merge on public.m + CTE cte_basic + -> Result + Output: 1, 'cte_basic val'::text + -> Hash Right Join + Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = o.k) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: o.k, o.v, o.* + -> Subquery Scan on o + Output: o.k, o.v, o.* + -> Result + Output: 0, 'merge source SubPlan'::text + SubPlan 2 + -> Limit + Output: ((cte_basic.b || ' merge update'::text)) + -> CTE Scan on cte_basic + Output: (cte_basic.b || ' merge update'::text) + Filter: (cte_basic.a = m.k) +(21 rows) + +-- InitPlan +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 1; + k | v +---+--------------------------- + 1 | cte_init val merge update +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + QUERY PLAN +-------------------------------------------------------------------- + Merge on public.m + CTE cte_init + -> Result + Output: 1, 'cte_init val'::text + InitPlan 2 (returns $1) + -> Limit + Output: ((cte_init.b || ' merge update'::text)) + -> CTE Scan on cte_init + Output: (cte_init.b || ' merge update'::text) + Filter: (cte_init.a = 1) + -> Hash Right Join + Output: o.k, o.v, o.*, m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = o.k) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: o.k, o.v, o.* + -> Subquery Scan on o + Output: o.k, o.v, o.* + -> Result + Output: 1, 'merge source InitPlan'::text +(21 rows) + +-- MERGE source comes from CTE: +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); +-- Examine +SELECT * FROM m where k = 15; + k | v +----+-------------------------------------------------------------- + 15 | merge_source_cte val(15,"merge_source_cte val") merge insert +(1 row) + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------- + Merge on public.m + CTE merge_source_cte + -> Result + Output: 15, 'merge_source_cte val'::text + InitPlan 2 (returns $1) + -> CTE Scan on merge_source_cte merge_source_cte_1 + Output: ((merge_source_cte_1.b || (merge_source_cte_1.*)::text) || ' merge update'::text) + Filter: (merge_source_cte_1.a = 15) + InitPlan 3 (returns $2) + -> CTE Scan on merge_source_cte merge_source_cte_2 + Output: ((merge_source_cte_2.*)::text || ' merge insert'::text) + -> Hash Right Join + Output: merge_source_cte.a, merge_source_cte.b, ROW(merge_source_cte.a, merge_source_cte.b), m_1.ctid, m_1.tableoid + Hash Cond: (m_1.k = merge_source_cte.a) + -> Seq Scan on public.m m_1 + Output: m_1.ctid, m_1.tableoid, m_1.k + -> Hash + Output: merge_source_cte.a, merge_source_cte.b + -> CTE Scan on merge_source_cte + Output: merge_source_cte.a, merge_source_cte.b +(20 rows) + +DROP TABLE m; -- check that run to completion happens in proper ordering TRUNCATE TABLE y; INSERT INTO y SELECT generate_series(1, 3); diff --git a/src/test/regress/sql/with.sql b/src/test/regress/sql/with.sql index f85645efde..dd73b334de 100644 --- a/src/test/regress/sql/with.sql +++ b/src/test/regress/sql/with.sql @@ -862,6 +862,57 @@ RETURNING k, v; DROP TABLE withz; +-- WITH referenced by MERGE statement +CREATE TABLE m AS SELECT i AS k, (i || ' v')::text v FROM generate_series(1, 16, 3) i; +ALTER TABLE m ADD UNIQUE (k); + +-- Basic: +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 0; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) +MERGE INTO m USING (select 0 k, 'merge source SubPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a = m.k LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + +-- InitPlan +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); +-- Examine +SELECT * FROM m where k = 1; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH cte_init AS (SELECT 1 a, 'cte_init val' b) +MERGE INTO m USING (select 1 k, 'merge source InitPlan' v) o ON m.k=o.k +WHEN MATCHED THEN UPDATE SET v = (SELECT b || ' merge update' FROM cte_init WHERE a = 1 LIMIT 1) +WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); + +-- MERGE source comes from CTE: +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); +-- Examine +SELECT * FROM m where k = 15; + +-- See EXPLAIN output for same query: +EXPLAIN (VERBOSE, COSTS OFF) +WITH merge_source_cte AS (SELECT 15 a, 'merge_source_cte val' b) +MERGE INTO m USING (select * from merge_source_cte) o ON m.k=o.a +WHEN MATCHED THEN UPDATE SET v = (SELECT b || merge_source_cte.*::text || ' merge update' FROM merge_source_cte WHERE a = 15) +WHEN NOT MATCHED THEN INSERT VALUES(o.a, o.b || (SELECT merge_source_cte.*::text || ' merge insert' FROM merge_source_cte)); + +DROP TABLE m; + -- check that run to completion happens in proper ordering TRUNCATE TABLE y; -- 2.14.3 (Apple Git-98) [application/octet-stream] v27-0001-Version-25c-of-MERGE-patch-based-on-ON-CONFLICT-.patch (345.8K, ../../CABOikdMO4G3dBL9HSzP9DBT051uGnd0uejnYAEZk+iYn3AWY2g@mail.gmail.com/7-v27-0001-Version-25c-of-MERGE-patch-based-on-ON-CONFLICT-.patch) download | inline diff: From fed8aeee3fbb3fb9f119752c9ee12d9a678f0bee Mon Sep 17 00:00:00 2001 From: Pavan Deolasee <[email protected]> Date: Sat, 24 Mar 2018 17:25:04 +0530 Subject: [PATCH v27 1/5] Version 25c of MERGE patch, based on ON CONFLICT DO UPDATE work Add a check for consistent lookup for result and mergeTarget relation, some comments and fixes Add tests to privileges.sql and ensure that we don't demand SELECT privileges on columns from source side Add additional test case for RLS --- contrib/test_decoding/expected/ddl.out | 46 + contrib/test_decoding/sql/ddl.sql | 16 + doc/src/sgml/libpq.sgml | 8 +- doc/src/sgml/mvcc.sgml | 28 +- doc/src/sgml/plpgsql.sgml | 3 +- doc/src/sgml/ref/allfiles.sgml | 1 + doc/src/sgml/ref/insert.sgml | 11 +- doc/src/sgml/ref/merge.sgml | 599 ++++++++ doc/src/sgml/reference.sgml | 1 + doc/src/sgml/trigger.sgml | 20 + src/backend/access/heap/heapam.c | 3 + src/backend/catalog/sql_features.txt | 6 +- src/backend/commands/explain.c | 30 + src/backend/commands/prepare.c | 1 + src/backend/commands/trigger.c | 156 +- src/backend/executor/Makefile | 2 +- src/backend/executor/README | 10 + src/backend/executor/execMain.c | 17 + src/backend/executor/execPartition.c | 116 ++ src/backend/executor/execReplication.c | 4 +- src/backend/executor/nodeMerge.c | 566 +++++++ src/backend/executor/nodeModifyTable.c | 378 ++++- src/backend/executor/spi.c | 3 + src/backend/nodes/copyfuncs.c | 40 + src/backend/nodes/equalfuncs.c | 32 + src/backend/nodes/nodeFuncs.c | 48 +- src/backend/nodes/outfuncs.c | 25 + src/backend/nodes/readfuncs.c | 6 + src/backend/optimizer/plan/createplan.c | 22 +- src/backend/optimizer/plan/planner.c | 29 +- src/backend/optimizer/plan/setrefs.c | 59 + src/backend/optimizer/prep/preptlist.c | 40 +- src/backend/optimizer/util/pathnode.c | 11 +- src/backend/optimizer/util/plancat.c | 4 + src/backend/optimizer/util/relnode.c | 1 - src/backend/parser/Makefile | 2 +- src/backend/parser/analyze.c | 18 +- src/backend/parser/gram.y | 158 +- src/backend/parser/parse_agg.c | 10 + src/backend/parser/parse_clause.c | 39 +- src/backend/parser/parse_collate.c | 1 + src/backend/parser/parse_expr.c | 3 + src/backend/parser/parse_func.c | 3 + src/backend/parser/parse_merge.c | 670 ++++++++ src/backend/parser/parse_relation.c | 10 + src/backend/rewrite/rewriteHandler.c | 112 +- src/backend/rewrite/rowsecurity.c | 97 ++ src/backend/tcop/pquery.c | 5 + src/backend/tcop/utility.c | 16 + src/include/access/heapam.h | 1 + src/include/commands/trigger.h | 6 +- src/include/executor/execPartition.h | 1 + src/include/executor/nodeMerge.h | 22 + src/include/executor/nodeModifyTable.h | 21 + src/include/executor/spi.h | 1 + src/include/nodes/execnodes.h | 64 +- src/include/nodes/nodes.h | 6 +- src/include/nodes/parsenodes.h | 39 +- src/include/nodes/plannodes.h | 8 +- src/include/nodes/relation.h | 7 +- src/include/optimizer/pathnode.h | 7 +- src/include/parser/analyze.h | 5 + src/include/parser/kwlist.h | 2 + src/include/parser/parse_clause.h | 5 +- src/include/parser/parse_merge.h | 19 + src/include/parser/parse_node.h | 6 +- src/include/rewrite/rewriteHandler.h | 1 + src/interfaces/libpq/fe-exec.c | 9 +- src/pl/plpgsql/src/pl_exec.c | 5 +- src/pl/plpgsql/src/pl_gram.y | 8 + src/pl/plpgsql/src/pl_scanner.c | 1 + src/pl/plpgsql/src/plpgsql.h | 4 +- src/test/isolation/expected/merge-delete.out | 97 ++ .../isolation/expected/merge-insert-update.out | 84 + .../isolation/expected/merge-match-recheck.out | 106 ++ src/test/isolation/expected/merge-update.out | 213 +++ src/test/isolation/isolation_schedule | 4 + src/test/isolation/specs/merge-delete.spec | 51 + src/test/isolation/specs/merge-insert-update.spec | 52 + src/test/isolation/specs/merge-match-recheck.spec | 79 + src/test/isolation/specs/merge-update.spec | 132 ++ src/test/regress/expected/identity.out | 55 + src/test/regress/expected/merge.out | 1599 ++++++++++++++++++++ src/test/regress/expected/privileges.out | 98 ++ src/test/regress/expected/rowsecurity.out | 182 +++ src/test/regress/expected/rules.out | 31 + src/test/regress/expected/triggers.out | 48 + src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/identity.sql | 45 + src/test/regress/sql/merge.sql | 1068 +++++++++++++ src/test/regress/sql/privileges.sql | 108 ++ src/test/regress/sql/rowsecurity.sql | 156 ++ src/test/regress/sql/rules.sql | 33 + src/test/regress/sql/triggers.sql | 47 + src/tools/pgindent/typedefs.list | 3 + 96 files changed, 7879 insertions(+), 149 deletions(-) create mode 100644 doc/src/sgml/ref/merge.sgml create mode 100644 src/backend/executor/nodeMerge.c create mode 100644 src/backend/parser/parse_merge.c create mode 100644 src/include/executor/nodeMerge.h create mode 100644 src/include/parser/parse_merge.h create mode 100644 src/test/isolation/expected/merge-delete.out create mode 100644 src/test/isolation/expected/merge-insert-update.out create mode 100644 src/test/isolation/expected/merge-match-recheck.out create mode 100644 src/test/isolation/expected/merge-update.out create mode 100644 src/test/isolation/specs/merge-delete.spec create mode 100644 src/test/isolation/specs/merge-insert-update.spec create mode 100644 src/test/isolation/specs/merge-match-recheck.spec create mode 100644 src/test/isolation/specs/merge-update.spec create mode 100644 src/test/regress/expected/merge.out create mode 100644 src/test/regress/sql/merge.sql diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index b7c76469fc..79c359d6e3 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -192,6 +192,52 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (33 rows) +-- MERGE support +BEGIN; +MERGE INTO replication_example t + USING (SELECT i as id, i as data, i as num FROM generate_series(-20, 5) i) s + ON t.id = s.id + WHEN MATCHED AND t.id < 0 THEN + UPDATE SET somenum = somenum + 1 + WHEN MATCHED AND t.id >= 0 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.*); +COMMIT; +/* display results */ +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); + data +-------------------------------------------------------------------------------------------------------------------------------------------------- + BEGIN + table public.replication_example: INSERT: id[integer]:-20 somedata[integer]:-20 somenum[integer]:-20 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-19 somedata[integer]:-19 somenum[integer]:-19 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-18 somedata[integer]:-18 somenum[integer]:-18 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-17 somedata[integer]:-17 somenum[integer]:-17 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: INSERT: id[integer]:-16 somedata[integer]:-16 somenum[integer]:-16 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-15 somedata[integer]:-15 somenum[integer]:-14 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-14 somedata[integer]:-14 somenum[integer]:-13 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-13 somedata[integer]:-13 somenum[integer]:-12 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-12 somedata[integer]:-12 somenum[integer]:-11 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-11 somedata[integer]:-11 somenum[integer]:-10 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-10 somedata[integer]:-10 somenum[integer]:-9 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-9 somedata[integer]:-9 somenum[integer]:-8 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-8 somedata[integer]:-8 somenum[integer]:-7 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-7 somedata[integer]:-7 somenum[integer]:-6 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-6 somedata[integer]:-6 somenum[integer]:-5 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-5 somedata[integer]:-5 somenum[integer]:-4 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-4 somedata[integer]:-4 somenum[integer]:-3 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-3 somedata[integer]:-3 somenum[integer]:-2 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-2 somedata[integer]:-2 somenum[integer]:-1 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: UPDATE: id[integer]:-1 somedata[integer]:-1 somenum[integer]:0 zaphod1[integer]:null zaphod2[integer]:null + table public.replication_example: DELETE: id[integer]:0 + table public.replication_example: DELETE: id[integer]:1 + table public.replication_example: DELETE: id[integer]:2 + table public.replication_example: DELETE: id[integer]:3 + table public.replication_example: DELETE: id[integer]:4 + table public.replication_example: DELETE: id[integer]:5 + COMMIT +(28 rows) + CREATE TABLE tr_unique(id2 serial unique NOT NULL, data int); INSERT INTO tr_unique(data) VALUES(10); ALTER TABLE tr_unique RENAME TO tr_pkey; diff --git a/contrib/test_decoding/sql/ddl.sql b/contrib/test_decoding/sql/ddl.sql index c4b10a4cf9..0e608b252f 100644 --- a/contrib/test_decoding/sql/ddl.sql +++ b/contrib/test_decoding/sql/ddl.sql @@ -93,6 +93,22 @@ COMMIT; /* display results */ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- MERGE support +BEGIN; +MERGE INTO replication_example t + USING (SELECT i as id, i as data, i as num FROM generate_series(-20, 5) i) s + ON t.id = s.id + WHEN MATCHED AND t.id < 0 THEN + UPDATE SET somenum = somenum + 1 + WHEN MATCHED AND t.id >= 0 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.*); +COMMIT; + +/* display results */ +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); + CREATE TABLE tr_unique(id2 serial unique NOT NULL, data int); INSERT INTO tr_unique(data) VALUES(10); ALTER TABLE tr_unique RENAME TO tr_pkey; diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml index 943adfef77..8729ccd5c5 100644 --- a/doc/src/sgml/libpq.sgml +++ b/doc/src/sgml/libpq.sgml @@ -3917,9 +3917,11 @@ char *PQcmdTuples(PGresult *res); <structname>PGresult</structname>. This function can only be used following the execution of a <command>SELECT</command>, <command>CREATE TABLE AS</command>, <command>INSERT</command>, <command>UPDATE</command>, <command>DELETE</command>, - <command>MOVE</command>, <command>FETCH</command>, or <command>COPY</command> statement, - or an <command>EXECUTE</command> of a prepared query that contains an - <command>INSERT</command>, <command>UPDATE</command>, or <command>DELETE</command> statement. + <command>MERGE</command>, <command>MOVE</command>, <command>FETCH</command>, + or <command>COPY</command> statement, or an <command>EXECUTE</command> of a + prepared query that contains an <command>INSERT</command>, + <command>UPDATE</command>, <command>DELETE</command> + or <command>MERGE</command> statement. If the command that generated the <structname>PGresult</structname> was anything else, <function>PQcmdTuples</function> returns an empty string. The caller should not free the return value directly. It will be freed when diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 24613e3c75..0e3e89af56 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -422,6 +422,31 @@ COMMIT; <literal>11</literal>, which no longer matches the criteria. </para> + <para> + The <command>MERGE</command> allows the user to specify various combinations + of <command>INSERT</command>, <command>UPDATE</command> or + <command>DELETE</command> subcommands. A <command>MERGE</command> command + with both <command>INSERT</command> and <command>UPDATE</command> + subcommands looks similar to <command>INSERT</command> with an + <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee + that either <command>INSERT</command> and <command>UPDATE</command> will occur. + + If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + but the join condition still passes for the current target and the current + source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + and perform its action on the latest version of the row, using standard + EvalPlanQual. MERGE actions can be conditional, so conditions must be + re-evaluated on the latest row, starting from the first action. + + On the other hand, if the row is concurrently updated or deleted so that + the join condition fails, then MERGE will execute a NOT MATCHED action, if it + exists and the AND WHEN qual evaluates to true. + + If MERGE attempts an INSERT and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. MERGE + does not attempt to avoid the ERROR by attempting an UPDATE. + </para> + <para> Because Read Committed mode starts each command with a new snapshot that includes all transactions committed up to that instant, @@ -900,7 +925,8 @@ ERROR: could not serialize access due to read/write dependencies among transact <para> The commands <command>UPDATE</command>, - <command>DELETE</command>, and <command>INSERT</command> + <command>DELETE</command>, <command>INSERT</command> and + <command>MERGE</command> acquire this lock mode on the target table (in addition to <literal>ACCESS SHARE</literal> locks on any other referenced tables). In general, this lock mode will be acquired by any diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml index 7ed926fd51..67b22a0d04 100644 --- a/doc/src/sgml/plpgsql.sgml +++ b/doc/src/sgml/plpgsql.sgml @@ -1246,7 +1246,7 @@ EXECUTE format('SELECT count(*) FROM %I ' </programlisting> Another restriction on parameter symbols is that they only work in <command>SELECT</command>, <command>INSERT</command>, <command>UPDATE</command>, and - <command>DELETE</command> commands. In other statement + <command>DELETE</command> and <command>MERGE</command> commands. In other statement types (generically called utility statements), you must insert values textually even if they are just data values. </para> @@ -1529,6 +1529,7 @@ GET DIAGNOSTICS integer_var = ROW_COUNT; <listitem> <para> <command>UPDATE</command>, <command>INSERT</command>, and <command>DELETE</command> + and <command>MERGE</command> statements set <literal>FOUND</literal> true if at least one row is affected, false if no row is affected. </para> diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 22e6893211..4e01e5641c 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -159,6 +159,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY load SYSTEM "load.sgml"> <!ENTITY lock SYSTEM "lock.sgml"> <!ENTITY move SYSTEM "move.sgml"> +<!ENTITY merge SYSTEM "merge.sgml"> <!ENTITY notify SYSTEM "notify.sgml"> <!ENTITY prepare SYSTEM "prepare.sgml"> <!ENTITY prepareTransaction SYSTEM "prepare_transaction.sgml"> diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml index 62e142fd8e..da294aaa46 100644 --- a/doc/src/sgml/ref/insert.sgml +++ b/doc/src/sgml/ref/insert.sgml @@ -579,6 +579,13 @@ INSERT <replaceable>oid</replaceable> <replaceable class="parameter">count</repl is a partition, an error will occur if one of the input rows violates the partition constraint. </para> + + <para> + You may also wish to consider using <command>MERGE</command>, since that + allows mixed <command>INSERT</command>, <command>UPDATE</command> and + <command>DELETE</command> within a single statement. + See <xref linkend="sql-merge"/>. + </para> </refsect1> <refsect1> @@ -749,7 +756,9 @@ INSERT INTO distributors (did, dname) VALUES (10, 'Conrad International') Also, the case in which a column name list is omitted, but not all the columns are filled from the <literal>VALUES</literal> clause or <replaceable>query</replaceable>, - is disallowed by the standard. + is disallowed by the standard. If you prefer a more SQL Standard + conforming statement than <literal>ON CONFLICT</literal>, see + <xref linkend="sql-merge"/>. </para> <para> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml new file mode 100644 index 0000000000..405a4cee29 --- /dev/null +++ b/doc/src/sgml/ref/merge.sgml @@ -0,0 +1,599 @@ +<!-- +doc/src/sgml/ref/merge.sgml +PostgreSQL documentation +--> + +<refentry id="sql-merge"> + + <refmeta> + <refentrytitle>MERGE</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>MERGE</refname> + <refpurpose>insert, update, or delete rows of a table based upon source data</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +MERGE INTO <replaceable class="parameter">target_table_name</replaceable> [ [ AS ] <replaceable class="parameter">target_alias</replaceable> ] +USING <replaceable class="parameter">data_source</replaceable> +ON <replaceable class="parameter">join_condition</replaceable> +<replaceable class="parameter">when_clause</replaceable> [...] + +where <replaceable class="parameter">data_source</replaceable> is + +{ <replaceable class="parameter">source_table_name</replaceable> | + ( source_query ) +} +[ [ AS ] <replaceable class="parameter">source_alias</replaceable> ] + +and <replaceable class="parameter">when_clause</replaceable> is + +{ WHEN MATCHED [ AND <replaceable class="parameter">condition</replaceable> ] THEN { <replaceable class="parameter">merge_update</replaceable> | <replaceable class="parameter">merge_delete</replaceable> } | + WHEN NOT MATCHED [ AND <replaceable class="parameter">condition</replaceable> ] THEN { <replaceable class="parameter">merge_insert</replaceable> | DO NOTHING } +} + +and <replaceable class="parameter">merge_update</replaceable> is + +UPDATE SET { <replaceable class="parameter">column_name</replaceable> = { <replaceable class="parameter">expression</replaceable> | DEFAULT } | + ( <replaceable class="parameter">column_name</replaceable> [, ...] ) = ( { <replaceable class="parameter">expression</replaceable> | DEFAULT } [, ...] ) + } [, ...] + +and <replaceable class="parameter">merge_insert</replaceable> is + +INSERT [( <replaceable class="parameter">column_name</replaceable> [, ...] )] +[ OVERRIDING { SYSTEM | USER } VALUE ] +{ VALUES ( { <replaceable class="parameter">expression</replaceable> | DEFAULT } [, ...] ) | DEFAULT VALUES } + +and <replaceable class="parameter">merge_delete</replaceable> is + +DELETE +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + <command>MERGE</command> performs actions that modify rows in the + <replaceable class="parameter">target_table_name</replaceable>, + using the <replaceable class="parameter">data_source</replaceable>. + <command>MERGE</command> provides a single <acronym>SQL</acronym> + statement that can conditionally <command>INSERT</command>, + <command>UPDATE</command> or <command>DELETE</command> rows, a task + that would otherwise require multiple procedural language statements. + </para> + + <para> + First, the <command>MERGE</command> command performs a join + from <replaceable class="parameter">data_source</replaceable> to + <replaceable class="parameter">target_table_name</replaceable> + producing zero or more candidate change rows. For each candidate change + row the status of <literal>MATCHED</literal> or <literal>NOT MATCHED</literal> is set + just once, after which <literal>WHEN</literal> clauses are evaluated + in the order specified. If one of them is activated, the specified + action occurs. No more than one <literal>WHEN</literal> clause can be + activated for any candidate change row. + </para> + + <para> + <command>MERGE</command> actions have the same effect as + regular <command>UPDATE</command>, <command>INSERT</command>, or + <command>DELETE</command> commands of the same names. The syntax of + those commands is different, notably that there is no <literal>WHERE</literal> + clause and no tablename is specified. All actions refer to the + <replaceable class="parameter">target_table_name</replaceable>, + though modifications to other tables may be made using triggers. + </para> + + <para> + When <literal>DO NOTHING</literal> action is specified, the source row is + skipped. Since actions are evaluated in the given order, <literal>DO + NOTHING</literal> can be handy to skip non-interesting source rows before + more fine-grained handling. + </para> + + <para> + There is no MERGE privilege. + You must have the <literal>UPDATE</literal> privilege on the column(s) + of the <replaceable class="parameter">target_table_name</replaceable> + referred to in the <literal>SET</literal> clause + if you specify an update action, the <literal>INSERT</literal> privilege + on the <replaceable class="parameter">target_table_name</replaceable> + if you specify an insert action and/or the <literal>DELETE</literal> + privilege on the <replaceable class="parameter">target_table_name</replaceable> + if you specify a delete action on the + <replaceable class="parameter">target_table_name</replaceable>. + Privileges are tested once at statement start and are checked + whether or not particular <literal>WHEN</literal> clauses are activated + during the subsequent execution. + You will require the <literal>SELECT</literal> privilege on the + <replaceable class="parameter">data_source</replaceable> and any column(s) + of the <replaceable class="parameter">target_table_name</replaceable> + referred to in a <literal>condition</literal>. + </para> + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry> + <term><replaceable class="parameter">target_table_name</replaceable></term> + <listitem> + <para> + The name (optionally schema-qualified) of the target table or materialized + view to merge into. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">target_alias</replaceable></term> + <listitem> + <para> + A substitute name for the target table. When an alias is + provided, it completely hides the actual name of the table. For + example, given <literal>MERGE foo AS f</literal>, the remainder of the + <command>MERGE</command> statement must refer to this table as + <literal>f</literal> not <literal>foo</literal>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">source_table_name</replaceable></term> + <listitem> + <para> + The name (optionally schema-qualified) of the source table, view or + transition table. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">source_query</replaceable></term> + <listitem> + <para> + A query (<command>SELECT</command> statement or <command>VALUES</command> + statement) that supplies the rows to be merged into the + <replaceable class="parameter">target_table_name</replaceable>. + Refer to the <xref linkend="sql-select"/> + statement or <xref linkend="sql-values"/> + statement for a description of the syntax. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">source_alias</replaceable></term> + <listitem> + <para> + A substitute name for the data source. When an alias is + provided, it completely hides whether table or query was specified. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">join_condition</replaceable></term> + <listitem> + <para> + <replaceable class="parameter">join_condition</replaceable> is + an expression resulting in a value of type + <type>boolean</type> (similar to a <literal>WHERE</literal> + clause) that specifies which rows in the + <replaceable class="parameter">data_source</replaceable> + match rows in the + <replaceable class="parameter">target_table_name</replaceable>. + </para> + <warning> + <para> + Only columns from <replaceable class="parameter">target_table_name</replaceable> + that attempt to match <replaceable class="parameter">data_source</replaceable> + rows should appear in <replaceable class="parameter">join_condition</replaceable>. + <replaceable class="parameter">join_condition</replaceable> subexpressions that + only reference <replaceable class="parameter">target_table_name</replaceable> + columns can only affect which action is taken, often in surprising ways. + </para> + </warning> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">when_clause</replaceable></term> + <listitem> + <para> + At least one <literal>WHEN</literal> clause is required. + </para> + <para> + If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> + and the candidate change row matches a row in the + <replaceable class="parameter">target_table_name</replaceable> + the <literal>WHEN</literal> clause is activated if the + <replaceable class="parameter">condition</replaceable> is + absent or is present and evaluates to <literal>true</literal>. + If the <literal>WHEN</literal> clause specifies <literal>WHEN NOT MATCHED</literal> + and the candidate change row does not match a row in the + <replaceable class="parameter">target_table_name</replaceable> + the <literal>WHEN</literal> clause is activated if the + <replaceable class="parameter">condition</replaceable> is + absent or is present and evaluates to <literal>true</literal>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">condition</replaceable></term> + <listitem> + <para> + An expression that returns a value of type <type>boolean</type>. + If this expression returns <literal>true</literal> then the <literal>WHEN</literal> + clause will be activated and the corresponding action will occur for + that row. The expression may not contain functions that possibly performs + writes to the database. + </para> + <para> + A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns + in both the source and the target relation. A condition on a + <literal>WHEN NOT MATCHED</literal> clause can only refer to columns from + the source relation, since by definition there is no matching target row. + Only the system attributes from the target table are accessible. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">merge_insert</replaceable></term> + <listitem> + <para> + The specification of an <literal>INSERT</literal> action that inserts + one row into the target table. + The target column names can be listed in any order. If no list of + column names is given at all, the default is all the columns of the + table in their declared order. + </para> + <para> + Each column not present in the explicit or implicit column list will be + filled with a default value, either its declared default value + or null if there is none. + </para> + <para> + If the expression for any column is not of the correct data type, + automatic type conversion will be attempted. + </para> + <para> + If <replaceable class="parameter">target_table_name</replaceable> + is a partitioned table, each row is routed to the appropriate partition + and inserted into it. + If <replaceable class="parameter">target_table_name</replaceable> + is a partition, an error will occur if one of the input rows violates + the partition constraint. + </para> + <para> + Column names may not be specified more than once. + <command>INSERT</command> actions cannot contain sub-selects. + </para> + <para> + The <literal>VALUES</literal> clause can only refer to columns from + the source relation, since by definition there is no matching target row. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">merge_update</replaceable></term> + <listitem> + <para> + The specification of an <literal>UPDATE</literal> action that updates + the current row of the <replaceable + class="parameter">target_table_name</replaceable>. + Column names may not be specified more than once. + </para> + <para> + Do not include the table name, as you would normally do with an + <xref linkend="sql-update"/> command. + For example, <literal>UPDATE tab SET col = 1</literal> is invalid. Also, + do not include a <literal>WHERE</literal> clause, since only the current + row can be updated. For example, + <literal>UPDATE SET col = 1 WHERE key = 57</literal> is invalid. + <command>UPDATE</command> actions cannot contain sub-selects in the + <literal>SET</literal> clause. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">merge_delete</replaceable></term> + <listitem> + <para> + Specifies a <literal>DELETE</literal> action that deletes the current row + of the <replaceable class="parameter">target_table_name</replaceable>. + Do not include the tablename or any other clauses, as you would normally + do with an <xref linkend="sql-delete"/> command. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">column_name</replaceable></term> + <listitem> + <para> + The name of a column in the <replaceable + class="parameter">target_table_name</replaceable>. The column name + can be qualified with a subfield name or array subscript, if + needed. (Inserting into only some fields of a composite + column leaves the other fields null.) When referencing a + column, do not include the table's name in the specification + of a target column. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>OVERRIDING SYSTEM VALUE</literal></term> + <listitem> + <para> + Without this clause, it is an error to specify an explicit value + (other than <literal>DEFAULT</literal>) for an identity column defined + as <literal>GENERATED ALWAYS</literal>. This clause overrides that + restriction. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>OVERRIDING USER VALUE</literal></term> + <listitem> + <para> + If this clause is specified, then any values supplied for identity + columns defined as <literal>GENERATED BY DEFAULT</literal> are ignored + and the default sequence-generated values are applied. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>DEFAULT VALUES</literal></term> + <listitem> + <para> + All columns will be filled with their default values. + (An <literal>OVERRIDING</literal> clause is not permitted in this + form.) + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">expression</replaceable></term> + <listitem> + <para> + An expression to assign to the column. The expression can use the + old values of this and other columns in the table. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>DEFAULT</literal></term> + <listitem> + <para> + Set the column to its default value (which will be NULL if no + specific default expression has been assigned to it). + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title>Outputs</title> + + <para> + On successful completion, a <command>MERGE</command> command returns a command + tag of the form +<screen> +MERGE <replaceable class="parameter">total-count</replaceable> +</screen> + The <replaceable class="parameter">total-count</replaceable> is the total + number of rows changed (whether updated, inserted or deleted). + If <replaceable class="parameter">total-count</replaceable> is 0, no rows + were changed in any way. + </para> + + </refsect1> + + <refsect1> + <title>Execution</title> + + <para> + The following steps take place during the execution of + <command>MERGE</command>. + <orderedlist> + <listitem> + <para> + Perform any BEFORE STATEMENT triggers for all actions specified, whether or + not their <literal>WHEN</literal> clauses are activated during execution. + </para> + </listitem> + <listitem> + <para> + Perform a join from source to target table. + The resulting query will be optimized normally and will produce + a set of candidate change row. For each candidate change row + <orderedlist> + <listitem> + <para> + Evaluate whether each row is MATCHED or NOT MATCHED. + </para> + </listitem> + <listitem> + <para> + Test each WHEN condition in the order specified until one activates. + </para> + </listitem> + <listitem> + <para> + When activated, perform the following actions + <orderedlist> + <listitem> + <para> + Perform any BEFORE ROW triggers that fire for the action's event type. + </para> + </listitem> + <listitem> + <para> + Apply the action specified, invoking any check constraints on the + target table. + However, it will not invoke rules. + </para> + </listitem> + <listitem> + <para> + Perform any AFTER ROW triggers that fire for the action's event type. + </para> + </listitem> + </orderedlist> + </para> + </listitem> + </orderedlist> + </para> + </listitem> + <listitem> + <para> + Perform any AFTER STATEMENT triggers for actions specified, whether or + not they actually occur. This is similar to the behavior of an + <command>UPDATE</command> statement that modifies no rows. + </para> + </listitem> + </orderedlist> + In summary, statement triggers for an event type (say, INSERT) will + be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level + triggers will fire only for the one event type <emphasis>activated</emphasis>. + So a <command>MERGE</command> might fire statement triggers for both + <command>UPDATE</command> and <command>INSERT</command>, even though only + <command>UPDATE</command> row triggers were fired. + </para> + + <para> + You should ensure that the join produces at most one candidate change row + for each target row. In other words, a target row shouldn't join to more + than one data source row. If it does, then only one of the candidate change + rows will be used to modify the target row, later attempts to modify will + cause an error. This can also occur if row triggers make changes to the + target table which are then subsequently modified by <command>MERGE</command>. + If the repeated action is an <command>INSERT</command> this will + cause a uniqueness violation while a repeated <command>UPDATE</command> or + <command>DELETE</command> will cause a cardinality violation; the latter behavior + is required by the <acronym>SQL</acronym> Standard. This differs from + historical <productname>PostgreSQL</productname> behavior of joins in + <command>UPDATE</command> and <command>DELETE</command> statements where second and + subsequent attempts to modify are simply ignored. + </para> + + <para> + If a <literal>WHEN</literal> clause omits an <literal>AND</literal> clause it becomes + the final reachable clause of that kind (<literal>MATCHED</literal> or + <literal>NOT MATCHED</literal>). If a later <literal>WHEN</literal> clause of that kind + is specified it would be provably unreachable and an error is raised. + If a final reachable clause is omitted it is possible that no action + will be taken for a candidate change row. + </para> + + </refsect1> + <refsect1> + <title>Notes</title> + + <para> + The order in which rows are generated from the data source is indeterminate + by default. A <replaceable class="parameter">source_query</replaceable> + can be used to specify a consistent ordering, if required, which might be + needed to avoid deadlocks between concurrent transactions. + </para> + + <para> + There is no <literal>RETURNING</literal> clause with <command>MERGE</command>. + Actions of <command>INSERT</command>, <command>UPDATE</command> and <command>DELETE</command> + cannot contain <literal>RETURNING</literal> or <literal>WITH</literal> clauses. + </para> + + <tip> + <para> + You may also wish to consider using <command>INSERT ... ON CONFLICT</command> as an + alternative statement which offers the ability to run an <command>UPDATE</command> + if a concurrent <command>INSERT</command> occurs. There are a variety of + differences and restrictions between the two statement types and they are not + interchangeable. + </para> + </tip> + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + Perform maintenance on CustomerAccounts based upon new Transactions. + +<programlisting> +MERGE CustomerAccount CA +USING RecentTransactions T +ON T.CustomerId = CA.CustomerId +WHEN MATCHED THEN + UPDATE SET Balance = Balance + TransactionValue +WHEN NOT MATCHED THEN + INSERT (CustomerId, Balance) + VALUES (T.CustomerId, T.TransactionValue); +</programlisting> + + notice that this would be exactly equivalent to the following + statement because the <literal>MATCHED</literal> result does not change + during execution + +<programlisting> +MERGE CustomerAccount CA +USING (Select CustomerId, TransactionValue From RecentTransactions) AS T +ON CA.CustomerId = T.CustomerId +WHEN NOT MATCHED THEN + INSERT (CustomerId, Balance) + VALUES (T.CustomerId, T.TransactionValue) +WHEN MATCHED THEN + UPDATE SET Balance = Balance + TransactionValue; +</programlisting> + </para> + + <para> + Attempt to insert a new stock item along with the quantity of stock. If + the item already exists, instead update the stock count of the existing + item. Don't allow entries that have zero stock. +<programlisting> +MERGE INTO wines w +USING wine_stock_changes s +ON s.winename = w.winename +WHEN NOT MATCHED AND s.stock_delta > 0 THEN + INSERT VALUES(s.winename, s.stock_delta) +WHEN MATCHED AND w.stock + s.stock_delta > 0 THEN + UPDATE SET stock = w.stock + s.stock_delta; +WHEN MATCHED THEN + DELETE; +</programlisting> + + The wine_stock_changes table might be, for example, a temporary table + recently loaded into the database. + </para> + + </refsect1> + + <refsect1> + <title>Compatibility</title> + <para> + This command conforms to the <acronym>SQL</acronym> standard. + </para> + <para> + The DO NOTHING action is an extension to the <acronym>SQL</acronym> standard. + </para> + </refsect1> +</refentry> diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index d27fb414f7..ef2270c467 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -186,6 +186,7 @@ &listen; &load; &lock; + &merge; &move; ¬ify; &prepare; diff --git a/doc/src/sgml/trigger.sgml b/doc/src/sgml/trigger.sgml index c43dbc9786..ac662bc64d 100644 --- a/doc/src/sgml/trigger.sgml +++ b/doc/src/sgml/trigger.sgml @@ -182,6 +182,26 @@ will be fired. </para> + <para> + No separate triggers are defined for <command>MERGE</command>. Instead, + statement-level or row-level <command>UPDATE</command>, + <command>DELETE</command> and <command>INSERT</command> triggers are fired + depedning on what actions are specified in the <command>MERGE</command> query + and what actions are activated. + </para> + + <para> + While running a <command>MERGE</command> command, statement-level + <literal>BEFORE</literal> and <literal>AFTER</literal> triggers are fired for + specific actions specified in the <command>MERGE</command>, irrespective of + whether the action is finally activated or not. This is same as + an <command>UPDATE</command> statement that updates no rows, yet + statement-level triggers are fired. The row-level triggers are fired only + when a row is actually updated, inserted or deleted. So it's perfectly legal + that while statement-level triggers are fired for certain type of action, no + row-level triggers are fired for the same kind of action. + </para> + <para> Trigger functions invoked by per-statement triggers should always return <symbol>NULL</symbol>. Trigger functions invoked by per-row diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index d7279248e7..decc3d37c3 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -3245,6 +3245,7 @@ l1: result == HeapTupleUpdated || result == HeapTupleBeingUpdated); Assert(!(tp.t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = tp.t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(tp.t_data); if (result == HeapTupleSelfUpdated) @@ -3887,6 +3888,7 @@ l2: result == HeapTupleUpdated || result == HeapTupleBeingUpdated); Assert(!(oldtup.t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = oldtup.t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(oldtup.t_data); if (result == HeapTupleSelfUpdated) @@ -5177,6 +5179,7 @@ failed: Assert(result == HeapTupleSelfUpdated || result == HeapTupleUpdated || result == HeapTupleWouldBlock); Assert(!(tuple->t_data->t_infomask & HEAP_XMAX_INVALID)); + hufd->result = result; hufd->ctid = tuple->t_data->t_ctid; hufd->xmax = HeapTupleHeaderGetUpdateXid(tuple->t_data); if (result == HeapTupleSelfUpdated) diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt index 20d61f3780..9612c135da 100644 --- a/src/backend/catalog/sql_features.txt +++ b/src/backend/catalog/sql_features.txt @@ -229,9 +229,9 @@ F311 Schema definition statement 02 CREATE TABLE for persistent base tables YES F311 Schema definition statement 03 CREATE VIEW YES F311 Schema definition statement 04 CREATE VIEW: WITH CHECK OPTION YES F311 Schema definition statement 05 GRANT statement YES -F312 MERGE statement NO consider INSERT ... ON CONFLICT DO UPDATE -F313 Enhanced MERGE statement NO -F314 MERGE statement with DELETE branch NO +F312 MERGE statement YES consider INSERT ... ON CONFLICT DO UPDATE +F313 Enhanced MERGE statement YES +F314 MERGE statement with DELETE branch YES F321 User authorization YES F341 Usage tables NO no ROUTINE_*_USAGE tables F361 Subprogram support YES diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index c38d178cd9..dc2f727d21 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -887,6 +887,9 @@ ExplainNode(PlanState *planstate, List *ancestors, case CMD_DELETE: pname = operation = "Delete"; break; + case CMD_MERGE: + pname = operation = "Merge"; + break; default: pname = "???"; break; @@ -2948,6 +2951,10 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, operation = "Delete"; foperation = "Foreign Delete"; break; + case CMD_MERGE: + operation = "Merge"; + foperation = "Foreign Merge"; + break; default: operation = "???"; foperation = "Foreign ???"; @@ -3070,6 +3077,29 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, other_path, 0, es); } } + else if (node->operation == CMD_MERGE) + { + /* EXPLAIN ANALYZE display of actual outcome for each tuple proposed */ + if (es->analyze && mtstate->ps.instrument) + { + double total; + double insert_path; + double update_path; + double delete_path; + + InstrEndLoop(mtstate->mt_plans[0]->instrument); + + /* count the number of source rows */ + total = mtstate->mt_plans[0]->instrument->ntuples; + update_path = mtstate->ps.instrument->nfiltered1; + delete_path = mtstate->ps.instrument->nfiltered2; + insert_path = total - update_path - delete_path; + + ExplainPropertyFloat("Tuples Inserted", NULL, insert_path, 0, es); + ExplainPropertyFloat("Tuples Updated", NULL, update_path, 0, es); + ExplainPropertyFloat("Tuples Deleted", NULL, delete_path, 0, es); + } + } if (labeltargets) ExplainCloseGroup("Target Tables", "Target Tables", false, es); diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c index b945b1556a..c3610b1874 100644 --- a/src/backend/commands/prepare.c +++ b/src/backend/commands/prepare.c @@ -151,6 +151,7 @@ PrepareQuery(PrepareStmt *stmt, const char *queryString, case CMD_INSERT: case CMD_UPDATE: case CMD_DELETE: + case CMD_MERGE: /* OK */ break; default: diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c index 9d8df5986e..331c37ad67 100644 --- a/src/backend/commands/trigger.c +++ b/src/backend/commands/trigger.c @@ -85,7 +85,8 @@ static HeapTuple GetTupleForTrigger(EState *estate, ResultRelInfo *relinfo, ItemPointer tid, LockTupleMode lockmode, - TupleTableSlot **newSlot); + TupleTableSlot **newSlot, + HeapUpdateFailureData *hufdp); static bool TriggerEnabled(EState *estate, ResultRelInfo *relinfo, Trigger *trigger, TriggerEvent event, Bitmapset *modifiedCols, @@ -2729,7 +2730,8 @@ bool ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, - HeapTuple fdw_trigtuple) + HeapTuple fdw_trigtuple, + HeapUpdateFailureData *hufdp) { TriggerDesc *trigdesc = relinfo->ri_TrigDesc; bool result = true; @@ -2743,7 +2745,7 @@ ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, if (fdw_trigtuple == NULL) { trigtuple = GetTupleForTrigger(estate, epqstate, relinfo, tupleid, - LockTupleExclusive, &newSlot); + LockTupleExclusive, &newSlot, hufdp); if (trigtuple == NULL) return false; } @@ -2814,6 +2816,7 @@ ExecARDeleteTriggers(EState *estate, ResultRelInfo *relinfo, relinfo, tupleid, LockTupleExclusive, + NULL, NULL); else trigtuple = fdw_trigtuple; @@ -2951,7 +2954,8 @@ ExecBRUpdateTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, HeapTuple fdw_trigtuple, - TupleTableSlot *slot) + TupleTableSlot *slot, + HeapUpdateFailureData *hufdp) { TriggerDesc *trigdesc = relinfo->ri_TrigDesc; HeapTuple slottuple = ExecMaterializeSlot(slot); @@ -2972,7 +2976,7 @@ ExecBRUpdateTriggers(EState *estate, EPQState *epqstate, { /* get a copy of the on-disk tuple we are planning to update */ trigtuple = GetTupleForTrigger(estate, epqstate, relinfo, tupleid, - lockmode, &newSlot); + lockmode, &newSlot, hufdp); if (trigtuple == NULL) return NULL; /* cancel the update action */ } @@ -3092,6 +3096,7 @@ ExecARUpdateTriggers(EState *estate, ResultRelInfo *relinfo, relinfo, tupleid, LockTupleExclusive, + NULL, NULL); else trigtuple = fdw_trigtuple; @@ -3240,7 +3245,8 @@ GetTupleForTrigger(EState *estate, ResultRelInfo *relinfo, ItemPointer tid, LockTupleMode lockmode, - TupleTableSlot **newSlot) + TupleTableSlot **newSlot, + HeapUpdateFailureData *hufdp) { Relation relation = relinfo->ri_RelationDesc; HeapTupleData tuple; @@ -3266,6 +3272,11 @@ ltrmark:; estate->es_output_cid, lockmode, LockWaitBlock, false, &buffer, &hufd); + + /* Let the caller know about failure reason, if any. */ + if (hufdp) + *hufdp = hufd; + switch (test) { case HeapTupleSelfUpdated: @@ -3302,10 +3313,17 @@ ltrmark:; /* it was updated, so look at the updated version */ TupleTableSlot *epqslot; + /* + * If we're running MERGE then we must install the + * new tuple in the slot of the underlying join query and + * not the result relation itself. If the join does not + * yeild any tuple, the caller will take the necessary + * action. + */ epqslot = EvalPlanQual(estate, epqstate, relation, - relinfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(relinfo), lockmode, &hufd.ctid, hufd.xmax); @@ -3828,8 +3846,14 @@ struct AfterTriggersTableData bool before_trig_done; /* did we already queue BS triggers? */ bool after_trig_done; /* did we already queue AS triggers? */ AfterTriggerEventList after_trig_events; /* if so, saved list pointer */ - Tuplestorestate *old_tuplestore; /* "old" transition table, if any */ - Tuplestorestate *new_tuplestore; /* "new" transition table, if any */ + /* "old" transition table for UPDATE, if any */ + Tuplestorestate *old_upd_tuplestore; + /* "new" transition table for UPDATE, if any */ + Tuplestorestate *new_upd_tuplestore; + /* "old" transition table for DELETE, if any */ + Tuplestorestate *old_del_tuplestore; + /* "new" transition table INSERT, if any */ + Tuplestorestate *new_ins_tuplestore; }; static AfterTriggersData afterTriggers; @@ -4296,13 +4320,19 @@ AfterTriggerExecute(AfterTriggerEvent event, { if (LocTriggerData.tg_trigger->tgoldtable) { - LocTriggerData.tg_oldtable = evtshared->ats_table->old_tuplestore; + if (TRIGGER_FIRED_BY_UPDATE(evtshared->ats_event)) + LocTriggerData.tg_oldtable = evtshared->ats_table->old_upd_tuplestore; + else + LocTriggerData.tg_oldtable = evtshared->ats_table->old_del_tuplestore; evtshared->ats_table->closed = true; } if (LocTriggerData.tg_trigger->tgnewtable) { - LocTriggerData.tg_newtable = evtshared->ats_table->new_tuplestore; + if (TRIGGER_FIRED_BY_INSERT(evtshared->ats_event)) + LocTriggerData.tg_newtable = evtshared->ats_table->new_ins_tuplestore; + else + LocTriggerData.tg_newtable = evtshared->ats_table->new_upd_tuplestore; evtshared->ats_table->closed = true; } } @@ -4637,8 +4667,10 @@ TransitionCaptureState * MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) { TransitionCaptureState *state; - bool need_old, - need_new; + bool need_old_upd, + need_new_upd, + need_old_del, + need_new_ins; AfterTriggersTableData *table; MemoryContext oldcxt; ResourceOwner saveResourceOwner; @@ -4650,23 +4682,31 @@ MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) switch (cmdType) { case CMD_INSERT: - need_old = false; - need_new = trigdesc->trig_insert_new_table; + need_old_upd = need_old_del = need_new_upd = false; + need_new_ins = trigdesc->trig_insert_new_table; break; case CMD_UPDATE: - need_old = trigdesc->trig_update_old_table; - need_new = trigdesc->trig_update_new_table; + need_old_upd = trigdesc->trig_update_old_table; + need_new_upd = trigdesc->trig_update_new_table; + need_old_del = need_new_ins = false; break; case CMD_DELETE: - need_old = trigdesc->trig_delete_old_table; - need_new = false; + need_old_del = trigdesc->trig_delete_old_table; + need_old_upd = need_new_upd = need_new_ins = false; + break; + case CMD_MERGE: + need_old_upd = trigdesc->trig_update_old_table; + need_new_upd = trigdesc->trig_update_new_table; + need_old_del = trigdesc->trig_delete_old_table; + need_new_ins = trigdesc->trig_insert_new_table; break; default: elog(ERROR, "unexpected CmdType: %d", (int) cmdType); - need_old = need_new = false; /* keep compiler quiet */ + /* keep compiler quiet */ + need_old_upd = need_new_upd = need_old_del = need_new_ins = false; break; } - if (!need_old && !need_new) + if (!need_old_upd && !need_new_upd && !need_new_ins && !need_old_del) return NULL; /* Check state, like AfterTriggerSaveEvent. */ @@ -4696,10 +4736,14 @@ MakeTransitionCaptureState(TriggerDesc *trigdesc, Oid relid, CmdType cmdType) saveResourceOwner = CurrentResourceOwner; CurrentResourceOwner = CurTransactionResourceOwner; - if (need_old && table->old_tuplestore == NULL) - table->old_tuplestore = tuplestore_begin_heap(false, false, work_mem); - if (need_new && table->new_tuplestore == NULL) - table->new_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_old_upd && table->old_upd_tuplestore == NULL) + table->old_upd_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_new_upd && table->new_upd_tuplestore == NULL) + table->new_upd_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_old_del && table->old_del_tuplestore == NULL) + table->old_del_tuplestore = tuplestore_begin_heap(false, false, work_mem); + if (need_new_ins && table->new_ins_tuplestore == NULL) + table->new_ins_tuplestore = tuplestore_begin_heap(false, false, work_mem); CurrentResourceOwner = saveResourceOwner; MemoryContextSwitchTo(oldcxt); @@ -4888,12 +4932,20 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs) { AfterTriggersTableData *table = (AfterTriggersTableData *) lfirst(lc); - ts = table->old_tuplestore; - table->old_tuplestore = NULL; + ts = table->old_upd_tuplestore; + table->old_upd_tuplestore = NULL; if (ts) tuplestore_end(ts); - ts = table->new_tuplestore; - table->new_tuplestore = NULL; + ts = table->new_upd_tuplestore; + table->new_upd_tuplestore = NULL; + if (ts) + tuplestore_end(ts); + ts = table->old_del_tuplestore; + table->old_del_tuplestore = NULL; + if (ts) + tuplestore_end(ts); + ts = table->new_ins_tuplestore; + table->new_ins_tuplestore = NULL; if (ts) tuplestore_end(ts); } @@ -5744,12 +5796,11 @@ AfterTriggerSaveEvent(EState *estate, ResultRelInfo *relinfo, newtup == NULL)); if (oldtup != NULL && - ((event == TRIGGER_EVENT_DELETE && delete_old_table) || - (event == TRIGGER_EVENT_UPDATE && update_old_table))) + (event == TRIGGER_EVENT_DELETE && delete_old_table)) { Tuplestorestate *old_tuplestore; - old_tuplestore = transition_capture->tcs_private->old_tuplestore; + old_tuplestore = transition_capture->tcs_private->old_del_tuplestore; if (map != NULL) { @@ -5761,13 +5812,48 @@ AfterTriggerSaveEvent(EState *estate, ResultRelInfo *relinfo, else tuplestore_puttuple(old_tuplestore, oldtup); } + if (oldtup != NULL && + (event == TRIGGER_EVENT_UPDATE && update_old_table)) + { + Tuplestorestate *old_tuplestore; + + old_tuplestore = transition_capture->tcs_private->old_upd_tuplestore; + + if (map != NULL) + { + HeapTuple converted = do_convert_tuple(oldtup, map); + + tuplestore_puttuple(old_tuplestore, converted); + pfree(converted); + } + else + tuplestore_puttuple(old_tuplestore, oldtup); + } + if (newtup != NULL && + (event == TRIGGER_EVENT_INSERT && insert_new_table)) + { + Tuplestorestate *new_tuplestore; + + new_tuplestore = transition_capture->tcs_private->new_ins_tuplestore; + + if (original_insert_tuple != NULL) + tuplestore_puttuple(new_tuplestore, original_insert_tuple); + else if (map != NULL) + { + HeapTuple converted = do_convert_tuple(newtup, map); + + tuplestore_puttuple(new_tuplestore, converted); + pfree(converted); + } + else + tuplestore_puttuple(new_tuplestore, newtup); + } if (newtup != NULL && - ((event == TRIGGER_EVENT_INSERT && insert_new_table) || - (event == TRIGGER_EVENT_UPDATE && update_new_table))) + (event == TRIGGER_EVENT_UPDATE && update_new_table)) { Tuplestorestate *new_tuplestore; - new_tuplestore = transition_capture->tcs_private->new_tuplestore; + new_tuplestore = transition_capture->tcs_private->new_upd_tuplestore; if (original_insert_tuple != NULL) tuplestore_puttuple(new_tuplestore, original_insert_tuple); diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefile index cc09895fa5..68675f9796 100644 --- a/src/backend/executor/Makefile +++ b/src/backend/executor/Makefile @@ -22,7 +22,7 @@ OBJS = execAmi.o execCurrent.o execExpr.o execExprInterp.o \ nodeCustom.o nodeFunctionscan.o nodeGather.o \ nodeHash.o nodeHashjoin.o nodeIndexscan.o nodeIndexonlyscan.o \ nodeLimit.o nodeLockRows.o nodeGatherMerge.o \ - nodeMaterial.o nodeMergeAppend.o nodeMergejoin.o nodeModifyTable.o \ + nodeMaterial.o nodeMergeAppend.o nodeMergejoin.o nodeMerge.o nodeModifyTable.o \ nodeNestloop.o nodeProjectSet.o nodeRecursiveunion.o nodeResult.o \ nodeSamplescan.o nodeSeqscan.o nodeSetOp.o nodeSort.o nodeUnique.o \ nodeValuesscan.o \ diff --git a/src/backend/executor/README b/src/backend/executor/README index 0d7cd552eb..05769772b7 100644 --- a/src/backend/executor/README +++ b/src/backend/executor/README @@ -37,6 +37,16 @@ the plan tree returns the computed tuples to be updated, plus a "junk" one. For DELETE, the plan tree need only deliver a CTID column, and the ModifyTable node visits each of those rows and marks the row deleted. +MERGE runs one generic plan that returns candidate target rows. Each row +consists of a super-row that contains all the columns needed by any of the +individual actions, plus a CTID and a TABLEOID junk columns. The CTID column is +required to know if a matching target row was found or not and the TABLEOID +column is needed to find the underlying target partition, in case when the +target table is a partition table. If the CTID column is set we attempt to +activate WHEN MATCHED actions, or if it is NULL then we will attempt to +activate WHEN NOT MATCHED actions. Once we know which action is activated we +form the final result row and apply only those changes. + XXX a great deal more documentation needs to be written here... diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 9a107aba56..e4d9b0b3f8 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -233,6 +233,7 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags) case CMD_INSERT: case CMD_DELETE: case CMD_UPDATE: + case CMD_MERGE: estate->es_output_cid = GetCurrentCommandId(true); break; @@ -1357,6 +1358,9 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_onConflictArbiterIndexes = NIL; resultRelInfo->ri_onConflict = NULL; + resultRelInfo->ri_mergeTargetRTI = 0; + resultRelInfo->ri_mergeState = (MergeState *) palloc0(sizeof (MergeState)); + /* * Partition constraint, which also includes the partition constraint of * all the ancestors that are partitions. Note that it will be checked @@ -2205,6 +2209,19 @@ ExecWithCheckOptions(WCOKind kind, ResultRelInfo *resultRelInfo, errmsg("new row violates row-level security policy for table \"%s\"", wco->relname))); break; + case WCO_RLS_MERGE_UPDATE_CHECK: + case WCO_RLS_MERGE_DELETE_CHECK: + if (wco->polname != NULL) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("target row violates row-level security policy \"%s\" (USING expression) for table \"%s\"", + wco->polname, wco->relname))); + else + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("target row violates row-level security policy (USING expression) for table \"%s\"", + wco->relname))); + break; case WCO_RLS_CONFLICT_CHECK: if (wco->polname != NULL) ereport(ERROR, diff --git a/src/backend/executor/execPartition.c b/src/backend/executor/execPartition.c index 9a13188649..a6a7885abd 100644 --- a/src/backend/executor/execPartition.c +++ b/src/backend/executor/execPartition.c @@ -67,6 +67,8 @@ ExecSetupPartitionTupleRouting(ModifyTableState *mtstate, Relation rel) ResultRelInfo *update_rri = NULL; int num_update_rri = 0, update_rri_index = 0; + bool is_update = false; + bool is_merge = false; PartitionTupleRouting *proute; int nparts; ModifyTable *node = mtstate ? (ModifyTable *) mtstate->ps.plan : NULL; @@ -89,13 +91,22 @@ ExecSetupPartitionTupleRouting(ModifyTableState *mtstate, Relation rel) /* Set up details specific to the type of tuple routing we are doing. */ if (node && node->operation == CMD_UPDATE) + is_update = true; + else if (node && node->operation == CMD_MERGE) + is_merge = true; + + if (is_update) { update_rri = mtstate->resultRelInfo; num_update_rri = list_length(node->plans); proute->subplan_partition_offsets = palloc(num_update_rri * sizeof(int)); proute->num_subplan_partition_offsets = num_update_rri; + } + + if (is_update || is_merge) + { /* * We need an additional tuple slot for storing transient tuples that * are converted to the root table descriptor. @@ -299,6 +310,25 @@ ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, return result; } +/* + * Given OID of the partition leaf, return the index of the leaf in the + * partition hierarchy. + */ +int +ExecFindPartitionByOid(PartitionTupleRouting *proute, Oid partoid) +{ + int i; + + for (i = 0; i < proute->num_partitions; i++) + { + if (proute->partition_oids[i] == partoid) + break; + } + + Assert(i < proute->num_partitions); + return i; +} + /* * ExecInitPartitionInfo * Initialize ResultRelInfo and other information for a partition if not @@ -337,6 +367,8 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, rootrel, estate->es_instrument); + leaf_part_rri->ri_PartitionLeafIndex = partidx; + /* * Verify result relation is a valid target for an INSERT. An UPDATE of a * partition-key becomes a DELETE+INSERT operation, so this check is still @@ -625,6 +657,90 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, Assert(proute->partitions[partidx] == NULL); proute->partitions[partidx] = leaf_part_rri; + /* + * Initialize information about this partition that's needed to handle + * MERGE. + */ + if (node && node->operation == CMD_MERGE) + { + TupleDesc partrelDesc = RelationGetDescr(partrel); + TupleConversionMap *map = proute->parent_child_tupconv_maps[partidx]; + int firstVarno = mtstate->resultRelInfo[0].ri_RangeTableIndex; + Relation firstResultRel = mtstate->resultRelInfo[0].ri_RelationDesc; + + /* + * If the root parent and partition have the same tuple + * descriptor, just reuse the original MERGE state for partition. + */ + if (map == NULL) + { + leaf_part_rri->ri_mergeState = resultRelInfo->ri_mergeState; + } + else + { + /* Convert expressions contain partition's attnos. */ + List *conv_tl, *conv_qual; + ListCell *l; + List *matchedActionStates = NIL; + List *notMatchedActionStates = NIL; + + foreach (l, node->mergeActionList) + { + MergeAction *action = lfirst_node(MergeAction, l); + MergeActionState *action_state = makeNode(MergeActionState); + TupleDesc tupDesc; + ExprContext *econtext; + + action_state->matched = action->matched; + action_state->commandType = action->commandType; + + conv_qual = (List *) action->qual; + conv_qual = map_partition_varattnos(conv_qual, + firstVarno, partrel, + firstResultRel, NULL); + + action_state->whenqual = ExecInitQual(conv_qual, &mtstate->ps); + + conv_tl = (List *) action->targetList; + conv_tl = map_partition_varattnos(conv_tl, + firstVarno, partrel, + firstResultRel, NULL); + + conv_tl = adjust_partition_tlist( conv_tl, map); + + tupDesc = ExecTypeFromTL(conv_tl, partrelDesc->tdhasoid); + action_state->tupDesc = tupDesc; + + /* build action projection state */ + econtext = mtstate->ps.ps_ExprContext; + action_state->proj = + ExecBuildProjectionInfo(conv_tl, econtext, + mtstate->mt_mergeproj, + &mtstate->ps, + partrelDesc); + + if (action_state->matched) + matchedActionStates = + lappend(matchedActionStates, action_state); + else + notMatchedActionStates = + lappend(notMatchedActionStates, action_state); + } + leaf_part_rri->ri_mergeState->matchedActionStates = + matchedActionStates; + leaf_part_rri->ri_mergeState->notMatchedActionStates = + notMatchedActionStates; + } + + /* + * get_partition_dispatch_recurse() and expand_partitioned_rtentry() + * fetch the leaf OIDs in the same order. So we can safely derive the + * index of the merge target relation corresponding to this partition + * by simply adding partidx + 1 to the root's merge target relation. + */ + leaf_part_rri->ri_mergeTargetRTI = node->mergeTargetRelation + + partidx + 1; + } MemoryContextSwitchTo(oldContext); return leaf_part_rri; diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 32891abbdf..971f92a938 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -454,7 +454,7 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate, { slot = ExecBRUpdateTriggers(estate, epqstate, resultRelInfo, &searchslot->tts_tuple->t_self, - NULL, slot); + NULL, slot, NULL); if (slot == NULL) /* "do nothing" */ skip_tuple = true; @@ -515,7 +515,7 @@ ExecSimpleRelationDelete(EState *estate, EPQState *epqstate, { skip_tuple = !ExecBRDeleteTriggers(estate, epqstate, resultRelInfo, &searchslot->tts_tuple->t_self, - NULL); + NULL, NULL); } if (!skip_tuple) diff --git a/src/backend/executor/nodeMerge.c b/src/backend/executor/nodeMerge.c new file mode 100644 index 0000000000..e633af7704 --- /dev/null +++ b/src/backend/executor/nodeMerge.c @@ -0,0 +1,566 @@ +/*------------------------------------------------------------------------- + * + * nodeMerge.c + * routines to handle Merge nodes. + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/executor/nodeMerge.c + * + *------------------------------------------------------------------------- + */ + + +#include "postgres.h" + +#include "access/htup_details.h" +#include "access/xact.h" +#include "commands/trigger.h" +#include "executor/execPartition.h" +#include "executor/executor.h" +#include "executor/nodeModifyTable.h" +#include "executor/nodeMerge.h" +#include "miscadmin.h" +#include "nodes/nodeFuncs.h" +#include "storage/bufmgr.h" +#include "storage/lmgr.h" +#include "utils/builtins.h" +#include "utils/memutils.h" +#include "utils/rel.h" +#include "utils/tqual.h" + + +/* + * Check and execute the first qualifying MATCHED action. The current target + * tuple is identified by tupleid. + * + * We start from the first WHEN MATCHED action and check if the additional WHEN + * quals pass. If the additional quals for the first action do not pass, we + * check the second, then the third and so on. If we reach to the end, no + * action is taken and we return true, indicating that no further action is + * required for this tuple. + * + * If we do find a qualifying action, then we attempt to execute the action. In + * case the tuple is concurrently updated, EvalPlanQual is run with the updated + * tuple to recheck the join quals. Note that the additional quals associated + * with individual actions are evaluated separately by the MERGE code, while + * EvalPlanQual checks for the join quals. If EvalPlanQual tells us that the + * updated tuple still passes the join quals, then we restart from the top and + * again look for a qualifying action. Otherwise, we return false indicating + * that a NOT MATCHED action must now be executed for the current source tuple. + */ +static bool +ExecMergeMatched(ModifyTableState *mtstate, EState *estate, + TupleTableSlot *slot, JunkFilter *junkfilter, + ItemPointer tupleid) +{ + ExprContext *econtext = mtstate->ps.ps_ExprContext; + bool isNull; + Datum datum; + Oid tableoid = InvalidOid; + List *mergeMatchedActionStates = NIL; + HeapUpdateFailureData hufd; + bool tuple_updated, + tuple_deleted; + Buffer buffer; + HeapTupleData tuple; + EPQState *epqstate = &mtstate->mt_epqstate; + ResultRelInfo *saved_resultRelInfo; + ResultRelInfo *resultRelInfo = estate->es_result_relation_info; + ListCell *l; + TupleTableSlot *saved_slot = slot; + + + /* + * We always fetch the tableoid while performing MATCHED MERGE action. + * This is strictly not required if the target table is not a partitioned + * table. But we are not yet optimising for that case. + */ + datum = ExecGetJunkAttribute(slot, junkfilter->jf_otherJunkAttNo, + &isNull); + Assert(!isNull); + tableoid = DatumGetObjectId(datum); + + if (mtstate->mt_partition_tuple_routing) + { + int leaf_part_index; + PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing; + + /* + * If we're dealing with a MATCHED tuple, then tableoid must have been + * set correctly. In case of partitioned table, we must now fetch the + * correct result relation corresponding to the child table emitting + * the matching target row. For normal table, there is just one result + * relation and it must be the one emitting the matching row. + */ + leaf_part_index = ExecFindPartitionByOid(proute, tableoid); + + resultRelInfo = proute->partitions[leaf_part_index]; + if (resultRelInfo == NULL) + { + resultRelInfo = ExecInitPartitionInfo(mtstate, + mtstate->resultRelInfo, + proute, estate, leaf_part_index); + Assert(resultRelInfo != NULL); + } + } + + /* + * Save the current information and work with the correct result relation. + */ + saved_resultRelInfo = resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + + /* + * And get the correct action lists. + */ + mergeMatchedActionStates = + resultRelInfo->ri_mergeState->matchedActionStates; + + /* + * If there are not WHEN MATCHED actions, we are done. + */ + if (mergeMatchedActionStates == NIL) + return true; + + /* + * Make tuple and any needed join variables available to ExecQual and + * ExecProject. The target's existing tuple is installed in the scantuple. + * Again, this target relation's slot is required only in the case of a + * MATCHED tuple and UPDATE/DELETE actions. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_existing, + resultRelInfo->ri_RelationDesc->rd_att); + econtext->ecxt_scantuple = mtstate->mt_existing; + econtext->ecxt_innertuple = slot; + econtext->ecxt_outertuple = NULL; + +lmerge_matched:; + slot = saved_slot; + buffer = InvalidBuffer; + + /* + * UPDATE/DELETE is only invoked for matched rows. And we must have found + * the tupleid of the target row in that case. We fetch using SnapshotAny + * because we might get called again after EvalPlanQual returns us a new + * tuple. This tuple may not be visible to our MVCC snapshot. + */ + Assert(tupleid != NULL); + + tuple.t_self = *tupleid; + if (!heap_fetch(resultRelInfo->ri_RelationDesc, SnapshotAny, &tuple, + &buffer, true, NULL)) + elog(ERROR, "Failed to fetch the target tuple"); + + /* Store target's existing tuple in the state's dedicated slot */ + ExecStoreTuple(&tuple, mtstate->mt_existing, buffer, false); + + foreach(l, mergeMatchedActionStates) + { + MergeActionState *action = (MergeActionState *) lfirst(l); + + /* + * Test condition, if any + * + * In the absence of a condition we perform the action unconditionally + * (no need to check separately since ExecQual() will return true if + * there are no conditions to evaluate). + */ + if (!ExecQual(action->whenqual, econtext)) + continue; + + /* + * Check if the existing target tuple meet the USING checks of + * UPDATE/DELETE RLS policies. If those checks fail, we throw an + * error. + * + * The WITH CHECK quals are applied in ExecUpdate() and hence we need + * not do anything special to handle them. + * + * NOTE: We must do this after WHEN quals are evaluated so that we + * check policies only when they matter. + */ + if (resultRelInfo->ri_WithCheckOptions) + { + ExecWithCheckOptions(action->commandType == CMD_UPDATE ? + WCO_RLS_MERGE_UPDATE_CHECK : WCO_RLS_MERGE_DELETE_CHECK, + resultRelInfo, + mtstate->mt_existing, + mtstate->ps.state); + } + + /* Perform stated action */ + switch (action->commandType) + { + case CMD_UPDATE: + + /* + * We set up the projection earlier, so all we do here is + * Project, no need for any other tasks prior to the + * ExecUpdate. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_mergeproj, action->tupDesc); + ExecProject(action->proj); + + /* + * We don't call ExecFilterJunk() because the projected tuple + * using the UPDATE action's targetlist doesn't have a junk + * attribute. + */ + slot = ExecUpdate(mtstate, tupleid, NULL, + mtstate->mt_mergeproj, + slot, epqstate, estate, + &tuple_updated, &hufd, + action, mtstate->canSetTag); + break; + + case CMD_DELETE: + /* Nothing to Project for a DELETE action */ + slot = ExecDelete(mtstate, tupleid, NULL, + slot, epqstate, estate, + &tuple_deleted, false, &hufd, action, + mtstate->canSetTag); + + break; + + default: + elog(ERROR, "unknown action in MERGE WHEN MATCHED clause"); + + } + + /* + * Check for any concurrent update/delete operation which may have + * prevented our update/delete. We also check for situations where we + * might be trying to update/delete the same tuple twice. + */ + if ((action->commandType == CMD_UPDATE && !tuple_updated) || + (action->commandType == CMD_DELETE && !tuple_deleted)) + + { + switch (hufd.result) + { + case HeapTupleMayBeUpdated: + break; + case HeapTupleInvisible: + + /* + * This state should never be reached since the underlying + * JOIN runs with a MVCC snapshot and should only return + * rows visible to us. + */ + elog(ERROR, "unexpected invisible tuple"); + break; + + case HeapTupleSelfUpdated: + + /* + * SQLStandard disallows this for MERGE. + */ + if (TransactionIdIsCurrentTransactionId(hufd.xmax)) + ereport(ERROR, + (errcode(ERRCODE_CARDINALITY_VIOLATION), + errmsg("MERGE command cannot affect row a second time"), + errhint("Ensure that not more than one source rows match any one target row"))); + /* This shouldn't happen */ + elog(ERROR, "attempted to update or delete invisible tuple"); + break; + + case HeapTupleUpdated: + + /* + * The target tuple was concurrently updated/deleted by + * some other transaction. + * + * If the current tuple is that last tuple in the update + * chain, then we know that the tuple was concurrently + * deleted. Just return and let the caller try NOT MATCHED + * actions. + * + * If the current tuple was concurrently updated, then we + * must run the EvalPlanQual() with the new version of the + * tuple. If EvalPlanQual() does not return a tuple (can + * that happen?), then again we switch to NOT MATCHED + * action. If it does return a tuple and the join qual is + * still satified, then we just need to recheck the + * MATCHED actions, starting from the top, and execute the + * first qualifying action. + */ + if (!ItemPointerEquals(tupleid, &hufd.ctid)) + { + TupleTableSlot *epqslot; + + /* + * Since we generate a JOIN query with a target table + * RTE different than the result relation RTE, we must + * pass in the RTI of the relation used in the join + * query and not the one from result relation. + */ + Assert(resultRelInfo->ri_mergeTargetRTI > 0); + epqslot = EvalPlanQual(estate, + epqstate, + resultRelInfo->ri_RelationDesc, + GetEPQRangeTableIndex(resultRelInfo), + LockTupleExclusive, + &hufd.ctid, + hufd.xmax); + + if (!TupIsNull(epqslot)) + { + (void) ExecGetJunkAttribute(epqslot, + resultRelInfo->ri_junkFilter->jf_junkAttNo, + &isNull); + + /* + * A valid ctid means that we are still dealing + * with MATCHED case. But we must retry from the + * start with the updated tuple to ensure that the + * first qualifying WHEN MATCHED action is + * executed. + * + * We don't use the new slot returned by + * EvalPlanQual because we anyways re-install the + * new target tuple in econtext->ecxt_scantuple + * before re-evaluating WHEN AND conditions and + * re-projecting the update targetlists. The + * source side tuple does not change and hence we + * can safely continue to use the old slot. + */ + if (!isNull) + { + /* + * Must update *tupleid to the TID of the + * newer tuple found in the update chain. + */ + *tupleid = hufd.ctid; + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + goto lmerge_matched; + } + } + } + + /* + * Tell the caller about the updated TID, restore the + * state back and return. + */ + *tupleid = hufd.ctid; + estate->es_result_relation_info = saved_resultRelInfo; + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + return false; + + default: + break; + + } + } + + if (action->commandType == CMD_UPDATE && tuple_updated) + InstrCountFiltered1(&mtstate->ps, 1); + if (action->commandType == CMD_DELETE && tuple_deleted) + InstrCountFiltered2(&mtstate->ps, 1); + + /* + * We've activated one of the WHEN clauses, so we don't search + * further. This is required behaviour, not an optimisation. + */ + estate->es_result_relation_info = saved_resultRelInfo; + break; + } + + if (BufferIsValid(buffer)) + ReleaseBuffer(buffer); + + /* + * Successfully executed an action or no qualifying action was found. + */ + return true; +} + +/* + * Execute the first qualifying NOT MATCHED action. + */ +static void +ExecMergeNotMatched(ModifyTableState *mtstate, EState *estate, + TupleTableSlot *slot) +{ + PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing; + ExprContext *econtext = mtstate->ps.ps_ExprContext; + List *mergeNotMatchedActionStates = NIL; + ResultRelInfo *resultRelInfo; + ListCell *l; + TupleTableSlot *myslot; + + /* + * We are dealing with NOT MATCHED tuple. Since for MERGE, partition tree + * is not expanded for the result relation, we continue to work with the + * currently active result relation, which should be of the root of the + * partition tree. + */ + resultRelInfo = mtstate->resultRelInfo; + + /* + * For INSERT actions, root relation's merge action is OK since the the + * INSERT's targetlist and the WHEN conditions can only refer to the + * source relation and hence it does not matter which result relation we + * work with. + */ + mergeNotMatchedActionStates = + resultRelInfo->ri_mergeState->notMatchedActionStates; + + /* + * Make source tuple available to ExecQual and ExecProject. We don't need + * the target tuple since the WHEN quals and the targetlist can't refer to + * the target columns. + */ + econtext->ecxt_scantuple = NULL; + econtext->ecxt_innertuple = slot; + econtext->ecxt_outertuple = NULL; + + foreach(l, mergeNotMatchedActionStates) + { + MergeActionState *action = (MergeActionState *) lfirst(l); + + /* + * Test condition, if any + * + * In the absence of a condition we perform the action unconditionally + * (no need to check separately since ExecQual() will return true if + * there are no conditions to evaluate). + */ + if (!ExecQual(action->whenqual, econtext)) + continue; + + /* Perform stated action */ + switch (action->commandType) + { + case CMD_INSERT: + + /* + * We set up the projection earlier, so all we do here is + * Project, no need for any other tasks prior to the + * ExecInsert. + */ + if (mtstate->mt_partition_tuple_routing) + ExecSetSlotDescriptor(mtstate->mt_mergeproj, action->tupDesc); + ExecProject(action->proj); + + /* + * ExecPrepareTupleRouting may modify the passed-in slot. Hence + * pass a local reference so that action->slot is not modified. + */ + myslot = mtstate->mt_mergeproj; + + /* Prepare for tuple routing if needed. */ + if (proute) + myslot = ExecPrepareTupleRouting(mtstate, estate, proute, + resultRelInfo, myslot); + slot = ExecInsert(mtstate, myslot, slot, + estate, action, + mtstate->canSetTag); + /* Revert ExecPrepareTupleRouting's state change. */ + if (proute) + estate->es_result_relation_info = resultRelInfo; + break; + case CMD_NOTHING: + /* Do Nothing */ + break; + default: + elog(ERROR, "unknown action in MERGE WHEN NOT MATCHED clause"); + } + + break; + } +} + +/* + * Perform MERGE. + */ +void +ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, + JunkFilter *junkfilter, ResultRelInfo *resultRelInfo) +{ + ItemPointer tupleid; + ItemPointerData tuple_ctid; + bool matched = false; + char relkind; + Datum datum; + bool isNull; + + relkind = resultRelInfo->ri_RelationDesc->rd_rel->relkind; + Assert(relkind == RELKIND_RELATION || + relkind == RELKIND_MATVIEW || + relkind == RELKIND_PARTITIONED_TABLE); + + /* + * We run a JOIN between the target relation and the source relation to + * find a set of candidate source rows that has matching row in the target + * table and a set of candidate source rows that does not have matching + * row in the target table. If the join returns us a tuple with target + * relation's tid set, that implies that the join found a matching row for + * the given source tuple. This case triggers the WHEN MATCHED clause of + * the MERGE. Whereas a NULL in the target relation's ctid column + * indicates a NOT MATCHED case. + */ + datum = ExecGetJunkAttribute(slot, junkfilter->jf_junkAttNo, &isNull); + + if (!isNull) + { + matched = true; + tupleid = (ItemPointer) DatumGetPointer(datum); + tuple_ctid = *tupleid; /* be sure we don't free ctid!! */ + tupleid = &tuple_ctid; + } + else + { + matched = false; + tupleid = NULL; /* we don't need it for INSERT actions */ + } + + /* + * If we are dealing with a WHEN MATCHED case, we look at the given WHEN + * MATCHED actions in an order and execute the first action which also + * satisfies the additional WHEN MATCHED AND quals. If an action without + * any additional quals is found, that action is executed. + * + * Similarly, if we are dealing with WHEN NOT MATCHED case, we look at the + * given WHEN NOT MATCHED actions in an ordr and execute the first + * qualifying action. + * + * Things get interesting in case of concurrent update/delete of the + * target tuple. Such concurrent update/delete is detected while we are + * executing a WHEN MATCHED action. + * + * A concurrent update for example can: + * + * 1. modify the target tuple so that it no longer satisfies the + * additional quals attached to the current WHEN MATCHED action OR 2. + * modify the target tuple so that the join quals no longer pass and hence + * the source tuple no longer has a match. + * + * In the first case, we are still dealing with a WHEN MATCHED case, but + * we should recheck the list of WHEN MATCHED actions and choose the first + * one that satisfies the new target tuple. In the second case, since the + * source tuple no longer matches the target tuple, we now instead find a + * qualifying WHEN NOT MATCHED action and execute that. + * + * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. + * + * ExecMergeMatched takes care of following the update chain and + * re-finding the qualifying WHEN MATCHED action, as long as the updated + * target tuple still satisfies the join quals i.e. it still remains a + * WHEN MATCHED case. If the tuple gets deleted or the join quals fail, it + * returns and we try ExecMergeNotMatched. Given that ExecMergeMatched + * always make progress by following the update chain and we never switch + * from ExecMergeNotMatched to ExecMergeMatched, there is no risk of a + * livelock. + */ + if (!matched || + !ExecMergeMatched(mtstate, estate, slot, junkfilter, tupleid)) + ExecMergeNotMatched(mtstate, estate, slot); +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 1b09868ff8..def7eec223 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -42,6 +42,7 @@ #include "commands/trigger.h" #include "executor/execPartition.h" #include "executor/executor.h" +#include "executor/nodeMerge.h" #include "executor/nodeModifyTable.h" #include "foreign/fdwapi.h" #include "miscadmin.h" @@ -62,17 +63,17 @@ static bool ExecOnConflictUpdate(ModifyTableState *mtstate, EState *estate, bool canSetTag, TupleTableSlot **returning); -static TupleTableSlot *ExecPrepareTupleRouting(ModifyTableState *mtstate, - EState *estate, - PartitionTupleRouting *proute, - ResultRelInfo *targetRelInfo, - TupleTableSlot *slot); static ResultRelInfo *getTargetResultRelInfo(ModifyTableState *node); static void ExecSetupChildParentMapForTcs(ModifyTableState *mtstate); static void ExecSetupChildParentMapForSubplan(ModifyTableState *mtstate); static TupleConversionMap *tupconv_map_for_subplan(ModifyTableState *node, int whichplan); +/* flags for mt_merge_subcommands */ +#define MERGE_INSERT 0x01 +#define MERGE_UPDATE 0x02 +#define MERGE_DELETE 0x04 + /* * Verify that the tuples to be produced by INSERT or UPDATE match the * target relation's rowtype @@ -259,11 +260,12 @@ ExecCheckTIDVisible(EState *estate, * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +extern TupleTableSlot * ExecInsert(ModifyTableState *mtstate, TupleTableSlot *slot, TupleTableSlot *planSlot, EState *estate, + MergeActionState *actionState, bool canSetTag) { HeapTuple tuple; @@ -390,9 +392,17 @@ ExecInsert(ModifyTableState *mtstate, * partition, we should instead check UPDATE policies, because we are * executing policies defined on the target table, and not those * defined on the child partitions. + * + * If we're running MERGE, we refer to the action that we're executing + * to know if we're doing an INSERT or UPDATE to a partition table. */ - wco_kind = (mtstate->operation == CMD_UPDATE) ? - WCO_RLS_UPDATE_CHECK : WCO_RLS_INSERT_CHECK; + if (mtstate->operation == CMD_UPDATE) + wco_kind = WCO_RLS_UPDATE_CHECK; + else if (mtstate->operation == CMD_INSERT) + wco_kind = WCO_RLS_INSERT_CHECK; + else if (mtstate->operation == CMD_MERGE) + wco_kind = (actionState->commandType == CMD_UPDATE) ? + WCO_RLS_UPDATE_CHECK : WCO_RLS_INSERT_CHECK; /* * ExecWithCheckOptions() will skip any WCOs which are not of the kind @@ -617,10 +627,19 @@ ExecInsert(ModifyTableState *mtstate, * passed to foreign table triggers; it is NULL when the foreign * table has no relevant triggers. * + * MERGE passes actionState of the action it's currently executing; + * regular DELETE passes NULL. This is used by ExecDelete to know if it's + * being called from MERGE or regular DELETE operation. + * + * If the DELETE fails because the tuple is concurrently updated/deleted + * by this or some other transaction, hufdp is filled with the reason as + * well as other important information. Currently only MERGE needs this + * information. + * * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +TupleTableSlot * ExecDelete(ModifyTableState *mtstate, ItemPointer tupleid, HeapTuple oldtuple, @@ -629,6 +648,8 @@ ExecDelete(ModifyTableState *mtstate, EState *estate, bool *tupleDeleted, bool processReturning, + HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag) { ResultRelInfo *resultRelInfo; @@ -641,6 +662,14 @@ ExecDelete(ModifyTableState *mtstate, if (tupleDeleted) *tupleDeleted = false; + /* + * Initialize hufdp. Since the caller is only interested in the failure + * status, initialize with the state that is used to indicate successful + * operation. + */ + if (hufdp) + hufdp->result = HeapTupleMayBeUpdated; + /* * get information on the (current) result relation */ @@ -654,7 +683,7 @@ ExecDelete(ModifyTableState *mtstate, bool dodelete; dodelete = ExecBRDeleteTriggers(estate, epqstate, resultRelInfo, - tupleid, oldtuple); + tupleid, oldtuple, hufdp); if (!dodelete) /* "do nothing" */ return NULL; @@ -721,6 +750,15 @@ ldelete:; estate->es_crosscheck_snapshot, true /* wait for commit */ , &hufd); + + /* + * Copy the necessary information, if the caller has asked for it. We + * must do this irrespective of whether the tuple was updated or + * deleted. + */ + if (hufdp) + *hufdp = hufd; + switch (result) { case HeapTupleSelfUpdated: @@ -755,7 +793,11 @@ ldelete:; errmsg("tuple to be updated was already modified by an operation triggered by the current command"), errhint("Consider using an AFTER trigger instead of a BEFORE trigger to propagate changes to other rows."))); - /* Else, already deleted by self; nothing to do */ + /* + * Else, already deleted by self; nothing to do but inform + * MERGE about it anyways so that it can take necessary + * action. + */ return NULL; case HeapTupleMayBeUpdated: @@ -766,14 +808,24 @@ ldelete:; ereport(ERROR, (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE), errmsg("could not serialize access due to concurrent update"))); + if (!ItemPointerEquals(tupleid, &hufd.ctid)) { TupleTableSlot *epqslot; + /* + * If we're executing MERGE, then the onus of running + * EvalPlanQual() and handling its outcome lies with the + * caller. + */ + if (actionState != NULL) + return NULL; + + /* Normal DELETE path. */ epqslot = EvalPlanQual(estate, epqstate, resultRelationDesc, - resultRelInfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(resultRelInfo), LockTupleExclusive, &hufd.ctid, hufd.xmax); @@ -783,7 +835,12 @@ ldelete:; goto ldelete; } } - /* tuple already deleted; nothing to do */ + + /* + * tuple already deleted; nothing to do. But MERGE might want + * to handle it differently. We've already filled-in hufdp + * with sufficient information for MERGE to look at. + */ return NULL; default: @@ -911,10 +968,21 @@ ldelete:; * foreign table triggers; it is NULL when the foreign table has * no relevant triggers. * + * MERGE passes actionState of the action it's currently executing; + * regular UPDATE passes NULL. This is used by ExecUpdate to know if it's + * being called from MERGE or regular UPDATE operation. ExecUpdate may + * pass this information to ExecInsert if it ends up running DELETE+INSERT + * for partition key updates. + * + * If the UPDATE fails because the tuple is concurrently updated/deleted + * by this or some other transaction, hufdp is filled with the reason as + * well as other important information. Currently only MERGE needs this + * information. + * * Returns RETURNING result if any, otherwise NULL. * ---------------------------------------------------------------- */ -static TupleTableSlot * +extern TupleTableSlot * ExecUpdate(ModifyTableState *mtstate, ItemPointer tupleid, HeapTuple oldtuple, @@ -922,6 +990,9 @@ ExecUpdate(ModifyTableState *mtstate, TupleTableSlot *planSlot, EPQState *epqstate, EState *estate, + bool *tuple_updated, + HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag) { HeapTuple tuple; @@ -938,6 +1009,17 @@ ExecUpdate(ModifyTableState *mtstate, if (IsBootstrapProcessingMode()) elog(ERROR, "cannot UPDATE during bootstrap"); + if (tuple_updated) + *tuple_updated = false; + + /* + * Initialize hufdp. Since the caller is only interested in the failure + * status, initialize with the state that is used to indicate successful + * operation. + */ + if (hufdp) + hufdp->result = HeapTupleMayBeUpdated; + /* * get the heap tuple out of the tuple table slot, making sure we have a * writable copy @@ -955,7 +1037,7 @@ ExecUpdate(ModifyTableState *mtstate, resultRelInfo->ri_TrigDesc->trig_update_before_row) { slot = ExecBRUpdateTriggers(estate, epqstate, resultRelInfo, - tupleid, oldtuple, slot); + tupleid, oldtuple, slot, hufdp); if (slot == NULL) /* "do nothing" */ return NULL; @@ -1079,8 +1161,9 @@ lreplace:; * Row movement, part 1. Delete the tuple, but skip RETURNING * processing. We want to return rows from INSERT. */ - ExecDelete(mtstate, tupleid, oldtuple, planSlot, epqstate, estate, - &tuple_deleted, false, false); + ExecDelete(mtstate, tupleid, oldtuple, planSlot, epqstate, + estate, &tuple_deleted, false, hufdp, NULL, + false); /* * For some reason if DELETE didn't happen (e.g. trigger prevented @@ -1116,16 +1199,36 @@ lreplace:; saved_tcs_map = mtstate->mt_transition_capture->tcs_map; /* - * resultRelInfo is one of the per-subplan resultRelInfos. So we - * should convert the tuple into root's tuple descriptor, since - * ExecInsert() starts the search from root. The tuple conversion - * map list is in the order of mtstate->resultRelInfo[], so to - * retrieve the one for this resultRel, we need to know the - * position of the resultRel in mtstate->resultRelInfo[]. + * We should convert the tuple into root's tuple descriptor, since + * ExecInsert() starts the search from root. To do that, we need to + * retrieve the tuple conversion map for this resultRelInfo. + * + * If we're running MERGE then resultRelInfo is per-partition + * resultRelInfo as initialised in ExecInitPartitionInfo(). Note + * that we don't expand inheritance for the resultRelation in case + * of MERGE and hence there is just one subplan. Whereas for + * regular UPDATE, resultRelInfo is one of the per-subplan + * resultRelInfos. In either case the position of this partition in + * tracked in ri_PartitionLeafIndex; + * + * Retrieve the map either by looking at the resultRelInfo's + * position in mtstate->resultRelInfo[] (for UPDATE) or by simply + * using the ri_PartitionLeafIndex value (for MERGE). */ - map_index = resultRelInfo - mtstate->resultRelInfo; - Assert(map_index >= 0 && map_index < mtstate->mt_nplans); - tupconv_map = tupconv_map_for_subplan(mtstate, map_index); + if (mtstate->operation == CMD_MERGE) + { + map_index = resultRelInfo->ri_PartitionLeafIndex; + Assert(mtstate->rootResultRelInfo == NULL); + tupconv_map = TupConvMapForLeaf(proute, + mtstate->resultRelInfo, + map_index); + } + else + { + map_index = resultRelInfo - mtstate->resultRelInfo; + Assert(map_index >= 0 && map_index < mtstate->mt_nplans); + tupconv_map = tupconv_map_for_subplan(mtstate, map_index); + } tuple = ConvertPartitionTupleSlot(tupconv_map, tuple, proute->root_tuple_slot, @@ -1135,12 +1238,16 @@ lreplace:; * Prepare for tuple routing, making it look like we're inserting * into the root. */ - Assert(mtstate->rootResultRelInfo != NULL); slot = ExecPrepareTupleRouting(mtstate, estate, proute, - mtstate->rootResultRelInfo, slot); + getTargetResultRelInfo(mtstate), + slot); ret_slot = ExecInsert(mtstate, slot, planSlot, - estate, canSetTag); + estate, actionState, canSetTag); + + /* Update is successful. */ + if (tuple_updated) + *tuple_updated = true; /* Revert ExecPrepareTupleRouting's node change. */ estate->es_result_relation_info = resultRelInfo; @@ -1179,6 +1286,15 @@ lreplace:; estate->es_crosscheck_snapshot, true /* wait for commit */ , &hufd, &lockmode); + + /* + * Copy the necessary information, if the caller has asked for it. We + * must do this irrespective of whether the tuple was updated or + * deleted. + */ + if (hufdp) + *hufdp = hufd; + switch (result) { case HeapTupleSelfUpdated: @@ -1223,26 +1339,42 @@ lreplace:; ereport(ERROR, (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE), errmsg("could not serialize access due to concurrent update"))); + if (!ItemPointerEquals(tupleid, &hufd.ctid)) { TupleTableSlot *epqslot; + /* + * If we're executing MERGE, then the onus of running + * EvalPlanQual() and handling its outcome lies with the + * caller. + */ + if (actionState != NULL) + return NULL; + + /* Regular UPDATE path. */ epqslot = EvalPlanQual(estate, epqstate, resultRelationDesc, - resultRelInfo->ri_RangeTableIndex, + GetEPQRangeTableIndex(resultRelInfo), lockmode, &hufd.ctid, hufd.xmax); if (!TupIsNull(epqslot)) { *tupleid = hufd.ctid; + /* Normal UPDATE path */ slot = ExecFilterJunk(resultRelInfo->ri_junkFilter, epqslot); tuple = ExecMaterializeSlot(slot); goto lreplace; } } - /* tuple already deleted; nothing to do */ + + /* + * tuple already deleted; nothing to do. But MERGE might want + * to handle it differently. We've already filled-in hufdp + * with sufficient information for MERGE to look at. + */ return NULL; default: @@ -1271,6 +1403,9 @@ lreplace:; estate, false, NULL, NIL); } + if (tuple_updated) + *tuple_updated = true; + if (canSetTag) (estate->es_processed)++; @@ -1365,9 +1500,9 @@ ExecOnConflictUpdate(ModifyTableState *mtstate, * there's no historical behavior to break. * * It is the user's responsibility to prevent this situation from - * occurring. These problems are why SQL-2003 similarly specifies - * that for SQL MERGE, an exception must be raised in the event of - * an attempt to update the same row twice. + * occurring. These problems are why SQL Standard similarly + * specifies that for SQL MERGE, an exception must be raised in + * the event of an attempt to update the same row twice. */ if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(tuple.t_data))) ereport(ERROR, @@ -1489,7 +1624,7 @@ ExecOnConflictUpdate(ModifyTableState *mtstate, *returning = ExecUpdate(mtstate, &tuple.t_self, NULL, mtstate->mt_conflproj, planSlot, &mtstate->mt_epqstate, mtstate->ps.state, - canSetTag); + NULL, NULL, NULL, canSetTag); ReleaseBuffer(buffer); return true; @@ -1527,6 +1662,14 @@ fireBSTriggers(ModifyTableState *node) case CMD_DELETE: ExecBSDeleteTriggers(node->ps.state, resultRelInfo); break; + case CMD_MERGE: + if (node->mt_merge_subcommands & MERGE_INSERT) + ExecBSInsertTriggers(node->ps.state, resultRelInfo); + if (node->mt_merge_subcommands & MERGE_UPDATE) + ExecBSUpdateTriggers(node->ps.state, resultRelInfo); + if (node->mt_merge_subcommands & MERGE_DELETE) + ExecBSDeleteTriggers(node->ps.state, resultRelInfo); + break; default: elog(ERROR, "unknown operation"); break; @@ -1582,6 +1725,17 @@ fireASTriggers(ModifyTableState *node) ExecASDeleteTriggers(node->ps.state, resultRelInfo, node->mt_transition_capture); break; + case CMD_MERGE: + if (node->mt_merge_subcommands & MERGE_DELETE) + ExecASDeleteTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + if (node->mt_merge_subcommands & MERGE_UPDATE) + ExecASUpdateTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + if (node->mt_merge_subcommands & MERGE_INSERT) + ExecASInsertTriggers(node->ps.state, resultRelInfo, + node->mt_transition_capture); + break; default: elog(ERROR, "unknown operation"); break; @@ -1644,7 +1798,7 @@ ExecSetupTransitionCaptureState(ModifyTableState *mtstate, EState *estate) * * Returns a slot holding the tuple of the partition rowtype. */ -static TupleTableSlot * +TupleTableSlot * ExecPrepareTupleRouting(ModifyTableState *mtstate, EState *estate, PartitionTupleRouting *proute, @@ -1967,6 +2121,7 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; + if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2015,6 +2170,12 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + if (operation == CMD_MERGE) + { + ExecMerge(node, estate, slot, junkfilter, resultRelInfo); + continue; + } + tupleid = NULL; oldtuple = NULL; if (junkfilter != NULL) @@ -2096,19 +2257,20 @@ ExecModifyTable(PlanState *pstate) slot = ExecPrepareTupleRouting(node, estate, proute, resultRelInfo, slot); slot = ExecInsert(node, slot, planSlot, - estate, node->canSetTag); + estate, NULL, node->canSetTag); /* Revert ExecPrepareTupleRouting's state change. */ if (proute) estate->es_result_relation_info = resultRelInfo; break; case CMD_UPDATE: slot = ExecUpdate(node, tupleid, oldtuple, slot, planSlot, - &node->mt_epqstate, estate, node->canSetTag); + &node->mt_epqstate, estate, + NULL, NULL, NULL, node->canSetTag); break; case CMD_DELETE: slot = ExecDelete(node, tupleid, oldtuple, planSlot, &node->mt_epqstate, estate, - NULL, true, node->canSetTag); + NULL, true, NULL, NULL, node->canSetTag); break; default: elog(ERROR, "unknown operation"); @@ -2198,6 +2360,16 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) saved_resultRelInfo = estate->es_result_relation_info; resultRelInfo = mtstate->resultRelInfo; + + /* + * mergeTargetRelation must be set if we're running MERGE and mustn't be + * set if we're not. + */ + Assert(operation != CMD_MERGE || node->mergeTargetRelation > 0); + Assert(operation == CMD_MERGE || node->mergeTargetRelation == 0); + + resultRelInfo->ri_mergeTargetRTI = node->mergeTargetRelation; + i = 0; foreach(l, node->plans) { @@ -2276,7 +2448,8 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) * partition key. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE && - (operation == CMD_INSERT || update_tuple_routing_needed)) + (operation == CMD_INSERT || operation == CMD_MERGE || + update_tuple_routing_needed)) mtstate->mt_partition_tuple_routing = ExecSetupPartitionTupleRouting(mtstate, rel); @@ -2287,6 +2460,15 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) if (!(eflags & EXEC_FLAG_EXPLAIN_ONLY)) ExecSetupTransitionCaptureState(mtstate, estate); + /* + * If we are doing MERGE then setup child-parent mapping. This will be + * required in case we end up doing a partition-key update, triggering a + * tuple routing. + */ + if (mtstate->operation == CMD_MERGE && + mtstate->mt_partition_tuple_routing != NULL) + ExecSetupChildParentMapForLeaf(mtstate->mt_partition_tuple_routing); + /* * Construct mapping from each of the per-subplan partition attnos to the * root attno. This is required when during update row movement the tuple @@ -2478,6 +2660,102 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) } } + resultRelInfo = mtstate->resultRelInfo; + + if (node->mergeActionList) + { + ListCell *l; + ExprContext *econtext; + List *mergeMatchedActionStates = NIL; + List *mergeNotMatchedActionStates = NIL; + TupleDesc relationDesc = resultRelInfo->ri_RelationDesc->rd_att; + + mtstate->mt_merge_subcommands = 0; + + if (mtstate->ps.ps_ExprContext == NULL) + ExecAssignExprContext(estate, &mtstate->ps); + + econtext = mtstate->ps.ps_ExprContext; + + /* initialize slot for the existing tuple */ + Assert(mtstate->mt_existing == NULL); + mtstate->mt_existing = + ExecInitExtraTupleSlot(mtstate->ps.state, + mtstate->mt_partition_tuple_routing ? + NULL : relationDesc); + + /* initialise slot for merge actions */ + Assert(mtstate->mt_mergeproj == NULL); + mtstate->mt_mergeproj = + ExecInitExtraTupleSlot(mtstate->ps.state, + mtstate->mt_partition_tuple_routing ? + NULL : relationDesc); + + /* + * Create a MergeActionState for each action on the mergeActionList + * and add it to either a list of matched actions or not-matched + * actions. + */ + foreach(l, node->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + MergeActionState *action_state = makeNode(MergeActionState); + TupleDesc tupDesc; + + action_state->matched = action->matched; + action_state->commandType = action->commandType; + action_state->whenqual = ExecInitQual((List *) action->qual, + &mtstate->ps); + + /* create target slot for this action's projection */ + tupDesc = ExecTypeFromTL((List *) action->targetList, + resultRelInfo->ri_RelationDesc->rd_rel->relhasoids); + action_state->tupDesc = tupDesc; + + /* build action projection state */ + action_state->proj = + ExecBuildProjectionInfo(action->targetList, econtext, + mtstate->mt_mergeproj, &mtstate->ps, + resultRelInfo->ri_RelationDesc->rd_att); + + /* + * We create two lists - one for WHEN MATCHED actions and one + * for WHEN NOT MATCHED actions - and stick the + * MergeActionState into the appropriate list. + */ + if (action_state->matched) + mergeMatchedActionStates = + lappend(mergeMatchedActionStates, action_state); + else + mergeNotMatchedActionStates = + lappend(mergeNotMatchedActionStates, action_state); + + switch (action->commandType) + { + case CMD_INSERT: + mtstate->mt_merge_subcommands |= MERGE_INSERT; + break; + case CMD_UPDATE: + mtstate->mt_merge_subcommands |= MERGE_UPDATE; + break; + case CMD_DELETE: + mtstate->mt_merge_subcommands |= MERGE_DELETE; + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown operation"); + break; + } + + resultRelInfo->ri_mergeState->matchedActionStates = + mergeMatchedActionStates; + resultRelInfo->ri_mergeState->notMatchedActionStates = + mergeNotMatchedActionStates; + + } + } + /* select first subplan */ mtstate->mt_whichplan = 0; subplan = (Plan *) linitial(node->plans); @@ -2491,7 +2769,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) * --- no need to look first. Typically, this will be a 'ctid' or * 'wholerow' attribute, but in the case of a foreign data wrapper it * might be a set of junk attributes sufficient to identify the remote - * row. + * row. We follow this logic for MERGE, so it always has a junk 'ctid'. * * If there are multiple result relations, each one needs its own junk * filter. Note multiple rels are only possible for UPDATE/DELETE, so we @@ -2519,6 +2797,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) break; case CMD_UPDATE: case CMD_DELETE: + case CMD_MERGE: junk_filter_needed = true; break; default: @@ -2534,6 +2813,11 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) JunkFilter *j; subplan = mtstate->mt_plans[i]->plan; + + /* + * XXX we probably need to check plan output for CMD_MERGE + * also + */ if (operation == CMD_INSERT || operation == CMD_UPDATE) ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc, subplan->targetlist); @@ -2542,7 +2826,9 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) resultRelInfo->ri_RelationDesc->rd_att->tdhasoid, ExecInitExtraTupleSlot(estate, NULL)); - if (operation == CMD_UPDATE || operation == CMD_DELETE) + if (operation == CMD_UPDATE || + operation == CMD_DELETE || + operation == CMD_MERGE) { /* For UPDATE/DELETE, find the appropriate junk attr now */ char relkind; @@ -2555,6 +2841,14 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) j->jf_junkAttNo = ExecFindJunkAttribute(j, "ctid"); if (!AttributeNumberIsValid(j->jf_junkAttNo)) elog(ERROR, "could not find junk ctid column"); + + if (operation == CMD_MERGE) + { + j->jf_otherJunkAttNo = ExecFindJunkAttribute(j, "tableoid"); + if (!AttributeNumberIsValid(j->jf_otherJunkAttNo)) + elog(ERROR, "could not find junk tableoid column"); + + } } else if (relkind == RELKIND_FOREIGN_TABLE) { diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c index 9fc4431b80..e050a317c0 100644 --- a/src/backend/executor/spi.c +++ b/src/backend/executor/spi.c @@ -2404,6 +2404,9 @@ _SPI_pquery(QueryDesc *queryDesc, bool fire_triggers, uint64 tcount) else res = SPI_OK_UPDATE; break; + case CMD_MERGE: + res = SPI_OK_MERGE; + break; default: return SPI_ERROR_OPUNKNOWN; } diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index c7293a60d7..770ed3b1a8 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -207,6 +207,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(partitioned_rels); COPY_SCALAR_FIELD(partColsUpdated); COPY_NODE_FIELD(resultRelations); + COPY_SCALAR_FIELD(mergeTargetRelation); COPY_SCALAR_FIELD(resultRelIndex); COPY_SCALAR_FIELD(rootResultRelIndex); COPY_NODE_FIELD(plans); @@ -222,6 +223,8 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(onConflictWhere); COPY_SCALAR_FIELD(exclRelRTI); COPY_NODE_FIELD(exclRelTlist); + COPY_NODE_FIELD(mergeSourceTargetList); + COPY_NODE_FIELD(mergeActionList); return newnode; } @@ -2977,6 +2980,9 @@ _copyQuery(const Query *from) COPY_NODE_FIELD(setOperations); COPY_NODE_FIELD(constraintDeps); COPY_NODE_FIELD(withCheckOptions); + COPY_SCALAR_FIELD(mergeTarget_relation); + COPY_NODE_FIELD(mergeSourceTargetList); + COPY_NODE_FIELD(mergeActionList); COPY_LOCATION_FIELD(stmt_location); COPY_LOCATION_FIELD(stmt_len); @@ -3040,6 +3046,34 @@ _copyUpdateStmt(const UpdateStmt *from) return newnode; } +static MergeStmt * +_copyMergeStmt(const MergeStmt *from) +{ + MergeStmt *newnode = makeNode(MergeStmt); + + COPY_NODE_FIELD(relation); + COPY_NODE_FIELD(source_relation); + COPY_NODE_FIELD(join_condition); + COPY_NODE_FIELD(mergeActionList); + + return newnode; +} + +static MergeAction * +_copyMergeAction(const MergeAction *from) +{ + MergeAction *newnode = makeNode(MergeAction); + + COPY_SCALAR_FIELD(matched); + COPY_SCALAR_FIELD(commandType); + COPY_NODE_FIELD(condition); + COPY_NODE_FIELD(qual); + COPY_NODE_FIELD(stmt); + COPY_NODE_FIELD(targetList); + + return newnode; +} + static SelectStmt * _copySelectStmt(const SelectStmt *from) { @@ -5102,6 +5136,12 @@ copyObjectImpl(const void *from) case T_UpdateStmt: retval = _copyUpdateStmt(from); break; + case T_MergeStmt: + retval = _copyMergeStmt(from); + break; + case T_MergeAction: + retval = _copyMergeAction(from); + break; case T_SelectStmt: retval = _copySelectStmt(from); break; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 765b1be74b..5a0151eece 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -987,6 +987,8 @@ _equalQuery(const Query *a, const Query *b) COMPARE_NODE_FIELD(setOperations); COMPARE_NODE_FIELD(constraintDeps); COMPARE_NODE_FIELD(withCheckOptions); + COMPARE_NODE_FIELD(mergeSourceTargetList); + COMPARE_NODE_FIELD(mergeActionList); COMPARE_LOCATION_FIELD(stmt_location); COMPARE_LOCATION_FIELD(stmt_len); @@ -1042,6 +1044,30 @@ _equalUpdateStmt(const UpdateStmt *a, const UpdateStmt *b) return true; } +static bool +_equalMergeStmt(const MergeStmt *a, const MergeStmt *b) +{ + COMPARE_NODE_FIELD(relation); + COMPARE_NODE_FIELD(source_relation); + COMPARE_NODE_FIELD(join_condition); + COMPARE_NODE_FIELD(mergeActionList); + + return true; +} + +static bool +_equalMergeAction(const MergeAction *a, const MergeAction *b) +{ + COMPARE_SCALAR_FIELD(matched); + COMPARE_SCALAR_FIELD(commandType); + COMPARE_NODE_FIELD(condition); + COMPARE_NODE_FIELD(qual); + COMPARE_NODE_FIELD(stmt); + COMPARE_NODE_FIELD(targetList); + + return true; +} + static bool _equalSelectStmt(const SelectStmt *a, const SelectStmt *b) { @@ -3233,6 +3259,12 @@ equal(const void *a, const void *b) case T_UpdateStmt: retval = _equalUpdateStmt(a, b); break; + case T_MergeStmt: + retval = _equalMergeStmt(a, b); + break; + case T_MergeAction: + retval = _equalMergeAction(a, b); + break; case T_SelectStmt: retval = _equalSelectStmt(a, b); break; diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 6c76c41ebe..68e2cec66e 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -2146,6 +2146,16 @@ expression_tree_walker(Node *node, return true; } break; + case T_MergeAction: + { + MergeAction *action = (MergeAction *) node; + + if (walker(action->targetList, context)) + return true; + if (walker(action->qual, context)) + return true; + } + break; case T_JoinExpr: { JoinExpr *join = (JoinExpr *) node; @@ -2255,6 +2265,10 @@ query_tree_walker(Query *query, return true; if (walker((Node *) query->onConflict, context)) return true; + if (walker((Node *) query->mergeSourceTargetList, context)) + return true; + if (walker((Node *) query->mergeActionList, context)) + return true; if (walker((Node *) query->returningList, context)) return true; if (walker((Node *) query->jointree, context)) @@ -2932,6 +2946,18 @@ expression_tree_mutator(Node *node, return (Node *) newnode; } break; + case T_MergeAction: + { + MergeAction *action = (MergeAction *) node; + MergeAction *newnode; + + FLATCOPY(newnode, action, MergeAction); + MUTATE(newnode->qual, action->qual, Node *); + MUTATE(newnode->targetList, action->targetList, List *); + + return (Node *) newnode; + } + break; case T_JoinExpr: { JoinExpr *join = (JoinExpr *) node; @@ -3083,6 +3109,8 @@ query_tree_mutator(Query *query, MUTATE(query->targetList, query->targetList, List *); MUTATE(query->withCheckOptions, query->withCheckOptions, List *); MUTATE(query->onConflict, query->onConflict, OnConflictExpr *); + MUTATE(query->mergeSourceTargetList, query->mergeSourceTargetList, List *); + MUTATE(query->mergeActionList, query->mergeActionList, List *); MUTATE(query->returningList, query->returningList, List *); MUTATE(query->jointree, query->jointree, FromExpr *); MUTATE(query->setOperations, query->setOperations, Node *); @@ -3224,9 +3252,9 @@ query_or_expression_tree_mutator(Node *node, * boundaries: we descend to everything that's possibly interesting. * * Currently, the node type coverage here extends only to DML statements - * (SELECT/INSERT/UPDATE/DELETE) and nodes that can appear in them, because - * this is used mainly during analysis of CTEs, and only DML statements can - * appear in CTEs. + * (SELECT/INSERT/UPDATE/DELETE/MERGE) and nodes that can appear in them, + * because this is used mainly during analysis of CTEs, and only DML + * statements can appear in CTEs. */ bool raw_expression_tree_walker(Node *node, @@ -3406,6 +3434,20 @@ raw_expression_tree_walker(Node *node, return true; } break; + case T_MergeStmt: + { + MergeStmt *stmt = (MergeStmt *) node; + + if (walker(stmt->relation, context)) + return true; + if (walker(stmt->source_relation, context)) + return true; + if (walker(stmt->join_condition, context)) + return true; + if (walker(stmt->mergeActionList, context)) + return true; + } + break; case T_SelectStmt: { SelectStmt *stmt = (SelectStmt *) node; diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f61ae03ac5..9ebea55048 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -375,6 +375,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(partitioned_rels); WRITE_BOOL_FIELD(partColsUpdated); WRITE_NODE_FIELD(resultRelations); + WRITE_INT_FIELD(mergeTargetRelation); WRITE_INT_FIELD(resultRelIndex); WRITE_INT_FIELD(rootResultRelIndex); WRITE_NODE_FIELD(plans); @@ -390,6 +391,21 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(onConflictWhere); WRITE_UINT_FIELD(exclRelRTI); WRITE_NODE_FIELD(exclRelTlist); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); +} + +static void +_outMergeAction(StringInfo str, const MergeAction *node) +{ + WRITE_NODE_TYPE("MERGEACTION"); + + WRITE_BOOL_FIELD(matched); + WRITE_ENUM_FIELD(commandType, CmdType); + WRITE_NODE_FIELD(condition); + WRITE_NODE_FIELD(qual); + /* We don't dump the stmt node */ + WRITE_NODE_FIELD(targetList); } static void @@ -2114,6 +2130,7 @@ _outModifyTablePath(StringInfo str, const ModifyTablePath *node) WRITE_NODE_FIELD(partitioned_rels); WRITE_BOOL_FIELD(partColsUpdated); WRITE_NODE_FIELD(resultRelations); + WRITE_INT_FIELD(mergeTargetRelation); WRITE_NODE_FIELD(subpaths); WRITE_NODE_FIELD(subroots); WRITE_NODE_FIELD(withCheckOptionLists); @@ -2121,6 +2138,8 @@ _outModifyTablePath(StringInfo str, const ModifyTablePath *node) WRITE_NODE_FIELD(rowMarks); WRITE_NODE_FIELD(onconflict); WRITE_INT_FIELD(epqParam); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); } static void @@ -2942,6 +2961,9 @@ _outQuery(StringInfo str, const Query *node) WRITE_NODE_FIELD(setOperations); WRITE_NODE_FIELD(constraintDeps); /* withCheckOptions intentionally omitted, see comment in parsenodes.h */ + WRITE_INT_FIELD(mergeTarget_relation); + WRITE_NODE_FIELD(mergeSourceTargetList); + WRITE_NODE_FIELD(mergeActionList); WRITE_LOCATION_FIELD(stmt_location); WRITE_LOCATION_FIELD(stmt_len); } @@ -3657,6 +3679,9 @@ outNode(StringInfo str, const void *obj) case T_ModifyTable: _outModifyTable(str, obj); break; + case T_MergeAction: + _outMergeAction(str, obj); + break; case T_Append: _outAppend(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index fd4586e73d..3b8071d056 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -270,6 +270,9 @@ _readQuery(void) READ_NODE_FIELD(setOperations); READ_NODE_FIELD(constraintDeps); /* withCheckOptions intentionally omitted, see comment in parsenodes.h */ + READ_INT_FIELD(mergeTarget_relation); + READ_NODE_FIELD(mergeSourceTargetList); + READ_NODE_FIELD(mergeActionList); READ_LOCATION_FIELD(stmt_location); READ_LOCATION_FIELD(stmt_len); @@ -1576,6 +1579,7 @@ _readModifyTable(void) READ_NODE_FIELD(partitioned_rels); READ_BOOL_FIELD(partColsUpdated); READ_NODE_FIELD(resultRelations); + READ_INT_FIELD(mergeTargetRelation); READ_INT_FIELD(resultRelIndex); READ_INT_FIELD(rootResultRelIndex); READ_NODE_FIELD(plans); @@ -1591,6 +1595,8 @@ _readModifyTable(void) READ_NODE_FIELD(onConflictWhere); READ_UINT_FIELD(exclRelRTI); READ_NODE_FIELD(exclRelTlist); + READ_NODE_FIELD(mergeSourceTargetList); + READ_NODE_FIELD(mergeActionList); READ_DONE(); } diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 8b4f031d96..b4d376b4ba 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -282,9 +282,13 @@ static ModifyTable *make_modifytable(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subplans, + List *resultRelations, + Index mergeTargetRelation, + List *subplans, List *withCheckOptionLists, List *returningLists, - List *rowMarks, OnConflictExpr *onconflict, int epqParam); + List *rowMarks, OnConflictExpr *onconflict, + List *mergeSourceTargetList, + List *mergeActionList, int epqParam); static GatherMerge *create_gather_merge_plan(PlannerInfo *root, GatherMergePath *best_path); @@ -2394,11 +2398,14 @@ create_modifytable_plan(PlannerInfo *root, ModifyTablePath *best_path) best_path->partitioned_rels, best_path->partColsUpdated, best_path->resultRelations, + best_path->mergeTargetRelation, subplans, best_path->withCheckOptionLists, best_path->returningLists, best_path->rowMarks, best_path->onconflict, + best_path->mergeSourceTargetList, + best_path->mergeActionList, best_path->epqParam); copy_generic_path_info(&plan->plan, &best_path->path); @@ -6465,9 +6472,13 @@ make_modifytable(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subplans, + List *resultRelations, + Index mergeTargetRelation, + List *subplans, List *withCheckOptionLists, List *returningLists, - List *rowMarks, OnConflictExpr *onconflict, int epqParam) + List *rowMarks, OnConflictExpr *onconflict, + List *mergeSourceTargetList, + List *mergeActionList, int epqParam) { ModifyTable *node = makeNode(ModifyTable); List *fdw_private_list; @@ -6493,6 +6504,7 @@ make_modifytable(PlannerInfo *root, node->partitioned_rels = partitioned_rels; node->partColsUpdated = partColsUpdated; node->resultRelations = resultRelations; + node->mergeTargetRelation = mergeTargetRelation; node->resultRelIndex = -1; /* will be set correctly in setrefs.c */ node->rootResultRelIndex = -1; /* will be set correctly in setrefs.c */ node->plans = subplans; @@ -6525,6 +6537,8 @@ make_modifytable(PlannerInfo *root, node->withCheckOptionLists = withCheckOptionLists; node->returningLists = returningLists; node->rowMarks = rowMarks; + node->mergeSourceTargetList = mergeSourceTargetList; + node->mergeActionList = mergeActionList; node->epqParam = epqParam; /* diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 52c21e6870..14972bc9b2 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -790,6 +790,24 @@ subquery_planner(PlannerGlobal *glob, Query *parse, /* exclRelTlist contains only Vars, so no preprocessing needed */ } + foreach(l, parse->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + action->targetList = (List *) + preprocess_expression(root, + (Node *) action->targetList, + EXPRKIND_TARGET); + action->qual = + preprocess_expression(root, + (Node *) action->qual, + EXPRKIND_QUAL); + } + + parse->mergeSourceTargetList = (List *) + preprocess_expression(root, (Node *) parse->mergeSourceTargetList, + EXPRKIND_TARGET); + root->append_rel_list = (List *) preprocess_expression(root, (Node *) root->append_rel_list, EXPRKIND_APPINFO); @@ -1531,6 +1549,7 @@ inheritance_planner(PlannerInfo *root) subroot->parse->returningList); Assert(!parse->onConflict); + Assert(parse->mergeActionList == NIL); } /* Result path must go into outer query's FINAL upperrel */ @@ -1589,12 +1608,15 @@ inheritance_planner(PlannerInfo *root) partitioned_rels, partColsUpdated, resultRelations, + 0, subpaths, subroots, withCheckOptionLists, returningLists, rowMarks, NULL, + NULL, + NULL, SS_assign_special_param(root))); } @@ -2128,8 +2150,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, } /* - * If this is an INSERT/UPDATE/DELETE, and we're not being called from - * inheritance_planner, add the ModifyTable node. + * If this is an INSERT/UPDATE/DELETE/MERGE, and we're not being + * called from inheritance_planner, add the ModifyTable node. */ if (parse->commandType != CMD_SELECT && !inheritance_update) { @@ -2169,12 +2191,15 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, NIL, false, list_make1_int(parse->resultRelation), + parse->mergeTarget_relation, list_make1(path), list_make1(root), withCheckOptionLists, returningLists, rowMarks, parse->onConflict, + parse->mergeSourceTargetList, + parse->mergeActionList, SS_assign_special_param(root)); } diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index 69dd327f0c..d30889ec7c 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -851,9 +851,68 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) fix_scan_list(root, splan->exclRelTlist, rtoffset); } + /* + * The MERGE produces the target rows by performing a right + * join between the target relation and the source relation + * (which could be a plain relation or a subquery). The INSERT + * and UPDATE actions of the MERGE requires access to the + * columns from the source relation. We arrange things so that + * the source relation attributes are available as INNER_VAR + * and the target relation attributes are available from the + * scan tuple. + */ + if (splan->mergeActionList != NIL) + { + /* + * mergeSourceTargetList is already setup correctly to + * include all Vars coming from the source relation. So we + * fix the targetList of individual action nodes by + * ensuring that the source relation Vars are referenced + * as INNER_VAR. Note that for this to work correctly, + * during execution, the ecxt_innertuple must be set to + * the tuple obtained from the source relation. + * + * We leave the Vars from the result relation (i.e. the + * target relation) unchanged i.e. those Vars would be + * picked from the scan slot. So during execution, we must + * ensure that ecxt_scantuple is setup correctly to refer + * to the tuple from the target relation. + */ + + indexed_tlist *itlist; + + itlist = build_tlist_index(splan->mergeSourceTargetList); + + foreach(l, splan->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + /* Fix targetList of each action. */ + action->targetList = fix_join_expr(root, + action->targetList, + NULL, itlist, + linitial_int(splan->resultRelations), + rtoffset); + + /* Fix quals too. */ + action->qual = (Node *) fix_join_expr(root, + (List *) action->qual, + NULL, itlist, + linitial_int(splan->resultRelations), + rtoffset); + } + } + splan->nominalRelation += rtoffset; splan->exclRelRTI += rtoffset; + /* + * Don't mess-up with mergeTargetRelation if it's set to zero + * i.e. an invalid value. + */ + if (splan->mergeTargetRelation > 0) + splan->mergeTargetRelation += rtoffset; + foreach(l, splan->partitioned_rels) { lfirst_int(l) += rtoffset; diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8603feef2b..4a864b2340 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -105,9 +105,13 @@ preprocess_targetlist(PlannerInfo *root) * scribbles on parse->targetList, which is not very desirable, but we * keep it that way to avoid changing APIs used by FDWs. */ - if (command_type == CMD_UPDATE || command_type == CMD_DELETE) + if (command_type == CMD_UPDATE || + command_type == CMD_DELETE) rewriteTargetListUD(parse, target_rte, target_relation); + if (command_type == CMD_MERGE) + rewriteTargetListMerge(parse, target_relation); + /* * for heap_form_tuple to work, the targetlist must match the exact order * of the attributes. We also need to fill in any missing attributes. -ay @@ -118,6 +122,39 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE command, handle targetlist of each MergeAction separately. We + * give the same treatment to MergeAction->targetList as we would have + * given to a regular INSERT/UPDATE/DELETE. + */ + if (command_type == CMD_MERGE) + { + ListCell *l; + + foreach(l, parse->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + switch (action->commandType) + { + case CMD_INSERT: + case CMD_UPDATE: + action->targetList = expand_targetlist(action->targetList, + action->commandType, + result_relation, + target_relation); + break; + case CMD_DELETE: + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + + } + } + } + /* * Add necessary junk columns for rowmarked rels. These values are needed * for locking of rels selected FOR UPDATE/SHARE, and to do EvalPlanQual @@ -348,6 +385,7 @@ expand_targetlist(List *tlist, int command_type, true /* byval */ ); } break; + case CMD_MERGE: case CMD_UPDATE: if (!att_tup->attisdropped) { diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 22133fcf12..416b3f9578 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -3284,17 +3284,21 @@ create_lockrows_path(PlannerInfo *root, RelOptInfo *rel, * 'rowMarks' is a list of PlanRowMarks (non-locking only) * 'onconflict' is the ON CONFLICT clause, or NULL * 'epqParam' is the ID of Param for EvalPlanQual re-eval + * 'mergeActionList' is a list of MERGE actions */ ModifyTablePath * create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subpaths, + List *resultRelations, + Index mergeTargetRelation, + List *subpaths, List *subroots, List *withCheckOptionLists, List *returningLists, List *rowMarks, OnConflictExpr *onconflict, - int epqParam) + List *mergeSourceTargetList, + List *mergeActionList, int epqParam) { ModifyTablePath *pathnode = makeNode(ModifyTablePath); double total_size; @@ -3359,6 +3363,7 @@ create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, pathnode->partitioned_rels = list_copy(partitioned_rels); pathnode->partColsUpdated = partColsUpdated; pathnode->resultRelations = resultRelations; + pathnode->mergeTargetRelation = mergeTargetRelation; pathnode->subpaths = subpaths; pathnode->subroots = subroots; pathnode->withCheckOptionLists = withCheckOptionLists; @@ -3366,6 +3371,8 @@ create_modifytable_path(PlannerInfo *root, RelOptInfo *rel, pathnode->rowMarks = rowMarks; pathnode->onconflict = onconflict; pathnode->epqParam = epqParam; + pathnode->mergeSourceTargetList = mergeSourceTargetList; + pathnode->mergeActionList = mergeActionList; return pathnode; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 0231f8bf7c..8a6baa7bea 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1835,6 +1835,10 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) trigDesc->trig_delete_before_row)) result = true; break; + /* There is no separate event for MERGE, only INSERT/UPDATE/DELETE */ + case CMD_MERGE: + result = false; + break; default: elog(ERROR, "unrecognized CmdType: %d", (int) event); break; diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index da8f0f93fc..901cf24e20 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -1237,7 +1237,6 @@ find_childrel_parents(PlannerInfo *root, RelOptInfo *rel) return result; } - /* * get_baserel_parampathinfo * Get the ParamPathInfo for a parameterized path for a base relation, diff --git a/src/backend/parser/Makefile b/src/backend/parser/Makefile index f14febdbda..95fdf0b973 100644 --- a/src/backend/parser/Makefile +++ b/src/backend/parser/Makefile @@ -14,7 +14,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS) OBJS= analyze.o gram.o scan.o parser.o \ parse_agg.o parse_clause.o parse_coerce.o parse_collate.o parse_cte.o \ - parse_enr.o parse_expr.o parse_func.o parse_node.o parse_oper.o \ + parse_enr.o parse_expr.o parse_func.o parse_merge.o parse_node.o parse_oper.o \ parse_param.o parse_relation.o parse_target.o parse_type.o \ parse_utilcmd.o scansup.o diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index a4b5aaef44..7eb9544efe 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -38,6 +38,7 @@ #include "parser/parse_cte.h" #include "parser/parse_expr.h" #include "parser/parse_func.h" +#include "parser/parse_merge.h" #include "parser/parse_oper.h" #include "parser/parse_param.h" #include "parser/parse_relation.h" @@ -53,9 +54,6 @@ post_parse_analyze_hook_type post_parse_analyze_hook = NULL; static Query *transformOptionalSelectInto(ParseState *pstate, Node *parseTree); static Query *transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt); static Query *transformInsertStmt(ParseState *pstate, InsertStmt *stmt); -static List *transformInsertRow(ParseState *pstate, List *exprlist, - List *stmtcols, List *icolumns, List *attrnos, - bool strip_indirection); static OnConflictExpr *transformOnConflictClause(ParseState *pstate, OnConflictClause *onConflictClause); static int count_rowexpr_columns(ParseState *pstate, Node *expr); @@ -68,8 +66,6 @@ static void determineRecursiveColTypes(ParseState *pstate, Node *larg, List *nrtargetlist); static Query *transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt); static List *transformReturningList(ParseState *pstate, List *returningList); -static List *transformUpdateTargetList(ParseState *pstate, - List *targetList); static Query *transformDeclareCursorStmt(ParseState *pstate, DeclareCursorStmt *stmt); static Query *transformExplainStmt(ParseState *pstate, @@ -267,6 +263,7 @@ transformStmt(ParseState *pstate, Node *parseTree) case T_InsertStmt: case T_UpdateStmt: case T_DeleteStmt: + case T_MergeStmt: (void) test_raw_expression_coverage(parseTree, NULL); break; default: @@ -291,6 +288,10 @@ transformStmt(ParseState *pstate, Node *parseTree) result = transformUpdateStmt(pstate, (UpdateStmt *) parseTree); break; + case T_MergeStmt: + result = transformMergeStmt(pstate, (MergeStmt *) parseTree); + break; + case T_SelectStmt: { SelectStmt *n = (SelectStmt *) parseTree; @@ -366,6 +367,7 @@ analyze_requires_snapshot(RawStmt *parseTree) case T_InsertStmt: case T_DeleteStmt: case T_UpdateStmt: + case T_MergeStmt: case T_SelectStmt: result = true; break; @@ -896,7 +898,7 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) * attrnos: integer column numbers (must be same length as icolumns) * strip_indirection: if true, remove any field/array assignment nodes */ -static List * +List * transformInsertRow(ParseState *pstate, List *exprlist, List *stmtcols, List *icolumns, List *attrnos, bool strip_indirection) @@ -2260,9 +2262,9 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt) /* * transformUpdateTargetList - - * handle SET clause in UPDATE/INSERT ... ON CONFLICT UPDATE + * handle SET clause in UPDATE/MERGE/INSERT ... ON CONFLICT UPDATE */ -static List * +List * transformUpdateTargetList(ParseState *pstate, List *origTlist) { List *tlist = NIL; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index cd5ba2d4d8..ebca5f3eb7 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -282,6 +282,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CreateMatViewStmt RefreshMatViewStmt CreateAmStmt CreatePublicationStmt AlterPublicationStmt CreateSubscriptionStmt AlterSubscriptionStmt DropSubscriptionStmt + MergeStmt %type <node> select_no_parens select_with_parens select_clause simple_select values_clause @@ -584,6 +585,10 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound partbound_datum_list range_datum_list %type <defelt> hash_partbound_elem +%type <node> merge_when_clause opt_and_condition +%type <list> merge_when_list +%type <node> merge_update merge_delete merge_insert + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -651,7 +656,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED - MAPPING MATCH MATERIALIZED MAXVALUE METHOD MINUTE_P MINVALUE MODE MONTH_P MOVE + MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MERGE METHOD + MINUTE_P MINVALUE MODE MONTH_P MOVE NAME_P NAMES NATIONAL NATURAL NCHAR NEW NEXT NO NONE NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF @@ -920,6 +926,7 @@ stmt : | RefreshMatViewStmt | LoadStmt | LockStmt + | MergeStmt | NotifyStmt | PrepareStmt | ReassignOwnedStmt @@ -10660,6 +10667,7 @@ ExplainableStmt: | InsertStmt | UpdateStmt | DeleteStmt + | MergeStmt | DeclareCursorStmt | CreateAsStmt | CreateMatViewStmt @@ -10722,6 +10730,7 @@ PreparableStmt: | InsertStmt | UpdateStmt | DeleteStmt /* by default all are $$=$1 */ + | MergeStmt ; /***************************************************************************** @@ -11088,6 +11097,151 @@ set_target_list: ; +/***************************************************************************** + * + * QUERY: + * MERGE STATEMENT + * + *****************************************************************************/ + +MergeStmt: + MERGE INTO relation_expr_opt_alias + USING table_ref + ON a_expr + merge_when_list + { + MergeStmt *m = makeNode(MergeStmt); + + m->relation = $3; + m->source_relation = $5; + m->join_condition = $7; + m->mergeActionList = $8; + + $$ = (Node *)m; + } + ; + + +merge_when_list: + merge_when_clause { $$ = list_make1($1); } + | merge_when_list merge_when_clause { $$ = lappend($1,$2); } + ; + +merge_when_clause: + WHEN MATCHED opt_and_condition THEN merge_update + { + MergeAction *m = makeNode(MergeAction); + + m->matched = true; + m->commandType = CMD_UPDATE; + m->condition = $3; + m->stmt = $5; + + $$ = (Node *)m; + } + | WHEN MATCHED opt_and_condition THEN merge_delete + { + MergeAction *m = makeNode(MergeAction); + + m->matched = true; + m->commandType = CMD_DELETE; + m->condition = $3; + m->stmt = $5; + + $$ = (Node *)m; + } + | WHEN NOT MATCHED opt_and_condition THEN merge_insert + { + MergeAction *m = makeNode(MergeAction); + + m->matched = false; + m->commandType = CMD_INSERT; + m->condition = $4; + m->stmt = $6; + + $$ = (Node *)m; + } + | WHEN NOT MATCHED opt_and_condition THEN DO NOTHING + { + MergeAction *m = makeNode(MergeAction); + + m->matched = false; + m->commandType = CMD_NOTHING; + m->condition = $4; + m->stmt = NULL; + + $$ = (Node *)m; + } + ; + +opt_and_condition: + AND a_expr { $$ = $2; } + | { $$ = NULL; } + ; + +merge_delete: + DELETE_P + { + DeleteStmt *n = makeNode(DeleteStmt); + $$ = (Node *)n; + } + ; + +merge_update: + UPDATE SET set_clause_list + { + UpdateStmt *n = makeNode(UpdateStmt); + n->targetList = $3; + + $$ = (Node *)n; + } + ; + +merge_insert: + INSERT values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->selectStmt = $2; + + $$ = (Node *)n; + } + | INSERT OVERRIDING override_kind VALUE_P values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->override = $3; + n->selectStmt = $5; + + $$ = (Node *)n; + } + | INSERT '(' insert_column_list ')' values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = $3; + n->selectStmt = $5; + + $$ = (Node *)n; + } + | INSERT '(' insert_column_list ')' OVERRIDING override_kind VALUE_P values_clause + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = $3; + n->override = $6; + n->selectStmt = $8; + + $$ = (Node *)n; + } + | INSERT DEFAULT VALUES + { + InsertStmt *n = makeNode(InsertStmt); + n->cols = NIL; + n->selectStmt = NULL; + + $$ = (Node *)n; + } + ; + /***************************************************************************** * * QUERY: @@ -15088,8 +15242,10 @@ unreserved_keyword: | LOGGED | MAPPING | MATCH + | MATCHED | MATERIALIZED | MAXVALUE + | MERGE | METHOD | MINUTE_P | MINVALUE diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 377a7ed6d0..544e7300b8 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -455,6 +455,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) case EXPR_KIND_VALUES_SINGLE: errkind = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + if (isAgg) + err = _("aggregate functions are not allowed in WHEN AND conditions"); + else + err = _("grouping operations are not allowed in WHEN AND conditions"); + + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: if (isAgg) @@ -873,6 +880,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_VALUES_SINGLE: errkind = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + err = _("window functions are not allowed in WHEN AND conditions"); + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: err = _("window functions are not allowed in check constraints"); diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 3a02307bd9..9df9828df8 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -76,9 +76,6 @@ static RangeTblEntry *transformRangeTableFunc(ParseState *pstate, RangeTableFunc *t); static TableSampleClause *transformRangeTableSample(ParseState *pstate, RangeTableSample *rts); -static Node *transformFromClauseItem(ParseState *pstate, Node *n, - RangeTblEntry **top_rte, int *top_rti, - List **namespace); static Node *buildMergedJoinVar(ParseState *pstate, JoinType jointype, Var *l_colvar, Var *r_colvar); static ParseNamespaceItem *makeNamespaceItem(RangeTblEntry *rte, @@ -139,6 +136,7 @@ transformFromClause(ParseState *pstate, List *frmList) n = transformFromClauseItem(pstate, n, &rte, &rtindex, + NULL, NULL, &namespace); checkNameSpaceConflicts(pstate, pstate->p_namespace, namespace); @@ -1100,9 +1098,10 @@ getRTEForSpecialRelationTypes(ParseState *pstate, RangeVar *rv) * as table/column names by this item. (The lateral_only flags in these items * are indeterminate and should be explicitly set by the caller before use.) */ -static Node * +Node * transformFromClauseItem(ParseState *pstate, Node *n, RangeTblEntry **top_rte, int *top_rti, + RangeTblEntry **right_rte, int *right_rti, List **namespace) { if (IsA(n, RangeVar)) @@ -1194,7 +1193,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, /* Recursively transform the contained relation */ rel = transformFromClauseItem(pstate, rts->relation, - top_rte, top_rti, namespace); + top_rte, top_rti, NULL, NULL, namespace); /* Currently, grammar could only return a RangeVar as contained rel */ rtr = castNode(RangeTblRef, rel); rte = rt_fetch(rtr->rtindex, pstate->p_rtable); @@ -1222,6 +1221,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, List *l_namespace, *r_namespace, *my_namespace, + *save_namespace, *l_colnames, *r_colnames, *res_colnames, @@ -1240,6 +1240,7 @@ transformFromClauseItem(ParseState *pstate, Node *n, j->larg = transformFromClauseItem(pstate, j->larg, &l_rte, &l_rtindex, + NULL, NULL, &l_namespace); /* @@ -1263,12 +1264,34 @@ transformFromClauseItem(ParseState *pstate, Node *n, sv_namespace_length = list_length(pstate->p_namespace); pstate->p_namespace = list_concat(pstate->p_namespace, l_namespace); + /* + * If we are running MERGE, don't make the other RTEs visible while + * parsing the source relation. It mustn't see them. + * + * XXX Currently, only MERGE passes non-NULL value for right_rte, so we + * can safely deduce if we're running MERGE or not by just looking at + * the right_rte. If that ever changes, we should look at other means + * to find that. + */ + if (right_rte) + { + save_namespace = pstate->p_namespace; + pstate->p_namespace = NIL; + } + /* And now we can process the RHS */ j->rarg = transformFromClauseItem(pstate, j->rarg, &r_rte, &r_rtindex, + NULL, NULL, &r_namespace); + /* + * And now restore the namespace again so that join-quals can see it. + */ + if (right_rte) + pstate->p_namespace = save_namespace; + /* Remove the left-side RTEs from the namespace list again */ pstate->p_namespace = list_truncate(pstate->p_namespace, sv_namespace_length); @@ -1295,6 +1318,12 @@ transformFromClauseItem(ParseState *pstate, Node *n, expandRTE(r_rte, r_rtindex, 0, -1, false, &r_colnames, &r_colvars); + if (right_rte) + *right_rte = r_rte; + + if (right_rti) + *right_rti = r_rtindex; + /* * Natural join does not explicitly specify columns; must generate * columns to join. Need to run through the list of columns from each diff --git a/src/backend/parser/parse_collate.c b/src/backend/parser/parse_collate.c index 6d34245083..51c73c4018 100644 --- a/src/backend/parser/parse_collate.c +++ b/src/backend/parser/parse_collate.c @@ -485,6 +485,7 @@ assign_collations_walker(Node *node, assign_collations_context *context) case T_FromExpr: case T_OnConflictExpr: case T_SortGroupClause: + case T_MergeAction: (void) expression_tree_walker(node, assign_collations_walker, (void *) &loccontext); diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 385e54a9b6..38fbe3366f 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -1818,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_RETURNING: case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: + case EXPR_KIND_MERGE_WHEN_AND: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3475,6 +3476,8 @@ ParseExprKindName(ParseExprKind exprKind) return "PARTITION BY"; case EXPR_KIND_CALL_ARGUMENT: return "CALL"; + case EXPR_KIND_MERGE_WHEN_AND: + return "MERGE WHEN AND"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index ea5d5212b4..615aee6d15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2277,6 +2277,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) /* okay, since we process this like a SELECT tlist */ pstate->p_hasTargetSRFs = true; break; + case EXPR_KIND_MERGE_WHEN_AND: + err = _("set-returning functions are not allowed in WHEN AND conditions"); + break; case EXPR_KIND_CHECK_CONSTRAINT: case EXPR_KIND_DOMAIN_CHECK: err = _("set-returning functions are not allowed in check constraints"); diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c new file mode 100644 index 0000000000..b6e0c46656 --- /dev/null +++ b/src/backend/parser/parse_merge.c @@ -0,0 +1,670 @@ +/*------------------------------------------------------------------------- + * + * parse_merge.c + * handle merge-statement in parser + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/parser/parse_merge.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "miscadmin.h" + +#include "access/sysattr.h" +#include "nodes/makefuncs.h" +#include "parser/analyze.h" +#include "parser/parse_collate.h" +#include "parser/parsetree.h" +#include "parser/parser.h" +#include "parser/parse_clause.h" +#include "parser/parse_merge.h" +#include "parser/parse_relation.h" +#include "parser/parse_target.h" +#include "utils/rel.h" +#include "utils/relcache.h" + +static int transformMergeJoinClause(ParseState *pstate, Node *merge, + List **mergeSourceTargetList); +static void setNamespaceForMergeAction(ParseState *pstate, + MergeAction *action); +static void setNamespaceVisibilityForRTE(List *namespace, RangeTblEntry *rte, + bool rel_visible, + bool cols_visible); +static List *expandSourceTL(ParseState *pstate, RangeTblEntry *rte, + int rtindex); + +/* + * Special handling for MERGE statement is required because we assemble + * the query manually. This is similar to setTargetTable() followed + * by transformFromClause() but with a few less steps. + * + * Process the FROM clause and add items to the query's range table, + * joinlist, and namespace. + * + * A special targetlist comprising of the columns from the right-subtree of + * the join is populated and returned. Note that when the JoinExpr is + * setup by transformMergeStmt, the left subtree has the target result + * relation and the right subtree has the source relation. + * + * Returns the rangetable index of the target relation. + */ +static int +transformMergeJoinClause(ParseState *pstate, Node *merge, + List **mergeSourceTargetList) +{ + RangeTblEntry *rte, + *rt_rte; + List *namespace; + int rtindex, + rt_rtindex; + Node *n; + int mergeTarget_relation = list_length(pstate->p_rtable) + 1; + Var *var; + TargetEntry *te; + + n = transformFromClauseItem(pstate, merge, + &rte, + &rtindex, + &rt_rte, + &rt_rtindex, + &namespace); + + pstate->p_joinlist = list_make1(n); + + /* + * We created an internal join between the target and the source relation + * to carry out the MERGE actions. Normally such an unaliased join hides + * the joining relations, unless the column references are qualified. + * Also, any unqualified column refernces are resolved to the Join RTE, if + * there is a matching entry in the targetlist. But the way MERGE + * execution is later setup, we expect all column references to resolve to + * either the source or the target relation. Hence we must not add the + * Join RTE to the namespace. + * + * The last entry must be for the top-level Join RTE. We don't want to + * resolve any references to the Join RTE. So discard that. + * + * We also do not want to resolve any references from the leftside of the + * Join since that corresponds to the target relation. References to the + * columns of the target relation must be resolved from the result + * relation and not the one that is used in the join. So the + * mergeTarget_relation is marked invisible to both qualified as well as + * unqualified references. + */ + Assert(list_length(namespace) > 1); + namespace = list_truncate(namespace, list_length(namespace) - 1); + pstate->p_namespace = list_concat(pstate->p_namespace, namespace); + + setNamespaceVisibilityForRTE(pstate->p_namespace, + rt_fetch(mergeTarget_relation, pstate->p_rtable), false, false); + + /* + * Expand the right relation and add its columns to the + * mergeSourceTargetList. Note that the right relation can either be a + * plain relation or a subquery or anything that can have a + * RangeTableEntry. + */ + *mergeSourceTargetList = expandSourceTL(pstate, rt_rte, rt_rtindex); + + /* + * Add a whole-row-Var entry to support references to "source.*". + */ + var = makeWholeRowVar(rt_rte, rt_rtindex, 0, false); + te = makeTargetEntry((Expr *) var, list_length(*mergeSourceTargetList) + 1, + NULL, true); + *mergeSourceTargetList = lappend(*mergeSourceTargetList, te); + + return mergeTarget_relation; +} + +/* + * Make appropriate changes to the namespace visibility while transforming + * individual action's quals and targetlist expressions. In particular, for + * INSERT actions we must only see the source relation (since INSERT action is + * invoked for NOT MATCHED tuples and hence there is no target tuple to deal + * with). On the other hand, UPDATE and DELETE actions can see both source and + * target relations. + * + * Also, since the internal Join node can hide the source and target + * relations, we must explicitly make the respective relation as visible so + * that columns can be referenced unqualified from these relations. + */ +static void +setNamespaceForMergeAction(ParseState *pstate, MergeAction *action) +{ + RangeTblEntry *targetRelRTE, + *sourceRelRTE; + + /* Assume target relation is at index 1 */ + targetRelRTE = rt_fetch(1, pstate->p_rtable); + + /* + * Assume that the top-level join RTE is at the end. The source relation + * is just before that. + */ + sourceRelRTE = rt_fetch(list_length(pstate->p_rtable) - 1, pstate->p_rtable); + + switch (action->commandType) + { + case CMD_INSERT: + + /* + * Inserts can't see target relation, but they can see source + * relation. + */ + setNamespaceVisibilityForRTE(pstate->p_namespace, + targetRelRTE, false, false); + setNamespaceVisibilityForRTE(pstate->p_namespace, + sourceRelRTE, true, true); + break; + + case CMD_UPDATE: + case CMD_DELETE: + + /* + * Updates and deletes can see both target and source relations. + */ + setNamespaceVisibilityForRTE(pstate->p_namespace, + targetRelRTE, true, true); + setNamespaceVisibilityForRTE(pstate->p_namespace, + sourceRelRTE, true, true); + break; + + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } +} + +/* + * transformMergeStmt - + * transforms a MERGE statement + */ +Query * +transformMergeStmt(ParseState *pstate, MergeStmt *stmt) +{ + Query *qry = makeNode(Query); + ListCell *l; + AclMode targetPerms = ACL_NO_RIGHTS; + bool is_terminal[2]; + JoinExpr *joinexpr; + RangeTblEntry *resultRelRTE, *mergeRelRTE; + + /* There can't be any outer WITH to worry about */ + Assert(pstate->p_ctenamespace == NIL); + + qry->commandType = CMD_MERGE; + + /* + * Check WHEN clauses for permissions and sanity + */ + is_terminal[0] = false; + is_terminal[1] = false; + foreach(l, stmt->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + uint when_type = (action->matched ? 0 : 1); + + /* + * Collect action types so we can check Target permissions + */ + switch (action->commandType) + { + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + SelectStmt *selectStmt = (SelectStmt *) istmt->selectStmt; + + /* + * The grammar allows attaching ORDER BY, LIMIT, FOR + * UPDATE, or WITH to a VALUES clause and also multiple + * VALUES clauses. If we have any of those, ERROR. + */ + if (selectStmt && (selectStmt->valuesLists == NIL || + selectStmt->sortClause != NIL || + selectStmt->limitOffset != NULL || + selectStmt->limitCount != NULL || + selectStmt->lockingClause != NIL || + selectStmt->withClause != NULL)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("SELECT not allowed in MERGE INSERT statement"))); + + if (selectStmt && list_length(selectStmt->valuesLists) > 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("Multiple VALUES clauses not allowed in MERGE INSERT statement"))); + + targetPerms |= ACL_INSERT; + } + break; + case CMD_UPDATE: + targetPerms |= ACL_UPDATE; + break; + case CMD_DELETE: + targetPerms |= ACL_DELETE; + break; + case CMD_NOTHING: + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } + + /* + * Check for unreachable WHEN clauses + */ + if (action->condition == NULL) + is_terminal[when_type] = true; + else if (is_terminal[when_type]) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unreachable WHEN clause specified after unconditional WHEN clause"))); + } + + /* + * Construct a query of the form SELECT relation.ctid --junk attribute + * ,relation.tableoid --junk attribute ,source_relation.<somecols> + * ,relation.<somecols> FROM relation RIGHT JOIN source_relation ON + * join_condition -- no WHERE clause - all conditions are applied in + * executor + * + * stmt->relation is the target relation, given as a RangeVar + * stmt->source_relation is a RangeVar or subquery + * + * We specify the join as a RIGHT JOIN as a simple way of forcing the + * first (larg) RTE to refer to the target table. + * + * The MERGE query's join can be tuned in some cases, see below for these + * special case tweaks. + * + * We set QSRC_PARSER to show query constructed in parse analysis + * + * Note that we have only one Query for a MERGE statement and the planner + * is called only once. That query is executed once to produce our stream + * of candidate change rows, so the query must contain all of the columns + * required by each of the targetlist or conditions for each action. + * + * As top-level statements INSERT, UPDATE and DELETE have a Query, whereas + * with MERGE the individual actions do not require separate planning, + * only different handling in the executor. See nodeModifyTable handling + * of commandType CMD_MERGE. + * + * A sub-query can include the Target, but otherwise the sub-query cannot + * reference the outermost Target table at all. + */ + qry->querySource = QSRC_PARSER; + + /* + * Setup the target table. Unlike regular UPDATE/DELETE, we don't expand + * inheritance for the target relation in case of MERGE. + * + * This special arrangement is required for handling partitioned tables + * because we perform an JOIN between the target and the source relation to + * identify the matching and not-matching rows. If we take the usual path + * of expanding the target table's inheritance and create one subplan per + * partition, then we we won't be able to correctly identify the matching + * and not-matching rows since for a given source row, there may not be a + * matching row in one partition, but it may exists in some other + * partition. So we must first append all the qualifying rows from all the + * partitions and then do the matching. + * + * Once a target row is returned by the underlying join, we find the + * correct partition and setup required state to carry out UPDATE/DELETE. + * All of this happens during execution. + */ + qry->resultRelation = setTargetTable(pstate, stmt->relation, + false, /* do not expand inheritance */ + true, targetPerms); + + /* + * Create a JOIN between the target and the source relation. + */ + joinexpr = makeNode(JoinExpr); + joinexpr->isNatural = false; + joinexpr->alias = NULL; + joinexpr->usingClause = NIL; + joinexpr->quals = stmt->join_condition; + joinexpr->larg = (Node *) stmt->relation; + joinexpr->rarg = (Node *) stmt->source_relation; + + /* + * Simplify the MERGE query as much as possible + * + * These seem like things that could go into Optimizer, but they are + * semantic simplications rather than optimizations, per se. + * + * If there are no INSERT actions we won't be using the non-matching + * candidate rows for anything, so no need for an outer join. We do still + * need an inner join for UPDATE and DELETE actions. + * + * Possible additional simplifications... + * + * XXX if we have a constant ON clause, we can skip join altogether + * + * XXX if we have a constant subquery, we can also skip join + * + * XXX if we were really keen we could look through the actionList and + * pull out common conditions, if there were no terminal clauses and put + * them into the main query as an early row filter but that seems like an + * atypical case and so checking for it would be likely to just be wasted + * effort. + */ + if (targetPerms & ACL_INSERT) + joinexpr->jointype = JOIN_RIGHT; + else + joinexpr->jointype = JOIN_INNER; + + /* + * We use a special purpose transformation here because the normal + * routines don't quite work right for the MERGE case. + * + * A special mergeSourceTargetList is setup by transformMergeJoinClause(). + * It refers to all the attributes provided by the source relation. This + * is later used by set_plan_refs() to fix the UPDATE/INSERT target lists + * to so that they can correctly fetch the attributes from the source + * relation. + * + * The target relation when used in the underlying join, gets a new RTE + * with rte->inh set to true. We remember this RTE (and later pass on to + * the planner and executor) for two main reasons: + * + * 1. If we ever need to run EvalPlanQual while performing MERGE, we must + * make the modified tuple available to the underlying join query, which is + * using a different RTE from the resultRelation RTE. + * + * 2. rewriteTargetListMerge() requires the RTE of the underlying join in + * order to add junk CTID and TABLEOID attributes. + */ + qry->mergeTarget_relation = transformMergeJoinClause(pstate, (Node *) joinexpr, + &qry->mergeSourceTargetList); + + /* + * The target table referenced in the MERGE is looked up twice; once while + * setting it up as the result relation and again when it's used in the + * underlying the join query. In some rare situations, it may happen that + * these lookups return different results, for example, if a new relation + * with the same name gets created in a schema which is ahead in the + * search_path, in between the two lookups. + * + * It's a very narrow case, but nevertheless we guard against it by simply + * checking if the OIDs returned by the two lookups is the same. If not, we + * just throw an error. + */ + Assert(qry->resultRelation > 0); + Assert(qry->mergeTarget_relation > 0); + + /* Fetch both the RTEs */ + resultRelRTE = rt_fetch(qry->resultRelation, pstate->p_rtable); + mergeRelRTE = rt_fetch(qry->mergeTarget_relation, pstate->p_rtable); + + if (resultRelRTE->relid != mergeRelRTE->relid) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("relation referenced by MERGE statement has changed"))); + + /* + * This query should just provide the source relation columns. Later, in + * preprocess_targetlist(), we shall also add "ctid" attribute of the + * target relation to ensure that the target tuple can be fetched + * correctly. + */ + qry->targetList = qry->mergeSourceTargetList; + + /* qry has no WHERE clause so absent quals are shown as NULL */ + qry->jointree = makeFromExpr(pstate->p_joinlist, NULL); + qry->rtable = pstate->p_rtable; + + /* + * XXX MERGE is unsupported in various cases + */ + if (!(pstate->p_target_relation->rd_rel->relkind == RELKIND_RELATION || + pstate->p_target_relation->rd_rel->relkind == RELKIND_MATVIEW || + pstate->p_target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for this relation type"))); + + if (pstate->p_target_relation->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && + pstate->p_target_relation->rd_rel->relhassubclass) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for relations with inheritance"))); + + if (pstate->p_target_relation->rd_rel->relhasrules) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("MERGE is not supported for relations with rules"))); + + /* + * We now have a good query shape, so now look at the when conditions and + * action targetlists. + * + * Overall, the MERGE Query's targetlist is NIL. + * + * Each individual action has its own targetlist that needs separate + * transformation. These transforms don't do anything to the overall + * targetlist, since that is only used for resjunk columns. + * + * We can reference any column in Target or Source, which is OK because + * both of those already have RTEs. There is nothing like the EXCLUDED + * pseudo-relation for INSERT ON CONFLICT. + */ + foreach(l, stmt->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(l); + + /* + * Set namespace for the specific action. This must be done before + * analysing the WHEN quals and the action targetlisst. + */ + setNamespaceForMergeAction(pstate, action); + + /* + * Transform the when condition. + * + * Note that these quals are NOT added to the join quals; instead they + * are evaluated sepaartely during execution to decide which of the + * WHEN MATCHED or WHEN NOT MATCHED actions to execute. + */ + action->qual = transformWhereClause(pstate, action->condition, + EXPR_KIND_MERGE_WHEN_AND, "WHEN"); + + /* + * Transform target lists for each INSERT and UPDATE action stmt + */ + switch (action->commandType) + { + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + SelectStmt *selectStmt = (SelectStmt *) istmt->selectStmt; + List *exprList = NIL; + ListCell *lc; + RangeTblEntry *rte; + ListCell *icols; + ListCell *attnos; + List *icolumns; + List *attrnos; + + pstate->p_is_insert = true; + + icolumns = checkInsertTargets(pstate, istmt->cols, &attrnos); + Assert(list_length(icolumns) == list_length(attrnos)); + + /* + * Handle INSERT much like in transformInsertStmt + */ + if (selectStmt == NULL) + { + /* + * We have INSERT ... DEFAULT VALUES. We can handle + * this case by emitting an empty targetlist --- all + * columns will be defaulted when the planner expands + * the targetlist. + */ + exprList = NIL; + } + else + { + /* + * Process INSERT ... VALUES with a single VALUES + * sublist. We treat this case separately for + * efficiency. The sublist is just computed directly + * as the Query's targetlist, with no VALUES RTE. So + * it works just like a SELECT without any FROM. + */ + List *valuesLists = selectStmt->valuesLists; + + Assert(list_length(valuesLists) == 1); + Assert(selectStmt->intoClause == NULL); + + /* + * Do basic expression transformation (same as a ROW() + * expr, but allow SetToDefault at top level) + */ + exprList = transformExpressionList(pstate, + (List *) linitial(valuesLists), + EXPR_KIND_VALUES_SINGLE, + true); + + /* Prepare row for assignment to target table */ + exprList = transformInsertRow(pstate, exprList, + istmt->cols, + icolumns, attrnos, + false); + } + + /* + * Generate action's target list using the computed list + * of expressions. Also, mark all the target columns as + * needing insert permissions. + */ + rte = pstate->p_target_rangetblentry; + icols = list_head(icolumns); + attnos = list_head(attrnos); + foreach(lc, exprList) + { + Expr *expr = (Expr *) lfirst(lc); + ResTarget *col; + AttrNumber attr_num; + TargetEntry *tle; + + col = lfirst_node(ResTarget, icols); + attr_num = (AttrNumber) lfirst_int(attnos); + + tle = makeTargetEntry(expr, + attr_num, + col->name, + false); + action->targetList = lappend(action->targetList, tle); + + rte->insertedCols = bms_add_member(rte->insertedCols, + attr_num - FirstLowInvalidHeapAttributeNumber); + + icols = lnext(icols); + attnos = lnext(attnos); + } + } + break; + case CMD_UPDATE: + { + UpdateStmt *ustmt = (UpdateStmt *) action->stmt; + + pstate->p_is_insert = false; + action->targetList = transformUpdateTargetList(pstate, ustmt->targetList); + } + break; + case CMD_DELETE: + break; + + case CMD_NOTHING: + action->targetList = NIL; + break; + default: + elog(ERROR, "unknown action in MERGE WHEN clause"); + } + } + + qry->mergeActionList = stmt->mergeActionList; + + /* XXX maybe later */ + qry->returningList = NULL; + + qry->hasTargetSRFs = false; + qry->hasSubLinks = pstate->p_hasSubLinks; + + assign_query_collations(pstate, qry); + + return qry; +} + +static void +setNamespaceVisibilityForRTE(List *namespace, RangeTblEntry *rte, + bool rel_visible, + bool cols_visible) +{ + ListCell *lc; + + foreach(lc, namespace) + { + ParseNamespaceItem *nsitem = (ParseNamespaceItem *) lfirst(lc); + + if (nsitem->p_rte == rte) + { + nsitem->p_rel_visible = rel_visible; + nsitem->p_cols_visible = cols_visible; + break; + } + } + +} + +/* + * Expand the source relation to include all attributes of this RTE. + * + * This function is very similar to expandRelAttrs except that we don't mark + * columns for SELECT privileges. That will be decided later when we transform + * the action targetlists and the WHEN quals for actual references to the + * source relation. + */ +static List * +expandSourceTL(ParseState *pstate, RangeTblEntry *rte, int rtindex) +{ + List *names, + *vars; + ListCell *name, + *var; + List *te_list = NIL; + + expandRTE(rte, rtindex, 0, -1, false, &names, &vars); + + /* + * Require read access to the table. + */ + rte->requiredPerms |= ACL_SELECT; + + forboth(name, names, var, vars) + { + char *label = strVal(lfirst(name)); + Var *varnode = (Var *) lfirst(var); + TargetEntry *te; + + te = makeTargetEntry((Expr *) varnode, + (AttrNumber) pstate->p_next_resno++, + label, + false); + te_list = lappend(te_list, te); + } + + Assert(name == NULL && var == NULL); /* lists not the same length? */ + + return te_list; +} diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c index 053ae02c9f..5583404e1b 100644 --- a/src/backend/parser/parse_relation.c +++ b/src/backend/parser/parse_relation.c @@ -728,6 +728,16 @@ scanRTEForColumn(ParseState *pstate, RangeTblEntry *rte, const char *colname, colname), parser_errposition(pstate, location))); + /* In MERGE when and condition, no system column is allowed */ + if (pstate->p_expr_kind == EXPR_KIND_MERGE_WHEN_AND && + attnum < InvalidAttrNumber && + !(attnum == TableOidAttributeNumber || attnum == ObjectIdAttributeNumber)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_COLUMN_REFERENCE), + errmsg("system column \"%s\" reference in WHEN AND condition is invalid", + colname), + parser_errposition(pstate, location))); + if (attnum != InvalidAttrNumber) { /* now check to see if column actually is defined */ diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index 361bde4261..8e5661df35 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1377,6 +1377,53 @@ rewriteTargetListUD(Query *parsetree, RangeTblEntry *target_rte, } } +void +rewriteTargetListMerge(Query *parsetree, Relation target_relation) +{ + Var *var = NULL; + const char *attrname; + TargetEntry *tle; + + Assert(target_relation->rd_rel->relkind == RELKIND_RELATION || + target_relation->rd_rel->relkind == RELKIND_MATVIEW || + target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE); + + /* + * Emit CTID so that executor can find the row to update or delete. + */ + var = makeVar(parsetree->mergeTarget_relation, + SelfItemPointerAttributeNumber, + TIDOID, + -1, + InvalidOid, + 0); + + attrname = "ctid"; + tle = makeTargetEntry((Expr *) var, + list_length(parsetree->targetList) + 1, + pstrdup(attrname), + true); + + parsetree->targetList = lappend(parsetree->targetList, tle); + + /* + * Emit TABLEOID so that executor can find the row to update or delete. + */ + var = makeVar(parsetree->mergeTarget_relation, + TableOidAttributeNumber, + OIDOID, + -1, + InvalidOid, + 0); + + attrname = "tableoid"; + tle = makeTargetEntry((Expr *) var, + list_length(parsetree->targetList) + 1, + pstrdup(attrname), + true); + + parsetree->targetList = lappend(parsetree->targetList, tle); +} /* * matchLocks - @@ -3331,6 +3378,7 @@ RewriteQuery(Query *parsetree, List *rewrite_events) } else if (event == CMD_UPDATE) { + Assert(parsetree->override == OVERRIDING_NOT_SET); parsetree->targetList = rewriteTargetListIU(parsetree->targetList, parsetree->commandType, @@ -3338,6 +3386,50 @@ RewriteQuery(Query *parsetree, List *rewrite_events) rt_entry_relation, parsetree->resultRelation, NULL); } + else if (event == CMD_MERGE) + { + Assert(parsetree->override == OVERRIDING_NOT_SET); + + /* + * Rewrite each action targetlist separately + */ + foreach(lc1, parsetree->mergeActionList) + { + MergeAction *action = (MergeAction *) lfirst(lc1); + + switch (action->commandType) + { + case CMD_NOTHING: + case CMD_DELETE: /* Nothing to do here */ + break; + case CMD_UPDATE: + action->targetList = + rewriteTargetListIU(action->targetList, + action->commandType, + parsetree->override, + rt_entry_relation, + parsetree->resultRelation, + NULL); + break; + case CMD_INSERT: + { + InsertStmt *istmt = (InsertStmt *) action->stmt; + + action->targetList = + rewriteTargetListIU(action->targetList, + action->commandType, + istmt->override, + rt_entry_relation, + parsetree->resultRelation, + NULL); + } + break; + default: + elog(ERROR, "unrecognized commandType: %d", action->commandType); + break; + } + } + } else if (event == CMD_DELETE) { /* Nothing to do here */ @@ -3351,13 +3443,19 @@ RewriteQuery(Query *parsetree, List *rewrite_events) locks = matchLocks(event, rt_entry_relation->rd_rules, result_relation, parsetree, &hasUpdate); - product_queries = fireRules(parsetree, - result_relation, - event, - locks, - &instead, - &returning, - &qual_product); + /* + * MERGE doesn't support rules as it's unclear how that could work. + */ + if (event == CMD_MERGE) + product_queries = NIL; + else + product_queries = fireRules(parsetree, + result_relation, + event, + locks, + &instead, + &returning, + &qual_product); /* * If there were no INSTEAD rules, and the target relation is a view diff --git a/src/backend/rewrite/rowsecurity.c b/src/backend/rewrite/rowsecurity.c index ce77a18bc9..6e85886e64 100644 --- a/src/backend/rewrite/rowsecurity.c +++ b/src/backend/rewrite/rowsecurity.c @@ -379,6 +379,95 @@ get_row_security_policies(Query *root, RangeTblEntry *rte, int rt_index, } } + /* + * FOR MERGE, we fetch policies for UPDATE, DELETE and INSERT (and ALL) + * and set them up so that we can enforce the appropriate policy depending + * on the final action we take. + * + * We don't fetch the SELECT policies since they are correctly applied to + * the root->mergeTarget_relation. The target rows are selected after + * joining the mergeTarget_relation and the source relation and hence it's + * enough to apply SELECT policies to the mergeTarget_relation. + * + * We don't push the UPDATE/DELETE USING quals to the RTE because we don't + * really want to apply them while scanning the relation since we don't + * know whether we will be doing a UPDATE or a DELETE at the end. We apply + * the respective policy once we decide the final action on the target + * tuple. + * + * XXX We are setting up USING quals as WITH CHECK. If RLS prohibits + * UPDATE/DELETE on the target row, we shall throw an error instead of + * silently ignoring the row. This is different than how normal + * UPDATE/DELETE works and more in line with INSERT ON CONFLICT DO UPDATE + * handling. + */ + if (commandType == CMD_MERGE) + { + List *merge_permissive_policies; + List *merge_restrictive_policies; + + /* + * Fetch the UPDATE policies and set them up to execute on the + * existing target row before doing UPDATE. + */ + get_policies_for_relation(rel, CMD_UPDATE, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + /* + * WCO_RLS_MERGE_UPDATE_CHECK is used to check UPDATE USING quals on + * the existing target row. + */ + add_with_check_options(rel, rt_index, + WCO_RLS_MERGE_UPDATE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + true); + + /* + * Same with DELETE policies. + */ + get_policies_for_relation(rel, CMD_DELETE, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + add_with_check_options(rel, rt_index, + WCO_RLS_MERGE_DELETE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + true); + + /* + * No special handling is required for INSERT policies. They will be + * checked and enforced during ExecInsert(). But we must add them to + * withCheckOptions. + */ + get_policies_for_relation(rel, CMD_INSERT, user_id, + &merge_permissive_policies, + &merge_restrictive_policies); + + add_with_check_options(rel, rt_index, + WCO_RLS_INSERT_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + false); + + /* Enforce the WITH CHECK clauses of the UPDATE policies */ + add_with_check_options(rel, rt_index, + WCO_RLS_UPDATE_CHECK, + merge_permissive_policies, + merge_restrictive_policies, + withCheckOptions, + hasSubLinks, + false); + } + heap_close(rel, NoLock); /* @@ -438,6 +527,14 @@ get_policies_for_relation(Relation relation, CmdType cmd, Oid user_id, if (policy->polcmd == ACL_DELETE_CHR) cmd_matches = true; break; + case CMD_MERGE: + + /* + * We do not support a separate policy for MERGE command. + * Instead it derives from the policies defined for other + * commands. + */ + break; default: elog(ERROR, "unrecognized policy command type %d", (int) cmd); diff --git a/src/backend/tcop/pquery.c b/src/backend/tcop/pquery.c index 66cc5c35c6..50f852a4aa 100644 --- a/src/backend/tcop/pquery.c +++ b/src/backend/tcop/pquery.c @@ -193,6 +193,11 @@ ProcessQuery(PlannedStmt *plan, "DELETE " UINT64_FORMAT, queryDesc->estate->es_processed); break; + case CMD_MERGE: + snprintf(completionTag, COMPLETION_TAG_BUFSIZE, + "MERGE " UINT64_FORMAT, + queryDesc->estate->es_processed); + break; default: strcpy(completionTag, "???"); break; diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index e144583bd1..2521944b9d 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -110,6 +110,7 @@ CommandIsReadOnly(PlannedStmt *pstmt) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: return false; case CMD_UTILITY: /* For now, treat all utility commands as read/write */ @@ -1832,6 +1833,8 @@ QueryReturnsTuples(Query *parsetree) case CMD_SELECT: /* returns tuples */ return true; + case CMD_MERGE: + return false; case CMD_INSERT: case CMD_UPDATE: case CMD_DELETE: @@ -2076,6 +2079,10 @@ CreateCommandTag(Node *parsetree) tag = "UPDATE"; break; + case T_MergeStmt: + tag = "MERGE"; + break; + case T_SelectStmt: tag = "SELECT"; break; @@ -2819,6 +2826,9 @@ CreateCommandTag(Node *parsetree) case CMD_DELETE: tag = "DELETE"; break; + case CMD_MERGE: + tag = "MERGE"; + break; case CMD_UTILITY: tag = CreateCommandTag(stmt->utilityStmt); break; @@ -2879,6 +2889,9 @@ CreateCommandTag(Node *parsetree) case CMD_DELETE: tag = "DELETE"; break; + case CMD_MERGE: + tag = "MERGE"; + break; case CMD_UTILITY: tag = CreateCommandTag(stmt->utilityStmt); break; @@ -2927,6 +2940,7 @@ GetCommandLogLevel(Node *parsetree) case T_InsertStmt: case T_DeleteStmt: case T_UpdateStmt: + case T_MergeStmt: lev = LOGSTMT_MOD; break; @@ -3366,6 +3380,7 @@ GetCommandLogLevel(Node *parsetree) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: lev = LOGSTMT_MOD; break; @@ -3396,6 +3411,7 @@ GetCommandLogLevel(Node *parsetree) case CMD_UPDATE: case CMD_INSERT: case CMD_DELETE: + case CMD_MERGE: lev = LOGSTMT_MOD; break; diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 4c0256b18a..100174138d 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -67,6 +67,7 @@ typedef enum LockTupleMode */ typedef struct HeapUpdateFailureData { + HTSU_Result result; ItemPointerData ctid; TransactionId xmax; CommandId cmax; diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h index a5b8610fa2..1b79a80310 100644 --- a/src/include/commands/trigger.h +++ b/src/include/commands/trigger.h @@ -206,7 +206,8 @@ extern bool ExecBRDeleteTriggers(EState *estate, EPQState *epqstate, ResultRelInfo *relinfo, ItemPointer tupleid, - HeapTuple fdw_trigtuple); + HeapTuple fdw_trigtuple, + HeapUpdateFailureData *hufdp); extern void ExecARDeleteTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, @@ -225,7 +226,8 @@ extern TupleTableSlot *ExecBRUpdateTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, HeapTuple fdw_trigtuple, - TupleTableSlot *slot); + TupleTableSlot *slot, + HeapUpdateFailureData *hufdp); extern void ExecARUpdateTriggers(EState *estate, ResultRelInfo *relinfo, ItemPointer tupleid, diff --git a/src/include/executor/execPartition.h b/src/include/executor/execPartition.h index 03a599ad57..9f55f6409e 100644 --- a/src/include/executor/execPartition.h +++ b/src/include/executor/execPartition.h @@ -114,6 +114,7 @@ extern int ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, TupleTableSlot *slot, EState *estate); +extern int ExecFindPartitionByOid(PartitionTupleRouting *proute, Oid partoid); extern ResultRelInfo *ExecInitPartitionInfo(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, PartitionTupleRouting *proute, diff --git a/src/include/executor/nodeMerge.h b/src/include/executor/nodeMerge.h new file mode 100644 index 0000000000..c222e9ee65 --- /dev/null +++ b/src/include/executor/nodeMerge.h @@ -0,0 +1,22 @@ +/*------------------------------------------------------------------------- + * + * nodeMerge.h + * + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/executor/nodeMerge.h + * + *------------------------------------------------------------------------- + */ +#ifndef NODEMERGE_H +#define NODEMERGE_H + +#include "nodes/execnodes.h" + +extern void +ExecMerge(ModifyTableState *mtstate, EState *estate, TupleTableSlot *slot, + JunkFilter *junkfilter, ResultRelInfo *resultRelInfo); + +#endif /* NODEMERGE_H */ diff --git a/src/include/executor/nodeModifyTable.h b/src/include/executor/nodeModifyTable.h index 0d7e579e1c..686cfa6171 100644 --- a/src/include/executor/nodeModifyTable.h +++ b/src/include/executor/nodeModifyTable.h @@ -18,5 +18,26 @@ extern ModifyTableState *ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags); extern void ExecEndModifyTable(ModifyTableState *node); extern void ExecReScanModifyTable(ModifyTableState *node); +extern TupleTableSlot *ExecPrepareTupleRouting(ModifyTableState *mtstate, + EState *estate, + struct PartitionTupleRouting *proute, + ResultRelInfo *targetRelInfo, + TupleTableSlot *slot); +extern TupleTableSlot *ExecDelete(ModifyTableState *mtstate, + ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *planSlot, + EPQState *epqstate, EState *estate, bool *tupleDeleted, + bool processReturning, HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag); +extern TupleTableSlot *ExecUpdate(ModifyTableState *mtstate, + ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *slot, + TupleTableSlot *planSlot, EPQState *epqstate, EState *estate, + bool *tuple_updated, HeapUpdateFailureData *hufdp, + MergeActionState *actionState, bool canSetTag); +extern TupleTableSlot *ExecInsert(ModifyTableState *mtstate, + TupleTableSlot *slot, + TupleTableSlot *planSlot, + EState *estate, + MergeActionState *actionState, + bool canSetTag); #endif /* NODEMODIFYTABLE_H */ diff --git a/src/include/executor/spi.h b/src/include/executor/spi.h index e5bdaecc4e..78410b9f77 100644 --- a/src/include/executor/spi.h +++ b/src/include/executor/spi.h @@ -64,6 +64,7 @@ typedef struct _SPI_plan *SPIPlanPtr; #define SPI_OK_REL_REGISTER 15 #define SPI_OK_REL_UNREGISTER 16 #define SPI_OK_TD_REGISTER 17 +#define SPI_OK_MERGE 18 #define SPI_OPT_NONATOMIC (1 << 0) diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 6070a42b6f..0bf1d7aeb6 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -360,8 +360,17 @@ typedef struct JunkFilter AttrNumber *jf_cleanMap; TupleTableSlot *jf_resultSlot; AttrNumber jf_junkAttNo; + AttrNumber jf_otherJunkAttNo; } JunkFilter; +typedef struct MergeState +{ + /* List of MERGE MATCHED action states */ + List *matchedActionStates; + /* List of MERGE NOT MATCHED action states */ + List *notMatchedActionStates; +} MergeState; + /* * OnConflictSetState * @@ -452,8 +461,38 @@ typedef struct ResultRelInfo /* relation descriptor for root partitioned table */ Relation ri_PartitionRoot; + + int ri_PartitionLeafIndex; + /* for running MERGE on this result relation */ + MergeState *ri_mergeState; + + /* + * While executing MERGE, the target relation is processed twice; once to + * as a result relation and to run a join between the target and the + * source. We generate two different RTEs for these two purposes, one with + * rte->inh set to false and other with rte->inh set to true. + * + * Since the plan re-evaluated by EvalPlanQual uses the second RTE, we must + * install the updated tuple in the scan corresponding to that RTE. The + * following member tracks the index of the second RTE for EvalPlanQual + * purposes. ri_mergeTargetRTI is non-zero only when MERGE is in-progress. + * We use ri_mergeTargetRTI to run EvalPlanQual for MERGE and + * ri_RangeTableIndex elsewhere. + */ + Index ri_mergeTargetRTI; } ResultRelInfo; +/* + * Get the Range table index for EvalPlanQual. + * + * We use the ri_mergeTargetRTI if set, otherwise use ri_RangeTableIndex. + * ri_mergeTargetRTI should really be ever set iff we're running MERGE. + */ +#define GetEPQRangeTableIndex(r) \ + (((r)->ri_mergeTargetRTI > 0) \ + ? (r)->ri_mergeTargetRTI \ + : (r)->ri_RangeTableIndex) + /* ---------------- * EState information * @@ -1012,6 +1051,24 @@ typedef struct ProjectSetState MemoryContext argcontext; /* context for SRF arguments */ } ProjectSetState; +/* ---------------- + * MergeActionState information + * ---------------- + */ +typedef struct MergeActionState +{ + NodeTag type; + bool matched; /* true if a WHEN MATCHED action, + * false if a NOT MATCHED action. */ + ExprState *whenqual; /* additional conditions attached to + * WHEN [NOT] MATCHED clause */ + CmdType commandType; /* INSERT/UPDATE/DELETE/DO NOTHING */ + ProjectionInfo *proj; /* projection information for the tuple + * produced by this action */ + TupleDesc tupDesc; /* tuple descriptor associated with the + * projection */ +} MergeActionState; + /* ---------------- * ModifyTableState information * ---------------- @@ -1019,7 +1076,7 @@ typedef struct ProjectSetState typedef struct ModifyTableState { PlanState ps; /* its first field is NodeTag */ - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ bool mt_done; /* are we done? */ PlanState **mt_plans; /* subplans (one per target rel) */ @@ -1035,6 +1092,8 @@ typedef struct ModifyTableState List *mt_excludedtlist; /* the excluded pseudo relation's tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection target */ + TupleTableSlot *mt_mergeproj; /* MERGE action projection target */ + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; @@ -1046,6 +1105,9 @@ typedef struct ModifyTableState /* Per plan map for tuple conversion from child to root */ TupleConversionMap **mt_per_subplan_tupconv_maps; + + int mt_merge_subcommands; /* Flags show which cmd types are + * present */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 443de22704..fce48026b6 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -97,6 +97,7 @@ typedef enum NodeTag T_PlanState, T_ResultState, T_ProjectSetState, + T_MergeActionState, T_ModifyTableState, T_AppendState, T_MergeAppendState, @@ -308,6 +309,8 @@ typedef enum NodeTag T_InsertStmt, T_DeleteStmt, T_UpdateStmt, + T_MergeStmt, + T_MergeAction, T_SelectStmt, T_AlterTableStmt, T_AlterTableCmd, @@ -657,7 +660,8 @@ typedef enum CmdType CMD_SELECT, /* select stmt */ CMD_UPDATE, /* update stmt */ CMD_INSERT, /* insert stmt */ - CMD_DELETE, + CMD_DELETE, /* delete stmt */ + CMD_MERGE, /* merge stmt */ CMD_UTILITY, /* cmds like create, destroy, copy, vacuum, * etc. */ CMD_NOTHING /* dummy command for instead nothing rules diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 92082b3a7a..0c904f4d7f 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -38,7 +38,7 @@ typedef enum OverridingKind typedef enum QuerySource { QSRC_ORIGINAL, /* original parsetree (explicit query) */ - QSRC_PARSER, /* added by parse analysis (now unused) */ + QSRC_PARSER, /* added by parse analysis in MERGE */ QSRC_INSTEAD_RULE, /* added by unconditional INSTEAD rule */ QSRC_QUAL_INSTEAD_RULE, /* added by conditional INSTEAD rule */ QSRC_NON_INSTEAD_RULE /* added by non-INSTEAD rule */ @@ -107,7 +107,7 @@ typedef struct Query { NodeTag type; - CmdType commandType; /* select|insert|update|delete|utility */ + CmdType commandType; /* select|insert|update|delete|merge|utility */ QuerySource querySource; /* where did I come from? */ @@ -118,7 +118,7 @@ typedef struct Query Node *utilityStmt; /* non-null if commandType == CMD_UTILITY */ int resultRelation; /* rtable index of target relation for - * INSERT/UPDATE/DELETE; 0 for SELECT */ + * INSERT/UPDATE/DELETE/MERGE; 0 for SELECT */ bool hasAggs; /* has aggregates in tlist or havingQual */ bool hasWindowFuncs; /* has window functions in tlist */ @@ -169,6 +169,9 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's, which are * only added during rewrite and therefore * are not written out as part of Query. */ + int mergeTarget_relation; + List *mergeSourceTargetList; + List *mergeActionList; /* list of actions for MERGE (only) */ /* * The following two fields identify the portion of the source text string @@ -1128,7 +1131,9 @@ typedef enum WCOKind WCO_VIEW_CHECK, /* WCO on an auto-updatable view */ WCO_RLS_INSERT_CHECK, /* RLS INSERT WITH CHECK policy */ WCO_RLS_UPDATE_CHECK, /* RLS UPDATE WITH CHECK policy */ - WCO_RLS_CONFLICT_CHECK /* RLS ON CONFLICT DO UPDATE USING policy */ + WCO_RLS_CONFLICT_CHECK, /* RLS ON CONFLICT DO UPDATE USING policy */ + WCO_RLS_MERGE_UPDATE_CHECK, /* RLS MERGE UPDATE USING policy */ + WCO_RLS_MERGE_DELETE_CHECK /* RLS MERGE DELETE USING policy */ } WCOKind; typedef struct WithCheckOption @@ -1503,6 +1508,32 @@ typedef struct UpdateStmt WithClause *withClause; /* WITH clause */ } UpdateStmt; +/* ---------------------- + * Merge Statement + * ---------------------- + */ +typedef struct MergeStmt +{ + NodeTag type; + RangeVar *relation; /* target relation to merge */ + Node *source_relation; /* source relation */ + Node *join_condition; /* join condition between source and target */ + List *mergeActionList; /* list of MergeAction(s) */ +} MergeStmt; + +typedef struct MergeAction +{ + NodeTag type; + bool matched; /* true if a WHEN MATCHED action, + * false if a WHEN NOT MATCHED action */ + Node *condition; /* WHEN AND conditions (raw parser) */ + Node *qual; /* transformed WHEN AND conditions */ + CmdType commandType; /* type of action - INSERT/UPDATE/DELETE/DO + * NOTHING */ + Node *stmt; /* T_UpdateStmt etc */ + List *targetList; /* the target list (of ResTarget) */ +} MergeAction; + /* ---------------------- * Select Statement * diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index c922216b7d..0a797f0a05 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -18,6 +18,7 @@ #include "lib/stringinfo.h" #include "nodes/bitmapset.h" #include "nodes/lockoptions.h" +#include "nodes/parsenodes.h" #include "nodes/primnodes.h" @@ -42,7 +43,7 @@ typedef struct PlannedStmt { NodeTag type; - CmdType commandType; /* select|insert|update|delete|utility */ + CmdType commandType; /* select|insert|update|delete|merge|utility */ uint64 queryId; /* query identifier (copied from Query) */ @@ -216,13 +217,14 @@ typedef struct ProjectSet typedef struct ModifyTable { Plan plan; - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ Index nominalRelation; /* Parent RT index for use of EXPLAIN */ /* RT indexes of non-leaf tables in a partition tree */ List *partitioned_rels; bool partColsUpdated; /* some part key in hierarchy updated */ List *resultRelations; /* integer list of RT indexes */ + Index mergeTargetRelation; /* RT index of the merge target */ int resultRelIndex; /* index of first resultRel in plan's list */ int rootResultRelIndex; /* index of the partitioned table root */ List *plans; /* plan(s) producing source data */ @@ -238,6 +240,8 @@ typedef struct ModifyTable Node *onConflictWhere; /* WHERE for ON CONFLICT UPDATE */ Index exclRelRTI; /* RTI of the EXCLUDED pseudo relation */ List *exclRelTlist; /* tlist of the EXCLUDED pseudo relation */ + List *mergeSourceTargetList; + List *mergeActionList; /* actions for MERGE */ } ModifyTable; /* ---------------- diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h index abbbda9e91..91dfff4cb5 100644 --- a/src/include/nodes/relation.h +++ b/src/include/nodes/relation.h @@ -1670,7 +1670,7 @@ typedef struct LockRowsPath } LockRowsPath; /* - * ModifyTablePath represents performing INSERT/UPDATE/DELETE modifications + * ModifyTablePath represents performing INSERT/UPDATE/DELETE/MERGE * * We represent most things that will be in the ModifyTable plan node * literally, except we have child Path(s) not Plan(s). But analysis of the @@ -1679,13 +1679,14 @@ typedef struct LockRowsPath typedef struct ModifyTablePath { Path path; - CmdType operation; /* INSERT, UPDATE, or DELETE */ + CmdType operation; /* INSERT, UPDATE, DELETE or MERGE */ bool canSetTag; /* do we set the command tag/es_processed? */ Index nominalRelation; /* Parent RT index for use of EXPLAIN */ /* RT indexes of non-leaf tables in a partition tree */ List *partitioned_rels; bool partColsUpdated; /* some part key in hierarchy updated */ List *resultRelations; /* integer list of RT indexes */ + Index mergeTargetRelation;/* RT index of merge target relation */ List *subpaths; /* Path(s) producing source data */ List *subroots; /* per-target-table PlannerInfos */ List *withCheckOptionLists; /* per-target-table WCO lists */ @@ -1693,6 +1694,8 @@ typedef struct ModifyTablePath List *rowMarks; /* PlanRowMarks (non-locking only) */ OnConflictExpr *onconflict; /* ON CONFLICT clause, or NULL */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ + List *mergeSourceTargetList; + List *mergeActionList; /* actions for MERGE */ } ModifyTablePath; /* diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index 381bc30813..895bf6959d 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -241,11 +241,14 @@ extern ModifyTablePath *create_modifytable_path(PlannerInfo *root, CmdType operation, bool canSetTag, Index nominalRelation, List *partitioned_rels, bool partColsUpdated, - List *resultRelations, List *subpaths, + List *resultRelations, + Index mergeTargetRelation, + List *subpaths, List *subroots, List *withCheckOptionLists, List *returningLists, List *rowMarks, OnConflictExpr *onconflict, - int epqParam); + List *mergeSourceTargetList, + List *mergeActionList, int epqParam); extern LimitPath *create_limit_path(PlannerInfo *root, RelOptInfo *rel, Path *subpath, Node *limitOffset, Node *limitCount, diff --git a/src/include/parser/analyze.h b/src/include/parser/analyze.h index 687ae1b5b7..41fb10666e 100644 --- a/src/include/parser/analyze.h +++ b/src/include/parser/analyze.h @@ -32,6 +32,11 @@ extern Query *parse_sub_analyze(Node *parseTree, ParseState *parentParseState, bool locked_from_parent, bool resolve_unknowns); +extern List *transformInsertRow(ParseState *pstate, List *exprlist, + List *stmtcols, List *icolumns, List *attrnos, + bool strip_indirection); +extern List *transformUpdateTargetList(ParseState *pstate, + List *targetList); extern Query *transformTopLevelStmt(ParseState *pstate, RawStmt *parseTree); extern Query *transformStmt(ParseState *pstate, Node *parseTree); diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index cf32197bc3..4dff55a8e9 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -244,8 +244,10 @@ PG_KEYWORD("locked", LOCKED, UNRESERVED_KEYWORD) PG_KEYWORD("logged", LOGGED, UNRESERVED_KEYWORD) PG_KEYWORD("mapping", MAPPING, UNRESERVED_KEYWORD) PG_KEYWORD("match", MATCH, UNRESERVED_KEYWORD) +PG_KEYWORD("matched", MATCHED, UNRESERVED_KEYWORD) PG_KEYWORD("materialized", MATERIALIZED, UNRESERVED_KEYWORD) PG_KEYWORD("maxvalue", MAXVALUE, UNRESERVED_KEYWORD) +PG_KEYWORD("merge", MERGE, UNRESERVED_KEYWORD) PG_KEYWORD("method", METHOD, UNRESERVED_KEYWORD) PG_KEYWORD("minute", MINUTE_P, UNRESERVED_KEYWORD) PG_KEYWORD("minvalue", MINVALUE, UNRESERVED_KEYWORD) diff --git a/src/include/parser/parse_clause.h b/src/include/parser/parse_clause.h index 2c0e092862..30121c98ed 100644 --- a/src/include/parser/parse_clause.h +++ b/src/include/parser/parse_clause.h @@ -20,7 +20,10 @@ extern void transformFromClause(ParseState *pstate, List *frmList); extern int setTargetTable(ParseState *pstate, RangeVar *relation, bool inh, bool alsoSource, AclMode requiredPerms); extern bool interpretOidsOption(List *defList, bool allowOids); - +extern Node *transformFromClauseItem(ParseState *pstate, Node *n, + RangeTblEntry **top_rte, int *top_rti, + RangeTblEntry **right_rte, int *right_rti, + List **namespace); extern Node *transformWhereClause(ParseState *pstate, Node *clause, ParseExprKind exprKind, const char *constructName); extern Node *transformLimitClause(ParseState *pstate, Node *clause, diff --git a/src/include/parser/parse_merge.h b/src/include/parser/parse_merge.h new file mode 100644 index 0000000000..0151809e09 --- /dev/null +++ b/src/include/parser/parse_merge.h @@ -0,0 +1,19 @@ +/*------------------------------------------------------------------------- + * + * parse_merge.h + * handle merge-stmt in parser + * + * + * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/parser/parse_merge.h + * + *------------------------------------------------------------------------- + */ +#ifndef PARSE_MERGE_H +#define PARSE_MERGE_H + +#include "parser/parse_node.h" +extern Query *transformMergeStmt(ParseState *pstate, MergeStmt *stmt); +#endif diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index 0230543810..8b80e79fd2 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -50,6 +50,7 @@ typedef enum ParseExprKind EXPR_KIND_INSERT_TARGET, /* INSERT target list item */ EXPR_KIND_UPDATE_SOURCE, /* UPDATE assignment source item */ EXPR_KIND_UPDATE_TARGET, /* UPDATE assignment target item */ + EXPR_KIND_MERGE_WHEN_AND, /* MERGE WHEN ... AND condition */ EXPR_KIND_GROUP_BY, /* GROUP BY */ EXPR_KIND_ORDER_BY, /* ORDER BY */ EXPR_KIND_DISTINCT_ON, /* DISTINCT ON */ @@ -127,7 +128,8 @@ typedef Node *(*CoerceParamHook) (ParseState *pstate, Param *param, * p_parent_cte: CommonTableExpr that immediately contains the current query, * if any. * - * p_target_relation: target relation, if query is INSERT, UPDATE, or DELETE. + * p_target_relation: target relation, if query is INSERT, UPDATE, DELETE + * or MERGE. * * p_target_rangetblentry: target relation's entry in the rtable list. * @@ -181,7 +183,7 @@ struct ParseState List *p_ctenamespace; /* current namespace for common table exprs */ List *p_future_ctes; /* common table exprs not yet in namespace */ CommonTableExpr *p_parent_cte; /* this query's containing CTE */ - Relation p_target_relation; /* INSERT/UPDATE/DELETE target rel */ + Relation p_target_relation; /* INSERT/UPDATE/DELETE/MERGE target rel */ RangeTblEntry *p_target_rangetblentry; /* target rel's RTE */ bool p_is_insert; /* process assignment like INSERT not UPDATE */ List *p_windowdefs; /* raw representations of window clauses */ diff --git a/src/include/rewrite/rewriteHandler.h b/src/include/rewrite/rewriteHandler.h index 8128199fc3..1ab5de3942 100644 --- a/src/include/rewrite/rewriteHandler.h +++ b/src/include/rewrite/rewriteHandler.h @@ -25,6 +25,7 @@ extern void AcquireRewriteLocks(Query *parsetree, extern Node *build_column_default(Relation rel, int attrno); extern void rewriteTargetListUD(Query *parsetree, RangeTblEntry *target_rte, Relation target_relation); +extern void rewriteTargetListMerge(Query *parsetree, Relation target_relation); extern Query *get_view_query(Relation view); extern const char *view_query_is_auto_updatable(Query *viewquery, diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c index 4c0114c514..a432636322 100644 --- a/src/interfaces/libpq/fe-exec.c +++ b/src/interfaces/libpq/fe-exec.c @@ -3055,9 +3055,9 @@ PQoidValue(const PGresult *res) /* * PQcmdTuples - - * If the last command was INSERT/UPDATE/DELETE/MOVE/FETCH/COPY, return - * a string containing the number of inserted/affected tuples. If not, - * return "". + * If the last command was INSERT/UPDATE/DELETE/MERGE/MOVE/FETCH/COPY, + * return a string containing the number of inserted/affected tuples. + * If not, return "". * * XXX: this should probably return an int */ @@ -3084,7 +3084,8 @@ PQcmdTuples(PGresult *res) strncmp(res->cmdStatus, "DELETE ", 7) == 0 || strncmp(res->cmdStatus, "UPDATE ", 7) == 0) p = res->cmdStatus + 7; - else if (strncmp(res->cmdStatus, "FETCH ", 6) == 0) + else if (strncmp(res->cmdStatus, "FETCH ", 6) == 0 || + strncmp(res->cmdStatus, "MERGE ", 6) == 0) p = res->cmdStatus + 6; else if (strncmp(res->cmdStatus, "MOVE ", 5) == 0 || strncmp(res->cmdStatus, "COPY ", 5) == 0) diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c index 38ea7a091f..8a1d32a95c 100644 --- a/src/pl/plpgsql/src/pl_exec.c +++ b/src/pl/plpgsql/src/pl_exec.c @@ -3932,7 +3932,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, /* * On the first call for this statement generate the plan, and detect - * whether the statement is INSERT/UPDATE/DELETE + * whether the statement is INSERT/UPDATE/DELETE/MERGE */ if (expr->plan == NULL) { @@ -3953,6 +3953,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, { if (q->commandType == CMD_INSERT || q->commandType == CMD_UPDATE || + q->commandType == CMD_MERGE || q->commandType == CMD_DELETE) stmt->mod_stmt = true; } @@ -4010,6 +4011,7 @@ exec_stmt_execsql(PLpgSQL_execstate *estate, case SPI_OK_INSERT_RETURNING: case SPI_OK_UPDATE_RETURNING: case SPI_OK_DELETE_RETURNING: + case SPI_OK_MERGE: Assert(stmt->mod_stmt); exec_set_found(estate, (SPI_processed != 0)); break; @@ -4187,6 +4189,7 @@ exec_stmt_dynexecute(PLpgSQL_execstate *estate, case SPI_OK_INSERT_RETURNING: case SPI_OK_UPDATE_RETURNING: case SPI_OK_DELETE_RETURNING: + case SPI_OK_MERGE: case SPI_OK_UTILITY: case SPI_OK_REWRITTEN: break; diff --git a/src/pl/plpgsql/src/pl_gram.y b/src/pl/plpgsql/src/pl_gram.y index 4c80936678..7d9fb2f039 100644 --- a/src/pl/plpgsql/src/pl_gram.y +++ b/src/pl/plpgsql/src/pl_gram.y @@ -303,6 +303,7 @@ static void check_raise_parameters(PLpgSQL_stmt_raise *stmt); %token <keyword> K_LAST %token <keyword> K_LOG %token <keyword> K_LOOP +%token <keyword> K_MERGE %token <keyword> K_MESSAGE %token <keyword> K_MESSAGE_TEXT %token <keyword> K_MOVE @@ -1930,6 +1931,10 @@ stmt_execsql : K_IMPORT { $$ = make_execsql_stmt(K_INSERT, @1); } + | K_MERGE + { + $$ = make_execsql_stmt(K_MERGE, @1); + } | T_WORD { int tok; @@ -2451,6 +2456,7 @@ unreserved_keyword : | K_IS | K_LAST | K_LOG + | K_MERGE | K_MESSAGE | K_MESSAGE_TEXT | K_MOVE @@ -2912,6 +2918,8 @@ make_execsql_stmt(int firsttoken, int location) { if (prev_tok == K_INSERT) continue; /* INSERT INTO is not an INTO-target */ + if (prev_tok == K_MERGE) + continue; /* MERGE INTO is not an INTO-target */ if (firsttoken == K_IMPORT) continue; /* IMPORT ... INTO is not an INTO-target */ if (have_into) diff --git a/src/pl/plpgsql/src/pl_scanner.c b/src/pl/plpgsql/src/pl_scanner.c index 65774f9902..85078ac15b 100644 --- a/src/pl/plpgsql/src/pl_scanner.c +++ b/src/pl/plpgsql/src/pl_scanner.c @@ -137,6 +137,7 @@ static const ScanKeyword unreserved_keywords[] = { PG_KEYWORD("is", K_IS, UNRESERVED_KEYWORD) PG_KEYWORD("last", K_LAST, UNRESERVED_KEYWORD) PG_KEYWORD("log", K_LOG, UNRESERVED_KEYWORD) + PG_KEYWORD("merge", K_MERGE, UNRESERVED_KEYWORD) PG_KEYWORD("message", K_MESSAGE, UNRESERVED_KEYWORD) PG_KEYWORD("message_text", K_MESSAGE_TEXT, UNRESERVED_KEYWORD) PG_KEYWORD("move", K_MOVE, UNRESERVED_KEYWORD) diff --git a/src/pl/plpgsql/src/plpgsql.h b/src/pl/plpgsql/src/plpgsql.h index f7619a63f9..8d6ef3326f 100644 --- a/src/pl/plpgsql/src/plpgsql.h +++ b/src/pl/plpgsql/src/plpgsql.h @@ -845,8 +845,8 @@ typedef struct PLpgSQL_stmt_execsql PLpgSQL_stmt_type cmd_type; int lineno; PLpgSQL_expr *sqlstmt; - bool mod_stmt; /* is the stmt INSERT/UPDATE/DELETE? Note: - * mod_stmt is set when we plan the query */ + bool mod_stmt; /* is the stmt INSERT/UPDATE/DELETE/MERGE? + * Note mod_stmt is set when we plan the query */ bool into; /* INTO supplied? */ bool strict; /* INTO STRICT flag */ PLpgSQL_variable *target; /* INTO target (record or row) */ diff --git a/src/test/isolation/expected/merge-delete.out b/src/test/isolation/expected/merge-delete.out new file mode 100644 index 0000000000..40e62901b7 --- /dev/null +++ b/src/test/isolation/expected/merge-delete.out @@ -0,0 +1,97 @@ +Parsed test spec with 2 sessions + +starting permutation: delete c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete c1 update1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete c1 update1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete c1 merge2 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: merge_delete c1 merge2 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: delete update1 c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; <waiting ...> +step c1: COMMIT; +step update1: <... completed> +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: merge_delete update1 c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step update1: UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; <waiting ...> +step c1: COMMIT; +step update1: <... completed> +step select2: SELECT * FROM target; +key val + +step c2: COMMIT; + +starting permutation: delete merge2 c1 select2 c2 +step delete: DELETE FROM target t WHERE t.key = 1; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; + +starting permutation: merge_delete merge2 c1 select2 c2 +step merge_delete: MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2a +step c2: COMMIT; diff --git a/src/test/isolation/expected/merge-insert-update.out b/src/test/isolation/expected/merge-insert-update.out new file mode 100644 index 0000000000..317fa16a3d --- /dev/null +++ b/src/test/isolation/expected/merge-insert-update.out @@ -0,0 +1,84 @@ +Parsed test spec with 2 sessions + +starting permutation: merge1 c1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +1 merge1 +step c2: COMMIT; + +starting permutation: merge1 c1 merge2 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step select2: SELECT * FROM target; +key val + +1 merge1 updated by merge2 +step c2: COMMIT; + +starting permutation: insert1 merge2 c1 select2 c2 +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: merge1 merge2 c1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: merge1 merge2 a1 select2 c2 +step merge1: MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step a1: ABORT; +step merge2: <... completed> +step select2: SELECT * FROM target; +key val + +1 merge2 +step c2: COMMIT; + +starting permutation: delete1 insert1 c1 merge2 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step c1: COMMIT; +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step select2: SELECT * FROM target; +key val + +1 insert1 updated by merge2 +step c2: COMMIT; + +starting permutation: delete1 insert1 merge2 c1 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; <waiting ...> +step c1: COMMIT; +step merge2: <... completed> +error in steps c1 merge2: ERROR: duplicate key value violates unique constraint "target_pkey" +step select2: SELECT * FROM target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +step c2: COMMIT; + +starting permutation: delete1 insert1 merge2i c1 select2 c2 +step delete1: DELETE FROM target WHERE key = 1; +step insert1: INSERT INTO target VALUES (1, 'insert1'); +step merge2i: MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +1 insert1 +step c2: COMMIT; diff --git a/src/test/isolation/expected/merge-match-recheck.out b/src/test/isolation/expected/merge-match-recheck.out new file mode 100644 index 0000000000..96a9f45ac8 --- /dev/null +++ b/src/test/isolation/expected/merge-match-recheck.out @@ -0,0 +1,106 @@ +Parsed test spec with 2 sessions + +starting permutation: update1 merge_status c2 select1 c1 +step update1: UPDATE target t SET balance = balance + 10, val = t.val || ' updated by update1' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 170 s2 setup updated by update1 when1 +step c1: COMMIT; + +starting permutation: update2 merge_status c2 select1 c1 +step update2: UPDATE target t SET status = 's2', val = t.val || ' updated by update2' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s3 setup updated by update2 when2 +step c1: COMMIT; + +starting permutation: update3 merge_status c2 select1 c1 +step update3: UPDATE target t SET status = 's3', val = t.val || ' updated by update3' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s4 setup updated by update3 when3 +step c1: COMMIT; + +starting permutation: update5 merge_status c2 select1 c1 +step update5: UPDATE target t SET status = 's5', val = t.val || ' updated by update5' WHERE t.key = 1; +step merge_status: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_status: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 160 s5 setup updated by update5 +step c1: COMMIT; + +starting permutation: update_bal1 merge_bal c2 select1 c1 +step update_bal1: UPDATE target t SET balance = 50, val = t.val || ' updated by update_bal1' WHERE t.key = 1; +step merge_bal: + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND balance < 100 THEN + UPDATE SET balance = balance * 2, val = t.val || ' when1' + WHEN MATCHED AND balance < 200 THEN + UPDATE SET balance = balance * 4, val = t.val || ' when2' + WHEN MATCHED AND balance < 300 THEN + UPDATE SET balance = balance * 8, val = t.val || ' when3'; + <waiting ...> +step c2: COMMIT; +step merge_bal: <... completed> +step select1: SELECT * FROM target; +key balance status val + +1 100 s1 setup updated by update_bal1 when1 +step c1: COMMIT; diff --git a/src/test/isolation/expected/merge-update.out b/src/test/isolation/expected/merge-update.out new file mode 100644 index 0000000000..60ae42ebd0 --- /dev/null +++ b/src/test/isolation/expected/merge-update.out @@ -0,0 +1,213 @@ +Parsed test spec with 2 sessions + +starting permutation: merge1 c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step c1: COMMIT; +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +step c2: COMMIT; + +starting permutation: merge1 c1 merge2a select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step c1: COMMIT; +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2a c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2a: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2a a1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2a: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step a1: ABORT; +step merge2a: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge2a +step c2: COMMIT; + +starting permutation: merge1 merge2b c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2b: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2b' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED AND t.key < 2 THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2b: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2b +step c2: COMMIT; + +starting permutation: merge1 merge2c c1 select2 c2 +step merge1: + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step merge2c: + MERGE INTO target t + USING (SELECT 1 as key, 'merge2c' as val) s + ON s.key = t.key AND t.key < 2 + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step merge2c: <... completed> +step select2: SELECT * FROM target; +key val + +2 setup1 updated by merge1 +1 merge2c +step c2: COMMIT; + +starting permutation: pa_merge1 pa_merge2a c1 pa_select2 c2 +step pa_merge1: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set val = t.val || ' updated by ' || s.val; + +step pa_merge2a: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step pa_merge2a: <... completed> +step pa_select2: SELECT * FROM pa_target; +key val + +2 initial +2 initial updated by pa_merge2a +step c2: COMMIT; + +starting permutation: pa_merge2 pa_merge2a c1 pa_select2 c2 +step pa_merge2: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + +step pa_merge2a: + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; + <waiting ...> +step c1: COMMIT; +step pa_merge2a: <... completed> +step pa_select2: SELECT * FROM pa_target; +key val + +1 pa_merge2a +2 initial +2 initial updated by pa_merge1 +step c2: COMMIT; diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index 53e1f192b0..d356d1fed8 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -33,6 +33,10 @@ test: insert-conflict-do-update test: insert-conflict-do-update-2 test: insert-conflict-do-update-3 test: insert-conflict-toast +test: merge-insert-update +test: merge-delete +test: merge-update +test: merge-match-recheck test: delete-abort-savept test: delete-abort-savept-2 test: aborted-keyrevoke diff --git a/src/test/isolation/specs/merge-delete.spec b/src/test/isolation/specs/merge-delete.spec new file mode 100644 index 0000000000..656954f847 --- /dev/null +++ b/src/test/isolation/specs/merge-delete.spec @@ -0,0 +1,51 @@ +# MERGE DELETE +# +# This test looks at the interactions involving concurrent deletes +# comparing the behavior of MERGE, DELETE and UPDATE + +setup +{ + CREATE TABLE target (key int primary key, val text); + INSERT INTO target VALUES (1, 'setup1'); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "delete" { DELETE FROM target t WHERE t.key = 1; } +step "merge_delete" { MERGE INTO target t USING (SELECT 1 as key) s ON s.key = t.key WHEN MATCHED THEN DELETE; } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "update1" { UPDATE target t SET val = t.val || ' updated by update1' WHERE t.key = 1; } +step "merge2" { MERGE INTO target t USING (SELECT 1 as key, 'merge2a' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; } +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } + +# Basic effects +permutation "delete" "c1" "select2" "c2" +permutation "merge_delete" "c1" "select2" "c2" + +# One after the other, no concurrency +permutation "delete" "c1" "update1" "select2" "c2" +permutation "merge_delete" "c1" "update1" "select2" "c2" +permutation "delete" "c1" "merge2" "select2" "c2" +permutation "merge_delete" "c1" "merge2" "select2" "c2" + +# Now with concurrency +permutation "delete" "update1" "c1" "select2" "c2" +permutation "merge_delete" "update1" "c1" "select2" "c2" +permutation "delete" "merge2" "c1" "select2" "c2" +permutation "merge_delete" "merge2" "c1" "select2" "c2" diff --git a/src/test/isolation/specs/merge-insert-update.spec b/src/test/isolation/specs/merge-insert-update.spec new file mode 100644 index 0000000000..704492be1f --- /dev/null +++ b/src/test/isolation/specs/merge-insert-update.spec @@ -0,0 +1,52 @@ +# MERGE INSERT UPDATE +# +# This looks at how we handle concurrent INSERTs, illustrating how the +# behavior differs from INSERT ... ON CONFLICT + +setup +{ + CREATE TABLE target (key int primary key, val text); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge1" { MERGE INTO target t USING (SELECT 1 as key, 'merge1' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge1'; } +step "delete1" { DELETE FROM target WHERE key = 1; } +step "insert1" { INSERT INTO target VALUES (1, 'insert1'); } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge2" { MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN NOT MATCHED THEN INSERT VALUES (s.key, s.val) WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; } + +step "merge2i" { MERGE INTO target t USING (SELECT 1 as key, 'merge2' as val) s ON s.key = t.key WHEN MATCHED THEN UPDATE set val = t.val || ' updated by merge2'; } + +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } +step "a2" { ABORT; } + +# Basic effects +permutation "merge1" "c1" "select2" "c2" +permutation "merge1" "c1" "merge2" "select2" "c2" + +# check concurrent inserts +permutation "insert1" "merge2" "c1" "select2" "c2" +permutation "merge1" "merge2" "c1" "select2" "c2" +permutation "merge1" "merge2" "a1" "select2" "c2" + +# check how we handle when visible row has been concurrently deleted, then same key re-inserted +permutation "delete1" "insert1" "c1" "merge2" "select2" "c2" +permutation "delete1" "insert1" "merge2" "c1" "select2" "c2" +permutation "delete1" "insert1" "merge2i" "c1" "select2" "c2" diff --git a/src/test/isolation/specs/merge-match-recheck.spec b/src/test/isolation/specs/merge-match-recheck.spec new file mode 100644 index 0000000000..193033da17 --- /dev/null +++ b/src/test/isolation/specs/merge-match-recheck.spec @@ -0,0 +1,79 @@ +# MERGE MATCHED RECHECK +# +# This test looks at what happens when we have complex +# WHEN MATCHED AND conditions and a concurrent UPDATE causes a +# recheck of the AND condition on the new row + +setup +{ + CREATE TABLE target (key int primary key, balance integer, status text, val text); + INSERT INTO target VALUES (1, 160, 's1', 'setup'); +} + +teardown +{ + DROP TABLE target; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge_status" +{ + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND status = 's1' THEN + UPDATE SET status = 's2', val = t.val || ' when1' + WHEN MATCHED AND status = 's2' THEN + UPDATE SET status = 's3', val = t.val || ' when2' + WHEN MATCHED AND status = 's3' THEN + UPDATE SET status = 's4', val = t.val || ' when3'; +} + +step "merge_bal" +{ + MERGE INTO target t + USING (SELECT 1 as key) s + ON s.key = t.key + WHEN MATCHED AND balance < 100 THEN + UPDATE SET balance = balance * 2, val = t.val || ' when1' + WHEN MATCHED AND balance < 200 THEN + UPDATE SET balance = balance * 4, val = t.val || ' when2' + WHEN MATCHED AND balance < 300 THEN + UPDATE SET balance = balance * 8, val = t.val || ' when3'; +} + +step "select1" { SELECT * FROM target; } +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "update1" { UPDATE target t SET balance = balance + 10, val = t.val || ' updated by update1' WHERE t.key = 1; } +step "update2" { UPDATE target t SET status = 's2', val = t.val || ' updated by update2' WHERE t.key = 1; } +step "update3" { UPDATE target t SET status = 's3', val = t.val || ' updated by update3' WHERE t.key = 1; } +step "update5" { UPDATE target t SET status = 's5', val = t.val || ' updated by update5' WHERE t.key = 1; } +step "update_bal1" { UPDATE target t SET balance = 50, val = t.val || ' updated by update_bal1' WHERE t.key = 1; } +step "select2" { SELECT * FROM target; } +step "c2" { COMMIT; } + +# merge_status sees concurrently updated row and rechecks WHEN conditions, but recheck passes and final status = 's2' +permutation "update1" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final status = 's3' not 's2' +permutation "update2" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final status = 's4' not 's2' +permutation "update3" "merge_status" "c2" "select1" "c1" + +# merge_status sees concurrently updated row and rechecks WHEN conditions, recheck fails, but we skip update and MERGE does nothing +permutation "update5" "merge_status" "c2" "select1" "c1" + +# merge_bal sees concurrently updated row and rechecks WHEN conditions, recheck fails, so final balance = 100 not 640 +permutation "update_bal1" "merge_bal" "c2" "select1" "c1" diff --git a/src/test/isolation/specs/merge-update.spec b/src/test/isolation/specs/merge-update.spec new file mode 100644 index 0000000000..64e849966e --- /dev/null +++ b/src/test/isolation/specs/merge-update.spec @@ -0,0 +1,132 @@ +# MERGE UPDATE +# +# This test exercises atypical cases +# 1. UPDATEs of PKs that change the join in the ON clause +# 2. UPDATEs with WHEN AND conditions that would fail after concurrent update +# 3. UPDATEs with extra ON conditions that would fail after concurrent update + +setup +{ + CREATE TABLE target (key int primary key, val text); + INSERT INTO target VALUES (1, 'setup1'); + + CREATE TABLE pa_target (key integer, val text) + PARTITION BY LIST (key); + CREATE TABLE part1 (key integer, val text); + CREATE TABLE part2 (val text, key integer); + CREATE TABLE part3 (key integer, val text); + + ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); + ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); + ALTER TABLE pa_target ATTACH PARTITION part3 DEFAULT; + + INSERT INTO pa_target VALUES (1, 'initial'); + INSERT INTO pa_target VALUES (2, 'initial'); +} + +teardown +{ + DROP TABLE target; + DROP TABLE pa_target CASCADE; +} + +session "s1" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge1" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "pa_merge1" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set val = t.val || ' updated by ' || s.val; +} +step "pa_merge2" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge1' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "c1" { COMMIT; } +step "a1" { ABORT; } + +session "s2" +setup +{ + BEGIN ISOLATION LEVEL READ COMMITTED; +} +step "merge2a" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "merge2b" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2b' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED AND t.key < 2 THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "merge2c" +{ + MERGE INTO target t + USING (SELECT 1 as key, 'merge2c' as val) s + ON s.key = t.key AND t.key < 2 + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "pa_merge2a" +{ + MERGE INTO pa_target t + USING (SELECT 1 as key, 'pa_merge2a' as val) s + ON s.key = t.key + WHEN NOT MATCHED THEN + INSERT VALUES (s.key, s.val) + WHEN MATCHED THEN + UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val; +} +step "select2" { SELECT * FROM target; } +step "pa_select2" { SELECT * FROM pa_target; } +step "c2" { COMMIT; } + +# Basic effects +permutation "merge1" "c1" "select2" "c2" + +# One after the other, no concurrency +permutation "merge1" "c1" "merge2a" "select2" "c2" + +# Now with concurrency +permutation "merge1" "merge2a" "c1" "select2" "c2" +permutation "merge1" "merge2a" "a1" "select2" "c2" +permutation "merge1" "merge2b" "c1" "select2" "c2" +permutation "merge1" "merge2c" "c1" "select2" "c2" +permutation "pa_merge1" "pa_merge2a" "c1" "pa_select2" "c2" +permutation "pa_merge2" "pa_merge2a" "c1" "pa_select2" "c2" diff --git a/src/test/regress/expected/identity.out b/src/test/regress/expected/identity.out index d7d5178f5d..3a6016c80a 100644 --- a/src/test/regress/expected/identity.out +++ b/src/test/regress/expected/identity.out @@ -386,3 +386,58 @@ CREATE TABLE itest_child PARTITION OF itest_parent ( ) FOR VALUES FROM ('2016-07-01') TO ('2016-08-01'); -- error ERROR: identity columns are not supported on partitions DROP TABLE itest_parent; +-- MERGE tests +CREATE TABLE itest14 (a int GENERATED ALWAYS AS IDENTITY, b text); +CREATE TABLE itest15 (a int GENERATED BY DEFAULT AS IDENTITY, b text); +MERGE INTO itest14 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); +ERROR: cannot insert into column "a" +DETAIL: Column "a" is an identity column defined as GENERATED ALWAYS. +HINT: Use OVERRIDING SYSTEM VALUE to override. +MERGE INTO itest14 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); +ERROR: cannot insert into column "a" +DETAIL: Column "a" is an identity column defined as GENERATED ALWAYS. +HINT: Use OVERRIDING SYSTEM VALUE to override. +MERGE INTO itest14 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); +MERGE INTO itest15 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); +SELECT * FROM itest14; + a | b +----+------------------- + 30 | inserted by merge +(1 row) + +SELECT * FROM itest15; + a | b +----+------------------- + 10 | inserted by merge + 1 | inserted by merge + 30 | inserted by merge +(3 rows) + +DROP TABLE itest14; +DROP TABLE itest15; diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out new file mode 100644 index 0000000000..05c4287078 --- /dev/null +++ b/src/test/regress/expected/merge.out @@ -0,0 +1,1599 @@ +-- +-- MERGE +-- +--\set VERBOSITY verbose +--set debug_print_rewritten = true; +--set debug_print_parse = true; +--set debug_print_pretty = true; +CREATE USER merge_privs; +CREATE USER merge_no_privs; +DROP TABLE IF EXISTS target; +NOTICE: table "target" does not exist, skipping +DROP TABLE IF EXISTS source; +NOTICE: table "source" does not exist, skipping +CREATE TABLE target (tid integer, balance integer); +CREATE TABLE source (sid integer, delta integer); --no index +INSERT INTO target VALUES (1, 10); +INSERT INTO target VALUES (2, 20); +INSERT INTO target VALUES (3, 30); +SELECT t.ctid is not null as matched, t.*, s.* FROM source s FULL OUTER JOIN target t ON s.sid = t.tid ORDER BY t.tid, s.sid; + matched | tid | balance | sid | delta +---------+-----+---------+-----+------- + t | 1 | 10 | | + t | 2 | 20 | | + t | 3 | 30 | | +(3 rows) + +ALTER TABLE target OWNER TO merge_privs; +ALTER TABLE source OWNER TO merge_privs; +CREATE TABLE target2 (tid integer, balance integer); +CREATE TABLE source2 (sid integer, delta integer); +ALTER TABLE target2 OWNER TO merge_no_privs; +ALTER TABLE source2 OWNER TO merge_no_privs; +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_privs; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Merge Join + Merge Cond: (t_1.tid = s.sid) + -> Sort + Sort Key: t_1.tid + -> Seq Scan on target t_1 + -> Sort + Sort Key: s.sid + -> Seq Scan on source s +(9 rows) + +-- +-- Errors +-- +MERGE INTO target t RANDOMWORD +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: syntax error at or near "RANDOMWORD" +LINE 1: MERGE INTO target t RANDOMWORD + ^ +-- MATCHED/INSERT error +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: syntax error at or near "INSERT" +LINE 5: INSERT DEFAULT VALUES + ^ +-- incorrectly specifying INTO target +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT INTO target DEFAULT VALUES +; +ERROR: syntax error at or near "INTO" +LINE 5: INSERT INTO target DEFAULT VALUES + ^ +-- NOT MATCHED/UPDATE +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: syntax error at or near "UPDATE" +LINE 5: UPDATE SET balance = 0 + ^ +-- UPDATE tablename +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE target SET balance = 0 +; +ERROR: syntax error at or near "target" +LINE 5: UPDATE target SET balance = 0 + ^ +-- permissions +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: permission denied for table source2 +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_no_privs; +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: permission denied for table target +GRANT UPDATE ON target2 TO merge_privs; +SET SESSION AUTHORIZATION merge_privs; +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN MATCHED THEN + DELETE +; +ERROR: permission denied for table target2 +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: permission denied for table target2 +-- check if the target can be accessed from source relation subquery; we should +-- not be able to do so +MERGE INTO target t +USING (SELECT * FROM source WHERE t.tid > sid) s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 2: USING (SELECT * FROM source WHERE t.tid > sid) s + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +-- +-- initial tests +-- +-- zero rows in source has no effect +MERGE INTO target +USING source +ON target.tid = source.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ROLLBACK; +-- insert some non-matching source rows to work from +INSERT INTO source VALUES (4, 40); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 4 | 40 +(1 row) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + DO NOTHING +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + | +(4 rows) + +ROLLBACK; +-- index plans +INSERT INTO target SELECT generate_series(1000,2500), 0; +ALTER TABLE target ADD PRIMARY KEY (tid); +ANALYZE target; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL); + QUERY PLAN +------------------------------------------ + Merge on target t + -> Hash Left Join + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s + -> Hash + -> Seq Scan on target t_1 +(6 rows) + +; +DELETE FROM target WHERE tid > 100; +ANALYZE target; +-- insert some matching source rows to work from +INSERT INTO source VALUES (2, 5); +INSERT INTO source VALUES (3, 20); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 3 | 20 + 4 | 40 +(3 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +-- equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 +(3 rows) + +ROLLBACK; +-- equivalent of a DELETE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 +(1 row) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | +(4 rows) + +ROLLBACK; +-- duplicate source row causes multiple target row update ERROR +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 2 | 5 + 3 | 20 + 4 | 40 +(4 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ERROR: MERGE command cannot affect row a second time +HINT: Ensure that not more than one source rows match any one target row +ROLLBACK; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +ERROR: MERGE command cannot affect row a second time +HINT: Ensure that not more than one source rows match any one target row +ROLLBACK; +-- correct source data +DELETE FROM source WHERE sid = 2; +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; + sid | delta +-----+------- + 2 | 5 + 3 | 20 + 4 | 40 +(3 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +-- remove constraints +alter table target drop CONSTRAINT target_pkey; +alter table target alter column tid drop not null; +-- multiple actions +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4) +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 + 4 | 4 +(4 rows) + +ROLLBACK; +-- should be equivalent +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4); +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 0 + 3 | 0 + 4 | 4 +(4 rows) + +ROLLBACK; +-- column references +-- do a simple equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 25 + 3 | 50 +(3 rows) + +ROLLBACK; +-- do a simple equivalent of an INSERT SELECT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- and again with explicitly identified column list +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- and again with a subtle error: referring to non-existent target row for NOT MATCHED +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 5: INSERT (tid, balance) VALUES (t.tid, s.delta) + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +-- and again with a constant ON clause +BEGIN; +MERGE INTO target t +USING source AS s +ON (SELECT true) +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 5: INSERT (tid, balance) VALUES (t.tid, s.delta) + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +SELECT * FROM target ORDER BY tid; +ERROR: current transaction is aborted, commands ignored until end of transaction block +ROLLBACK; +-- now the classic UPSERT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 25 + 3 | 50 + 4 | 40 +(4 rows) + +ROLLBACK; +-- unreachable WHEN clause should ERROR +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN /* Terminal WHEN clause for MATCHED */ + DELETE +WHEN MATCHED AND s.delta > 0 THEN + UPDATE SET balance = t.balance - s.delta +; +ERROR: unreachable WHEN clause specified after unconditional WHEN clause +ROLLBACK; +-- conditional WHEN clause +CREATE TABLE wq_target (tid integer not null, balance integer DEFAULT -1); +CREATE TABLE wq_source (balance integer, sid integer); +INSERT INTO wq_source (sid, balance) VALUES (1, 100); +BEGIN; +-- try a simple INSERT with default values first +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +ROLLBACK; +-- this time with a FALSE condition +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND FALSE THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- +(0 rows) + +-- this time with an actual condition which returns false +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance <> 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- +(0 rows) + +BEGIN; +-- and now with a condition which returns true +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +ROLLBACK; +-- conditions in the NOT MATCHED clause can only refer to source columns +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND t.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +ERROR: invalid reference to FROM-clause entry for table "t" +LINE 3: WHEN NOT MATCHED AND t.balance = 100 THEN + ^ +HINT: There is an entry for table "t", but it cannot be referenced from this part of the query. +SELECT * FROM wq_target; +ERROR: current transaction is aborted, commands ignored until end of transaction block +ROLLBACK; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | -1 +(1 row) + +-- conditions in MATCHED clause can refer to both source and target +SELECT * FROM wq_source; + balance | sid +---------+----- + 100 | 1 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +-- check if AND works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 99 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 199 +(1 row) + +-- check if OR works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 199 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 199 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 299 +(1 row) + +-- check if subqueries work in the conditions? +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance > (SELECT max(balance) FROM target) THEN + UPDATE SET balance = t.balance + s.balance; +-- check if we can access system columns in the conditions +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.xmin = t.xmax THEN + UPDATE SET balance = t.balance + s.balance; +ERROR: system column "xmin" reference in WHEN AND condition is invalid +LINE 3: WHEN MATCHED AND t.xmin = t.xmax THEN + ^ +ALTER TABLE wq_target SET WITH OIDS; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 399 +(1 row) + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.oid >= 0 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + tid | balance +-----+--------- + 1 | 499 +(1 row) + +-- test preventing WHEN AND conditions from writing to the database +create or replace function merge_when_and_write() returns boolean +language plpgsql as +$$ +BEGIN + INSERT INTO target VALUES (100, 100); + RETURN TRUE; +END; +$$; +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND (merge_when_and_write()) THEN + UPDATE SET balance = t.balance + s.balance; +ROLLBACK; +drop function merge_when_and_write(); +DROP TABLE wq_target, wq_source; +-- test triggers +create or replace function merge_trigfunc () returns trigger +language plpgsql as +$$ +BEGIN + RAISE NOTICE '% % % trigger', TG_WHEN, TG_OP, TG_LEVEL; + IF (TG_WHEN = 'BEFORE' AND TG_LEVEL = 'ROW') THEN + IF (TG_OP = 'DELETE') THEN + RETURN OLD; + ELSE + RETURN NEW; + END IF; + ELSE + RETURN NULL; + END IF; +END; +$$; +CREATE TRIGGER merge_bsi BEFORE INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsu BEFORE UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsd BEFORE DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asi AFTER INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asu AFTER UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asd AFTER DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bri BEFORE INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bru BEFORE UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_brd BEFORE DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ari AFTER INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_aru AFTER UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ard AFTER DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +-- now the classic UPSERT, with a DELETE +BEGIN; +UPDATE target SET balance = 0 WHERE tid = 3; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE DELETE STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: BEFORE DELETE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER DELETE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER DELETE STATEMENT trigger +NOTICE: AFTER UPDATE STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger + QUERY PLAN +------------------------------------------------------------------ + Merge on target t (actual rows=0 loops=1) + Tuples Inserted: 1 + Tuples Updated: 1 + Tuples Deleted: 1 + -> Hash Left Join (actual rows=3 loops=1) + Hash Cond: (s.sid = t_1.tid) + -> Seq Scan on source s (actual rows=3 loops=1) + -> Hash (actual rows=3 loops=1) + Buckets: 1024 Batches: 1 Memory Usage: 9kB + -> Seq Scan on target t_1 (actual rows=3 loops=1) + Trigger merge_ard: calls=1 + Trigger merge_ari: calls=1 + Trigger merge_aru: calls=1 + Trigger merge_asd: calls=1 + Trigger merge_asi: calls=1 + Trigger merge_asu: calls=1 + Trigger merge_brd: calls=1 + Trigger merge_bri: calls=1 + Trigger merge_bru: calls=1 + Trigger merge_bsd: calls=1 + Trigger merge_bsi: calls=1 + Trigger merge_bsu: calls=1 +(22 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 15 + 4 | 40 +(3 rows) + +ROLLBACK; +-- test from PL/pgSQL +-- make sure MERGE INTO isn't interpreted to mean returning variables like SELECT INTO +BEGIN; +DO LANGUAGE plpgsql $$ +BEGIN +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +; +END; +$$; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +ROLLBACK; +--source constants +BEGIN; +MERGE INTO target t +USING (SELECT 9 AS sid, 57 AS delta) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 9 | 57 +(4 rows) + +ROLLBACK; +--source query +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta as newname FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.newname) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +--self-merge +BEGIN; +MERGE INTO target t1 +USING target t2 +ON t1.tid = t2.tid +WHEN MATCHED THEN + UPDATE SET balance = t1.balance + t2.balance +WHEN NOT MATCHED THEN + INSERT VALUES (t2.tid, t2.balance) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 20 + 2 | 40 + 3 | 60 +(3 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING (SELECT tid as sid, balance as delta FROM target WHERE balance > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +BEGIN; +MERGE INTO target t +USING +(SELECT sid, max(delta) AS delta + FROM source + GROUP BY sid + HAVING count(*) = 1 + ORDER BY sid ASC) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +NOTICE: BEFORE INSERT STATEMENT trigger +NOTICE: BEFORE INSERT ROW trigger +NOTICE: AFTER INSERT ROW trigger +NOTICE: AFTER INSERT STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 30 + 4 | 40 +(4 rows) + +ROLLBACK; +-- plpgsql parameters and results +BEGIN; +CREATE FUNCTION merge_func (p_id integer, p_bal integer) +RETURNS INTEGER +LANGUAGE plpgsql +AS $$ +DECLARE + result integer; +BEGIN +MERGE INTO target t +USING (SELECT p_id AS sid) AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance - p_bal +; +IF FOUND THEN + GET DIAGNOSTICS result := ROW_COUNT; +END IF; +RETURN result; +END; +$$; +SELECT merge_func(3, 4); +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger + merge_func +------------ + 1 +(1 row) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 10 + 2 | 20 + 3 | 26 +(3 rows) + +ROLLBACK; +-- PREPARE +BEGIN; +prepare foom as merge into target t using (select 1 as sid) s on (t.tid = s.sid) when matched then update set balance = 1; +execute foom; +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 1 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +BEGIN; +PREPARE foom2 (integer, integer) AS +MERGE INTO target t +USING (SELECT 1) s +ON t.tid = $1 +WHEN MATCHED THEN +UPDATE SET balance = $2; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +execute foom2 (1, 1); +NOTICE: BEFORE UPDATE STATEMENT trigger +NOTICE: BEFORE UPDATE ROW trigger +NOTICE: AFTER UPDATE ROW trigger +NOTICE: AFTER UPDATE STATEMENT trigger + QUERY PLAN +------------------------------------------------------ + Merge on target t (actual rows=0 loops=1) + Tuples Inserted: 0 + Tuples Updated: 1 + Tuples Deleted: 0 + -> Seq Scan on target t_1 (actual rows=1 loops=1) + Filter: (tid = 1) + Rows Removed by Filter: 2 + Trigger merge_aru: calls=1 + Trigger merge_asu: calls=1 + Trigger merge_bru: calls=1 + Trigger merge_bsu: calls=1 +(11 rows) + +SELECT * FROM target ORDER BY tid; + tid | balance +-----+--------- + 1 | 1 + 2 | 20 + 3 | 30 +(3 rows) + +ROLLBACK; +-- subqueries in source relation +CREATE TABLE sq_target (tid integer NOT NULL, balance integer); +CREATE TABLE sq_source (delta integer, sid integer, balance integer DEFAULT 0); +INSERT INTO sq_target(tid, balance) VALUES (1,100), (2,200), (3,300); +INSERT INTO sq_source(sid, delta) VALUES (1,10), (2,20), (4,40); +BEGIN; +MERGE INTO sq_target t +USING (SELECT * FROM sq_source) s +ON tid = sid +WHEN MATCHED AND t.balance > delta THEN + UPDATE SET balance = t.balance + delta; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 3 | 300 + 1 | 110 + 2 | 220 +(3 rows) + +ROLLBACK; +-- try a view +CREATE VIEW v AS SELECT * FROM sq_source WHERE sid < 2; +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = v.balance + delta; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 2 | 200 + 3 | 300 + 1 | 10 +(3 rows) + +ROLLBACK; +-- ambiguous reference to a column +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +ERROR: column reference "balance" is ambiguous +LINE 5: UPDATE SET balance = balance + delta + ^ +ROLLBACK; +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +SELECT * FROM sq_target; + tid | balance +-----+--------- + 2 | 200 + 3 | 300 + -1 | -11 +(3 rows) + +ROLLBACK; +-- CTEs +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +WITH targq AS ( + SELECT * FROM v +) +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +; +ERROR: syntax error at or near "MERGE" +LINE 4: MERGE INTO sq_target t + ^ +ROLLBACK; +-- RETURNING +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +RETURNING * +; +ERROR: syntax error at or near "RETURNING" +LINE 10: RETURNING * + ^ +ROLLBACK; +-- Subqueries +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = (SELECT count(*) FROM sq_target) +; +ROLLBACK; +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND (SELECT count(*) > 0 FROM sq_target) THEN + UPDATE SET balance = 42 +; +ROLLBACK; +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid AND (SELECT count(*) > 0 FROM sq_target) +WHEN MATCHED THEN + UPDATE SET balance = 42 +; +ROLLBACK; +DROP TABLE sq_target, sq_source CASCADE; +NOTICE: drop cascades to view v +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 PARTITION OF pa_target FOR VALUES IN (1,4); +CREATE TABLE part2 PARTITION OF pa_target FOR VALUES IN (2,5,6); +CREATE TABLE part3 PARTITION OF pa_target FOR VALUES IN (3,8,9); +CREATE TABLE part4 PARTITION OF pa_target DEFAULT; +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 330 | initial updated by merge + 4 | 40 | inserted by merge + 5 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 7 | 770 | initial updated by merge + 8 | 80 | inserted by merge + 9 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 11 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 13 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 30 | inserted by merge + 3 | 300 | initial + 4 | 40 | inserted by merge + 5 | 500 | initial + 5 | 50 | inserted by merge + 6 | 60 | inserted by merge + 7 | 700 | initial + 7 | 70 | inserted by merge + 8 | 80 | inserted by merge + 9 | 90 | inserted by merge + 9 | 900 | initial + 10 | 100 | inserted by merge + 11 | 1100 | initial + 11 | 110 | inserted by merge + 12 | 120 | inserted by merge + 13 | 1300 | initial + 13 | 130 | inserted by merge + 14 | 140 | inserted by merge +(20 rows) + +ROLLBACK; +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 2 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 4 | 40 | inserted by merge + 4 | 330 | initial updated by merge + 6 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 8 | 80 | inserted by merge + 8 | 770 | initial updated by merge + 10 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 12 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 14 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +DROP TABLE pa_target CASCADE; +-- The target table is partitioned in the same way, but this time by attaching +-- partitions which have columns in different order, dropped columns etc. +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 (tid integer, balance float, val text); +CREATE TABLE part2 (balance float, tid integer, val text); +CREATE TABLE part3 (tid integer, balance float, val text); +CREATE TABLE part4 (extraid text, tid integer, balance float, val text); +ALTER TABLE part4 DROP COLUMN extraid; +ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); +ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); +ALTER TABLE pa_target ATTACH PARTITION part3 FOR VALUES IN (3,8,9); +ALTER TABLE pa_target ATTACH PARTITION part4 DEFAULT; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 330 | initial updated by merge + 4 | 40 | inserted by merge + 5 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 7 | 770 | initial updated by merge + 8 | 80 | inserted by merge + 9 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 11 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 13 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 1 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 3 | 30 | inserted by merge + 3 | 300 | initial + 4 | 40 | inserted by merge + 5 | 500 | initial + 5 | 50 | inserted by merge + 6 | 60 | inserted by merge + 7 | 700 | initial + 7 | 70 | inserted by merge + 8 | 80 | inserted by merge + 9 | 90 | inserted by merge + 9 | 900 | initial + 10 | 100 | inserted by merge + 11 | 1100 | initial + 11 | 110 | inserted by merge + 12 | 120 | inserted by merge + 13 | 1300 | initial + 13 | 130 | inserted by merge + 14 | 140 | inserted by merge +(20 rows) + +ROLLBACK; +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + tid | balance | val +-----+---------+-------------------------- + 2 | 110 | initial updated by merge + 2 | 20 | inserted by merge + 4 | 40 | inserted by merge + 4 | 330 | initial updated by merge + 6 | 550 | initial updated by merge + 6 | 60 | inserted by merge + 8 | 80 | inserted by merge + 8 | 770 | initial updated by merge + 10 | 990 | initial updated by merge + 10 | 100 | inserted by merge + 12 | 1210 | initial updated by merge + 12 | 120 | inserted by merge + 14 | 1430 | initial updated by merge + 14 | 140 | inserted by merge +(14 rows) + +ROLLBACK; +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; +-- Sub-partitionin +CREATE TABLE pa_target (logts timestamp, tid integer, balance float, val text) + PARTITION BY RANGE (logts); +CREATE TABLE part_m01 PARTITION OF pa_target + FOR VALUES FROM ('2017-01-01') TO ('2017-02-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m01_odd PARTITION OF part_m01 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m01_even PARTITION OF part_m01 + FOR VALUES IN (2,4,6,8); +CREATE TABLE part_m02 PARTITION OF pa_target + FOR VALUES FROM ('2017-02-01') TO ('2017-03-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m02_odd PARTITION OF part_m02 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m02_even PARTITION OF part_m02 + FOR VALUES IN (2,4,6,8); +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT '2017-01-31', id, id * 100, 'initial' FROM generate_series(1,9,3) AS id; +INSERT INTO pa_target SELECT '2017-02-28', id, id * 100, 'initial' FROM generate_series(2,9,3) AS id; +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING (SELECT '2017-01-15' AS slogts, * FROM pa_source WHERE sid < 10) s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; + logts | tid | balance | val +--------------------------+-----+---------+-------------------------- + Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge + Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge + Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge + Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge + Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge +(9 rows) + +ROLLBACK; +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; +-- some complex joins on the source side +CREATE TABLE cj_target (tid integer, balance float, val text); +CREATE TABLE cj_source1 (sid1 integer, scat integer, delta integer); +CREATE TABLE cj_source2 (sid2 integer, sval text); +INSERT INTO cj_source1 VALUES (1, 10, 100); +INSERT INTO cj_source1 VALUES (1, 20, 200); +INSERT INTO cj_source1 VALUES (2, 20, 300); +INSERT INTO cj_source1 VALUES (3, 10, 400); +INSERT INTO cj_source2 VALUES (1, 'initial source2'); +INSERT INTO cj_source2 VALUES (2, 'initial source2'); +INSERT INTO cj_source2 VALUES (3, 'initial source2'); +-- source relation is an unalised join +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid1, delta, sval); +-- try accessing columns from either side of the source join +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta, sval) +WHEN MATCHED THEN + DELETE; +-- some simple expressions in INSERT targetlist +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta + scat, sval) +WHEN MATCHED THEN + UPDATE SET val = val || ' updated by merge'; +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN MATCHED THEN + UPDATE SET val = val || ' ' || delta::text; +SELECT * FROM cj_target; + tid | balance | val +-----+---------+---------------------------------- + 3 | 400 | initial source2 updated by merge + 1 | 220 | initial source2 200 + 1 | 110 | initial source2 200 + 2 | 320 | initial source2 300 +(4 rows) + +ALTER TABLE cj_source1 RENAME COLUMN sid1 TO sid; +ALTER TABLE cj_source2 RENAME COLUMN sid2 TO sid; +TRUNCATE cj_target; +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON s1.sid = s2.sid +ON t.tid = s1.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s2.sid, delta, sval); +DROP TABLE cj_source2, cj_source1, cj_target; +-- Function scans +CREATE TABLE fs_target (a int, b int, c text); +MERGE INTO fs_target t +USING generate_series(1,100,1) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1); +MERGE INTO fs_target t +USING generate_series(1,100,2) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id, c = 'updated '|| id.*::text +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1, 'inserted ' || id.*::text); +SELECT count(*) FROM fs_target; + count +------- + 100 +(1 row) + +DROP TABLE fs_target; +-- SERIALIZABLE test +-- handled in isolation tests +-- prepare +RESET SESSION AUTHORIZATION; +DROP TABLE target, target2; +DROP TABLE source, source2; +DROP FUNCTION merge_trigfunc(); +DROP USER merge_privs; +DROP USER merge_no_privs; diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index ac8968d24f..864f2c1345 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -517,6 +517,104 @@ SELECT atest6 FROM atest6; -- ok (0 rows) COPY atest6 TO stdout; -- ok +-- test column privileges with MERGE +SET SESSION AUTHORIZATION regress_priv_user1; +CREATE TABLE mtarget (a int, b text); +CREATE TABLE msource (a int, b text); +INSERT INTO mtarget VALUES (1, 'init1'), (2, 'init2'); +INSERT INTO msource VALUES (1, 'source1'), (2, 'source2'), (3, 'source3'); +GRANT SELECT (a) ON msource TO regress_priv_user4; +GRANT SELECT (a) ON mtarget TO regress_priv_user4; +GRANT INSERT (a,b) ON mtarget TO regress_priv_user4; +GRANT UPDATE (b) ON mtarget TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; +-- +-- test source privileges +-- +-- fail (no SELECT priv on s.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table msource +-- fail (s.b used in the INSERTed values) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table msource +-- fail (s.b used in the WHEN quals) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND s.b = 'x' THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table msource +-- this should be ok since only s.a is accessed +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'ok' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ROLLBACK; +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT SELECT (b) ON msource TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; +-- should now be ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ROLLBACK; +-- +-- test target privileges +-- +-- fail (no SELECT priv on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ERROR: permission denied for table mtarget +-- fail (no UPDATE on t.a) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b, a = t.a + 1 +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table mtarget +-- fail (no SELECT on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ERROR: permission denied for table mtarget +-- ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b; +ROLLBACK; +-- fail (no DELETE) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ERROR: permission denied for table mtarget +-- grant delete privileges +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT DELETE ON mtarget TO regress_priv_user4; +-- should be ok now +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ROLLBACK; -- check error reporting with column privs SET SESSION AUTHORIZATION regress_priv_user1; CREATE TABLE t1 (c1 int, c2 int, c3 int check (c3 < 5), primary key (c1, c2)); diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index f1ae40df61..bf7af3ba82 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -2138,6 +2138,188 @@ ERROR: new row violates row-level security policy (USING expression) for table INSERT INTO document VALUES (1, (SELECT cid from category WHERE cname = 'novel'), 1, 'regress_rls_bob', 'my first novel') ON CONFLICT (did) DO UPDATE SET dauthor = 'regress_rls_carol'; ERROR: new row violates row-level security policy for table "document" +-- +-- MERGE +-- +RESET SESSION AUTHORIZATION; +DROP POLICY p3_with_all ON document; +ALTER TABLE document ADD COLUMN dnotes text DEFAULT ''; +-- all documents are readable +CREATE POLICY p1 ON document FOR SELECT USING (true); +-- one may insert documents only authored by them +CREATE POLICY p2 ON document FOR INSERT WITH CHECK (dauthor = current_user); +-- one may only update documents in 'novel' category +CREATE POLICY p3 ON document FOR UPDATE + USING (cid = (SELECT cid from category WHERE cname = 'novel')) + WITH CHECK (dauthor = current_user); +-- one may only delete documents in 'manga' category +CREATE POLICY p4 ON document FOR DELETE + USING (cid = (SELECT cid from category WHERE cname = 'manga')); +SELECT * FROM document; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-------------------+----------------------------------+-------- + 1 | 11 | 1 | regress_rls_bob | my first novel | + 3 | 22 | 2 | regress_rls_bob | my science fiction | + 4 | 44 | 1 | regress_rls_bob | my first manga | + 5 | 44 | 2 | regress_rls_bob | my second manga | + 6 | 22 | 1 | regress_rls_carol | great science fiction | + 7 | 33 | 2 | regress_rls_carol | great technology book | + 8 | 44 | 1 | regress_rls_carol | great manga | + 9 | 22 | 1 | regress_rls_dave | awesome science fiction | + 10 | 33 | 2 | regress_rls_dave | awesome technology book | + 11 | 33 | 1 | regress_rls_carol | hoge | + 33 | 22 | 1 | regress_rls_bob | okay science fiction | + 2 | 11 | 2 | regress_rls_bob | my first novel | + 78 | 33 | 1 | regress_rls_bob | some technology novel | + 79 | 33 | 1 | regress_rls_bob | technology book, can only insert | +(14 rows) + +SET SESSION AUTHORIZATION regress_rls_bob; +-- Fails, since update violates WITH CHECK qual on dauthor +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge1 ', dauthor = 'regress_rls_alice'; +ERROR: new row violates row-level security policy for table "document" +-- Should be OK since USING and WITH CHECK quals pass +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge2 '; +-- Even when dauthor is updated explicitly, but to the existing value +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge3 ', dauthor = 'regress_rls_bob'; +-- There is a MATCH for did = 3, but UPDATE's USING qual does not allow +-- updating an item in category 'science fiction' +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge '; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- The same thing with DELETE action, but fails again because no permissions +-- to delete items in 'science fiction' category that did 3 belongs to. +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- Document with did 4 belongs to 'manga' category which is allowed for +-- deletion. But this fails because the UPDATE action is matched first and +-- UPDATE policy does not allow updation in the category. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes = '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +-- UPDATE action is not matched this time because of the WHEN AND qual. +-- DELETE still fails because role regress_rls_bob does not have SELECT +-- privileges on 'manga' category row in the category table. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +ERROR: target row violates row-level security policy (USING expression) for table "document" +SELECT * FROM document WHERE did = 4; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-----------------+----------------+-------- + 4 | 44 | 1 | regress_rls_bob | my first manga | +(1 row) + +-- Switch to regress_rls_carol role and try the DELETE again. It should succeed +-- this time +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_carol; +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; +-- Switch back to regress_rls_bob role +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_bob; +-- Try INSERT action. This fails because we are trying to insert +-- dauthor = regress_rls_dave and INSERT's WITH CHECK does not allow +-- that +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_dave', 'another novel'); +ERROR: new row violates row-level security policy for table "document" +-- This should be fine +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +-- ok +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge4 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +-- drop and create a new SELECT policy which prevents us from reading +-- any document except with category 'magna' +RESET SESSION AUTHORIZATION; +DROP POLICY p1 ON document; +CREATE POLICY p1 ON document FOR SELECT + USING (cid = (SELECT cid from category WHERE cname = 'manga')); +SET SESSION AUTHORIZATION regress_rls_bob; +-- MERGE can no longer see the matching row and hence attempts the +-- NOT MATCHED action, which results in unique key violation +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge5 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); +ERROR: duplicate key value violates unique constraint "document_pkey" +RESET SESSION AUTHORIZATION; +-- drop the restrictive SELECT policy so that we can look at the +-- final state of the table +DROP POLICY p1 ON document; +-- Just check everything went per plan +SELECT * FROM document; + did | cid | dlevel | dauthor | dtitle | dnotes +-----+-----+--------+-------------------+----------------------------------+----------------------------------------------------------------------- + 3 | 22 | 2 | regress_rls_bob | my science fiction | + 5 | 44 | 2 | regress_rls_bob | my second manga | + 6 | 22 | 1 | regress_rls_carol | great science fiction | + 7 | 33 | 2 | regress_rls_carol | great technology book | + 8 | 44 | 1 | regress_rls_carol | great manga | + 9 | 22 | 1 | regress_rls_dave | awesome science fiction | + 10 | 33 | 2 | regress_rls_dave | awesome technology book | + 11 | 33 | 1 | regress_rls_carol | hoge | + 33 | 22 | 1 | regress_rls_bob | okay science fiction | + 2 | 11 | 2 | regress_rls_bob | my first novel | + 78 | 33 | 1 | regress_rls_bob | some technology novel | + 79 | 33 | 1 | regress_rls_bob | technology book, can only insert | + 12 | 11 | 1 | regress_rls_bob | another novel | + 1 | 11 | 1 | regress_rls_bob | my first novel | notes added by merge2 notes added by merge3 notes added by merge4 +(14 rows) + -- -- ROLE/GROUP -- diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 5149b72fe9..d369a73173 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -3263,6 +3263,37 @@ CREATE RULE rules_parted_table_insert AS ON INSERT to rules_parted_table ALTER RULE rules_parted_table_insert ON rules_parted_table RENAME TO rules_parted_table_insert_redirect; DROP TABLE rules_parted_table; -- +-- test MERGE +-- +CREATE TABLE rule_merge1 (a int, b text); +CREATE TABLE rule_merge2 (a int, b text); +CREATE RULE rule1 AS ON INSERT TO rule_merge1 + DO INSTEAD INSERT INTO rule_merge2 VALUES (NEW.*); +CREATE RULE rule2 AS ON UPDATE TO rule_merge1 + DO INSTEAD UPDATE rule_merge2 SET a = NEW.a, b = NEW.b + WHERE a = OLD.a; +CREATE RULE rule3 AS ON DELETE TO rule_merge1 + DO INSTEAD DELETE FROM rule_merge2 WHERE a = OLD.a; +-- MERGE not supported for table with rules +MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); +ERROR: MERGE is not supported for relations with rules +-- should be ok with the other table though +MERGE INTO rule_merge2 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); +-- -- Test enabling/disabling -- CREATE TABLE ruletest1 (a int); diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out index 387e40d67d..7f4a94ef7d 100644 --- a/src/test/regress/expected/triggers.out +++ b/src/test/regress/expected/triggers.out @@ -2761,6 +2761,54 @@ delete from self_ref where a = 1; NOTICE: trigger_func(self_ref) called: action = DELETE, when = BEFORE, level = STATEMENT NOTICE: trigger = self_ref_s_trig, old table = (1,), (2,1), (3,2), (4,3) drop table self_ref; +-- +-- test transition tables with MERGE +-- +create table merge_target_table (a int primary key, b text); +create trigger merge_target_table_insert_trig + after insert on merge_target_table referencing new table as new_table + for each statement execute procedure dump_insert(); +create trigger merge_target_table_update_trig + after update on merge_target_table referencing old table as old_table new table as new_table + for each statement execute procedure dump_update(); +create trigger merge_target_table_delete_trig + after delete on merge_target_table referencing old table as old_table + for each statement execute procedure dump_delete(); +create table merge_source_table (a int, b text); +insert into merge_source_table + values (1, 'initial1'), (2, 'initial2'), + (3, 'initial3'), (4, 'initial4'); +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_insert_trig, new table = (1,initial1), (2,initial2), (3,initial3), (4,initial4) +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_delete_trig, old table = (3,initial3), (4,initial4) +NOTICE: trigger = merge_target_table_update_trig, old table = (1,initial1), (2,initial2), new table = (1,"initial1 updated by merge"), (2,"initial2 updated by merge") +NOTICE: trigger = merge_target_table_insert_trig, new table = <NULL> +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated again by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); +NOTICE: trigger = merge_target_table_delete_trig, old table = <NULL> +NOTICE: trigger = merge_target_table_update_trig, old table = (1,"initial1 updated by merge"), (2,"initial2 updated by merge"), new table = (1,"initial1 updated by merge updated again by merge"), (2,"initial2 updated by merge updated again by merge") +NOTICE: trigger = merge_target_table_insert_trig, new table = (3,initial3), (4,initial4) +drop table merge_source_table, merge_target_table; -- cleanup drop function dump_insert(); drop function dump_update(); diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index d858a0e7db..20d6745730 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -84,7 +84,7 @@ test: select_into select_distinct select_distinct_on select_implicit select_havi # ---------- # Another group of parallel tests # ---------- -test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password func_index +test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password func_index merge # ---------- # Another group of parallel tests diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 99f8ca37ba..a08169f256 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -123,6 +123,7 @@ test: tablesample test: groupingsets test: drop_operator test: password +test: merge test: alter_generic test: alter_operator test: misc diff --git a/src/test/regress/sql/identity.sql b/src/test/regress/sql/identity.sql index a35f331f4e..f8f34eaf18 100644 --- a/src/test/regress/sql/identity.sql +++ b/src/test/regress/sql/identity.sql @@ -246,3 +246,48 @@ CREATE TABLE itest_child PARTITION OF itest_parent ( f3 WITH OPTIONS GENERATED ALWAYS AS IDENTITY ) FOR VALUES FROM ('2016-07-01') TO ('2016-08-01'); -- error DROP TABLE itest_parent; + +-- MERGE tests +CREATE TABLE itest14 (a int GENERATED ALWAYS AS IDENTITY, b text); +CREATE TABLE itest15 (a int GENERATED BY DEFAULT AS IDENTITY, b text); + +MERGE INTO itest14 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); + +MERGE INTO itest14 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest14 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 10 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 20 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING USER VALUE VALUES (s.s_a, s.s_b); + +MERGE INTO itest15 t +USING (SELECT 30 AS s_a, 'inserted by merge' AS s_b) s +ON t.a = s.s_a +WHEN NOT MATCHED THEN + INSERT (a, b) OVERRIDING SYSTEM VALUE VALUES (s.s_a, s.s_b); + +SELECT * FROM itest14; +SELECT * FROM itest15; +DROP TABLE itest14; +DROP TABLE itest15; diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql new file mode 100644 index 0000000000..8b5244fc63 --- /dev/null +++ b/src/test/regress/sql/merge.sql @@ -0,0 +1,1068 @@ +-- +-- MERGE +-- +--\set VERBOSITY verbose + +--set debug_print_rewritten = true; +--set debug_print_parse = true; +--set debug_print_pretty = true; + + +CREATE USER merge_privs; +CREATE USER merge_no_privs; +DROP TABLE IF EXISTS target; +DROP TABLE IF EXISTS source; +CREATE TABLE target (tid integer, balance integer); +CREATE TABLE source (sid integer, delta integer); --no index +INSERT INTO target VALUES (1, 10); +INSERT INTO target VALUES (2, 20); +INSERT INTO target VALUES (3, 30); +SELECT t.ctid is not null as matched, t.*, s.* FROM source s FULL OUTER JOIN target t ON s.sid = t.tid ORDER BY t.tid, s.sid; + +ALTER TABLE target OWNER TO merge_privs; +ALTER TABLE source OWNER TO merge_privs; + +CREATE TABLE target2 (tid integer, balance integer); +CREATE TABLE source2 (sid integer, delta integer); + +ALTER TABLE target2 OWNER TO merge_no_privs; +ALTER TABLE source2 OWNER TO merge_no_privs; + +GRANT INSERT ON target TO merge_no_privs; + +SET SESSION AUTHORIZATION merge_privs; + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; + +-- +-- Errors +-- +MERGE INTO target t RANDOMWORD +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +-- MATCHED/INSERT error +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + INSERT DEFAULT VALUES +; +-- incorrectly specifying INTO target +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT INTO target DEFAULT VALUES +; +-- NOT MATCHED/UPDATE +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + UPDATE SET balance = 0 +; +-- UPDATE tablename +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE target SET balance = 0 +; + +-- permissions + +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +GRANT INSERT ON target TO merge_no_privs; +SET SESSION AUTHORIZATION merge_no_privs; + +MERGE INTO target +USING source2 +ON target.tid = source2.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +GRANT UPDATE ON target2 TO merge_privs; +SET SESSION AUTHORIZATION merge_privs; + +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN MATCHED THEN + DELETE +; + +MERGE INTO target2 +USING source +ON target2.tid = source.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; + +-- check if the target can be accessed from source relation subquery; we should +-- not be able to do so +MERGE INTO target t +USING (SELECT * FROM source WHERE t.tid > sid) s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; + +-- +-- initial tests +-- +-- zero rows in source has no effect +MERGE INTO target +USING source +ON target.tid = source.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +ROLLBACK; + +-- insert some non-matching source rows to work from +INSERT INTO source VALUES (4, 40); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + DO NOTHING +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT DEFAULT VALUES +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- index plans +INSERT INTO target SELECT generate_series(1000,2500), 0; +ALTER TABLE target ADD PRIMARY KEY (tid); +ANALYZE target; + +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +EXPLAIN (COSTS OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL); +; +DELETE FROM target WHERE tid > 100; +ANALYZE target; + +-- insert some matching source rows to work from +INSERT INTO source VALUES (2, 5); +INSERT INTO source VALUES (3, 20); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +-- equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- equivalent of a DELETE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, NULL) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- duplicate source row causes multiple target row update ERROR +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + DELETE +; +ROLLBACK; + +-- correct source data +DELETE FROM source WHERE sid = 2; +INSERT INTO source VALUES (2, 5); +SELECT * FROM source ORDER BY sid; +SELECT * FROM target ORDER BY tid; + +-- remove constraints +alter table target drop CONSTRAINT target_pkey; +alter table target alter column tid drop not null; + +-- multiple actions +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4) +WHEN MATCHED THEN + UPDATE SET balance = 0 +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- should be equivalent +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = 0 +WHEN NOT MATCHED THEN + INSERT VALUES (4, 4); +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- column references +-- do a simple equivalent of an UPDATE join +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- do a simple equivalent of an INSERT SELECT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- and again with explicitly identified column list +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- and again with a subtle error: referring to non-existent target row for NOT MATCHED +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; + +-- and again with a constant ON clause +BEGIN; +MERGE INTO target t +USING source AS s +ON (SELECT true) +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (t.tid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- now the classic UPSERT +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance + s.delta +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- unreachable WHEN clause should ERROR +BEGIN; +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED THEN /* Terminal WHEN clause for MATCHED */ + DELETE +WHEN MATCHED AND s.delta > 0 THEN + UPDATE SET balance = t.balance - s.delta +; +ROLLBACK; + +-- conditional WHEN clause +CREATE TABLE wq_target (tid integer not null, balance integer DEFAULT -1); +CREATE TABLE wq_source (balance integer, sid integer); + +INSERT INTO wq_source (sid, balance) VALUES (1, 100); + +BEGIN; +-- try a simple INSERT with default values first +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +-- this time with a FALSE condition +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND FALSE THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +-- this time with an actual condition which returns false +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance <> 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +BEGIN; +-- and now with a condition which returns true +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +-- conditions in the NOT MATCHED clause can only refer to source columns +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND t.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; +ROLLBACK; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN NOT MATCHED AND s.balance = 100 THEN + INSERT (tid) VALUES (s.sid); +SELECT * FROM wq_target; + +-- conditions in MATCHED clause can refer to both source and target +SELECT * FROM wq_source; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if AND works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 AND s.balance = 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if OR works +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 99 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance = 199 OR s.balance > 100 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- check if subqueries work in the conditions? +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.balance > (SELECT max(balance) FROM target) THEN + UPDATE SET balance = t.balance + s.balance; + +-- check if we can access system columns in the conditions +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.xmin = t.xmax THEN + UPDATE SET balance = t.balance + s.balance; + +ALTER TABLE wq_target SET WITH OIDS; +SELECT * FROM wq_target; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND t.oid >= 0 THEN + UPDATE SET balance = t.balance + s.balance; +SELECT * FROM wq_target; + +-- test preventing WHEN AND conditions from writing to the database +create or replace function merge_when_and_write() returns boolean +language plpgsql as +$$ +BEGIN + INSERT INTO target VALUES (100, 100); + RETURN TRUE; +END; +$$; + +BEGIN; +MERGE INTO wq_target t +USING wq_source s ON t.tid = s.sid +WHEN MATCHED AND (merge_when_and_write()) THEN + UPDATE SET balance = t.balance + s.balance; +ROLLBACK; +drop function merge_when_and_write(); + +DROP TABLE wq_target, wq_source; + +-- test triggers +create or replace function merge_trigfunc () returns trigger +language plpgsql as +$$ +BEGIN + RAISE NOTICE '% % % trigger', TG_WHEN, TG_OP, TG_LEVEL; + IF (TG_WHEN = 'BEFORE' AND TG_LEVEL = 'ROW') THEN + IF (TG_OP = 'DELETE') THEN + RETURN OLD; + ELSE + RETURN NEW; + END IF; + ELSE + RETURN NULL; + END IF; +END; +$$; +CREATE TRIGGER merge_bsi BEFORE INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsu BEFORE UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bsd BEFORE DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asi AFTER INSERT ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asu AFTER UPDATE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_asd AFTER DELETE ON target FOR EACH STATEMENT EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bri BEFORE INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_bru BEFORE UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_brd BEFORE DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ari AFTER INSERT ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_aru AFTER UPDATE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); +CREATE TRIGGER merge_ard AFTER DELETE ON target FOR EACH ROW EXECUTE PROCEDURE merge_trigfunc (); + +-- now the classic UPSERT, with a DELETE +BEGIN; +UPDATE target SET balance = 0 WHERE tid = 3; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- test from PL/pgSQL +-- make sure MERGE INTO isn't interpreted to mean returning variables like SELECT INTO +BEGIN; +DO LANGUAGE plpgsql $$ +BEGIN +MERGE INTO target t +USING source AS s +ON t.tid = s.sid +WHEN MATCHED AND t.balance > s.delta THEN + UPDATE SET balance = t.balance - s.delta +; +END; +$$; +ROLLBACK; + +--source constants +BEGIN; +MERGE INTO target t +USING (SELECT 9 AS sid, 57 AS delta) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +--source query +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING (SELECT sid, delta as newname FROM source WHERE delta > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.newname) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +--self-merge +BEGIN; +MERGE INTO target t1 +USING target t2 +ON t1.tid = t2.tid +WHEN MATCHED THEN + UPDATE SET balance = t1.balance + t2.balance +WHEN NOT MATCHED THEN + INSERT VALUES (t2.tid, t2.balance) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING (SELECT tid as sid, balance as delta FROM target WHERE balance > 0) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +MERGE INTO target t +USING +(SELECT sid, max(delta) AS delta + FROM source + GROUP BY sid + HAVING count(*) = 1 + ORDER BY sid ASC) AS s +ON t.tid = s.sid +WHEN NOT MATCHED THEN + INSERT (tid, balance) VALUES (s.sid, s.delta) +; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- plpgsql parameters and results +BEGIN; +CREATE FUNCTION merge_func (p_id integer, p_bal integer) +RETURNS INTEGER +LANGUAGE plpgsql +AS $$ +DECLARE + result integer; +BEGIN +MERGE INTO target t +USING (SELECT p_id AS sid) AS s +ON t.tid = s.sid +WHEN MATCHED THEN + UPDATE SET balance = t.balance - p_bal +; +IF FOUND THEN + GET DIAGNOSTICS result := ROW_COUNT; +END IF; +RETURN result; +END; +$$; +SELECT merge_func(3, 4); +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- PREPARE +BEGIN; +prepare foom as merge into target t using (select 1 as sid) s on (t.tid = s.sid) when matched then update set balance = 1; +execute foom; +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +BEGIN; +PREPARE foom2 (integer, integer) AS +MERGE INTO target t +USING (SELECT 1) s +ON t.tid = $1 +WHEN MATCHED THEN +UPDATE SET balance = $2; +EXPLAIN (ANALYZE ON, COSTS OFF, SUMMARY OFF, TIMING OFF) +execute foom2 (1, 1); +SELECT * FROM target ORDER BY tid; +ROLLBACK; + +-- subqueries in source relation + +CREATE TABLE sq_target (tid integer NOT NULL, balance integer); +CREATE TABLE sq_source (delta integer, sid integer, balance integer DEFAULT 0); + +INSERT INTO sq_target(tid, balance) VALUES (1,100), (2,200), (3,300); +INSERT INTO sq_source(sid, delta) VALUES (1,10), (2,20), (4,40); + +BEGIN; +MERGE INTO sq_target t +USING (SELECT * FROM sq_source) s +ON tid = sid +WHEN MATCHED AND t.balance > delta THEN + UPDATE SET balance = t.balance + delta; +SELECT * FROM sq_target; +ROLLBACK; + +-- try a view +CREATE VIEW v AS SELECT * FROM sq_source WHERE sid < 2; + +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = v.balance + delta; +SELECT * FROM sq_target; +ROLLBACK; + +-- ambiguous reference to a column +BEGIN; +MERGE INTO sq_target +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +ROLLBACK; + +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE; +SELECT * FROM sq_target; +ROLLBACK; + +-- CTEs +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +WITH targq AS ( + SELECT * FROM v +) +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +; +ROLLBACK; + +-- RETURNING +BEGIN; +INSERT INTO sq_source (sid, balance, delta) VALUES (-1, -1, -10); +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND tid > 2 THEN + UPDATE SET balance = t.balance + delta +WHEN NOT MATCHED THEN + INSERT (balance, tid) VALUES (balance + delta, sid) +WHEN MATCHED AND tid < 2 THEN + DELETE +RETURNING * +; +ROLLBACK; + +-- Subqueries +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED THEN + UPDATE SET balance = (SELECT count(*) FROM sq_target) +; +ROLLBACK; + +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid +WHEN MATCHED AND (SELECT count(*) > 0 FROM sq_target) THEN + UPDATE SET balance = 42 +; +ROLLBACK; + +BEGIN; +MERGE INTO sq_target t +USING v +ON tid = sid AND (SELECT count(*) > 0 FROM sq_target) +WHEN MATCHED THEN + UPDATE SET balance = 42 +; +ROLLBACK; + +DROP TABLE sq_target, sq_source CASCADE; + +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); + +CREATE TABLE part1 PARTITION OF pa_target FOR VALUES IN (1,4); +CREATE TABLE part2 PARTITION OF pa_target FOR VALUES IN (2,5,6); +CREATE TABLE part3 PARTITION OF pa_target FOR VALUES IN (3,8,9); +CREATE TABLE part4 PARTITION OF pa_target DEFAULT; + +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_target CASCADE; + +-- The target table is partitioned in the same way, but this time by attaching +-- partitions which have columns in different order, dropped columns etc. +CREATE TABLE pa_target (tid integer, balance float, val text) + PARTITION BY LIST (tid); +CREATE TABLE part1 (tid integer, balance float, val text); +CREATE TABLE part2 (balance float, tid integer, val text); +CREATE TABLE part3 (tid integer, balance float, val text); +CREATE TABLE part4 (extraid text, tid integer, balance float, val text); +ALTER TABLE part4 DROP COLUMN extraid; + +ALTER TABLE pa_target ATTACH PARTITION part1 FOR VALUES IN (1,4); +ALTER TABLE pa_target ATTACH PARTITION part2 FOR VALUES IN (2,5,6); +ALTER TABLE pa_target ATTACH PARTITION part3 FOR VALUES IN (3,8,9); +ALTER TABLE pa_target ATTACH PARTITION part4 DEFAULT; + +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT id, id * 100, 'initial' FROM generate_series(1,14,2) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- same with a constant qual +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid AND tid = 1 + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +-- try updating the partition key column +BEGIN; +MERGE INTO pa_target t + USING pa_source s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; + +-- Sub-partitionin +CREATE TABLE pa_target (logts timestamp, tid integer, balance float, val text) + PARTITION BY RANGE (logts); + +CREATE TABLE part_m01 PARTITION OF pa_target + FOR VALUES FROM ('2017-01-01') TO ('2017-02-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m01_odd PARTITION OF part_m01 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m01_even PARTITION OF part_m01 + FOR VALUES IN (2,4,6,8); +CREATE TABLE part_m02 PARTITION OF pa_target + FOR VALUES FROM ('2017-02-01') TO ('2017-03-01') + PARTITION BY LIST (tid); +CREATE TABLE part_m02_odd PARTITION OF part_m02 + FOR VALUES IN (1,3,5,7,9); +CREATE TABLE part_m02_even PARTITION OF part_m02 + FOR VALUES IN (2,4,6,8); + +CREATE TABLE pa_source (sid integer, delta float); +-- insert many rows to the source table +INSERT INTO pa_source SELECT id, id * 10 FROM generate_series(1,14) AS id; +-- insert a few rows in the target table (odd numbered tid) +INSERT INTO pa_target SELECT '2017-01-31', id, id * 100, 'initial' FROM generate_series(1,9,3) AS id; +INSERT INTO pa_target SELECT '2017-02-28', id, id * 100, 'initial' FROM generate_series(2,9,3) AS id; + +-- try simple MERGE +BEGIN; +MERGE INTO pa_target t + USING (SELECT '2017-01-15' AS slogts, * FROM pa_source WHERE sid < 10) s + ON t.tid = s.sid + WHEN MATCHED THEN + UPDATE SET balance = balance + delta, val = val || ' updated by merge' + WHEN NOT MATCHED THEN + INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge'); +SELECT * FROM pa_target ORDER BY tid; +ROLLBACK; + +DROP TABLE pa_source; +DROP TABLE pa_target CASCADE; + +-- some complex joins on the source side + +CREATE TABLE cj_target (tid integer, balance float, val text); +CREATE TABLE cj_source1 (sid1 integer, scat integer, delta integer); +CREATE TABLE cj_source2 (sid2 integer, sval text); +INSERT INTO cj_source1 VALUES (1, 10, 100); +INSERT INTO cj_source1 VALUES (1, 20, 200); +INSERT INTO cj_source1 VALUES (2, 20, 300); +INSERT INTO cj_source1 VALUES (3, 10, 400); +INSERT INTO cj_source2 VALUES (1, 'initial source2'); +INSERT INTO cj_source2 VALUES (2, 'initial source2'); +INSERT INTO cj_source2 VALUES (3, 'initial source2'); + +-- source relation is an unalised join +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid1, delta, sval); + +-- try accessing columns from either side of the source join +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta, sval) +WHEN MATCHED THEN + DELETE; + +-- some simple expressions in INSERT targetlist +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 +ON t.tid = sid1 +WHEN NOT MATCHED THEN + INSERT VALUES (sid2, delta + scat, sval) +WHEN MATCHED THEN + UPDATE SET val = val || ' updated by merge'; + +MERGE INTO cj_target t +USING cj_source2 s2 + INNER JOIN cj_source1 s1 ON sid1 = sid2 AND scat = 20 +ON t.tid = sid1 +WHEN MATCHED THEN + UPDATE SET val = val || ' ' || delta::text; + +SELECT * FROM cj_target; + +ALTER TABLE cj_source1 RENAME COLUMN sid1 TO sid; +ALTER TABLE cj_source2 RENAME COLUMN sid2 TO sid; + +TRUNCATE cj_target; + +MERGE INTO cj_target t +USING cj_source1 s1 + INNER JOIN cj_source2 s2 ON s1.sid = s2.sid +ON t.tid = s1.sid +WHEN NOT MATCHED THEN + INSERT VALUES (s2.sid, delta, sval); + +DROP TABLE cj_source2, cj_source1, cj_target; + +-- Function scans +CREATE TABLE fs_target (a int, b int, c text); +MERGE INTO fs_target t +USING generate_series(1,100,1) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1); + +MERGE INTO fs_target t +USING generate_series(1,100,2) AS id +ON t.a = id +WHEN MATCHED THEN + UPDATE SET b = b + id, c = 'updated '|| id.*::text +WHEN NOT MATCHED THEN + INSERT VALUES (id, -1, 'inserted ' || id.*::text); + +SELECT count(*) FROM fs_target; +DROP TABLE fs_target; + +-- SERIALIZABLE test +-- handled in isolation tests + +-- prepare + +RESET SESSION AUTHORIZATION; +DROP TABLE target, target2; +DROP TABLE source, source2; +DROP FUNCTION merge_trigfunc(); +DROP USER merge_privs; +DROP USER merge_no_privs; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index f7f3bbbeeb..0a8abf2076 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -349,6 +349,114 @@ UPDATE atest5 SET one = 1; -- fail SELECT atest6 FROM atest6; -- ok COPY atest6 TO stdout; -- ok +-- test column privileges with MERGE +SET SESSION AUTHORIZATION regress_priv_user1; +CREATE TABLE mtarget (a int, b text); +CREATE TABLE msource (a int, b text); +INSERT INTO mtarget VALUES (1, 'init1'), (2, 'init2'); +INSERT INTO msource VALUES (1, 'source1'), (2, 'source2'), (3, 'source3'); + +GRANT SELECT (a) ON msource TO regress_priv_user4; +GRANT SELECT (a) ON mtarget TO regress_priv_user4; +GRANT INSERT (a,b) ON mtarget TO regress_priv_user4; +GRANT UPDATE (b) ON mtarget TO regress_priv_user4; + +SET SESSION AUTHORIZATION regress_priv_user4; + +-- +-- test source privileges +-- + +-- fail (no SELECT priv on s.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- fail (s.b used in the INSERTed values) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- fail (s.b used in the WHEN quals) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND s.b = 'x' THEN + UPDATE SET b = 'x' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- this should be ok since only s.a is accessed +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = 'ok' +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); +ROLLBACK; + +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT SELECT (b) ON msource TO regress_priv_user4; +SET SESSION AUTHORIZATION regress_priv_user4; + +-- should now be ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); +ROLLBACK; + +-- +-- test target privileges +-- + +-- fail (no SELECT priv on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = t.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, NULL); + +-- fail (no UPDATE on t.a) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b, a = t.a + 1 +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- fail (no SELECT on t.b) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + UPDATE SET b = s.b +WHEN NOT MATCHED THEN + INSERT VALUES (a, b); + +-- ok +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED THEN + UPDATE SET b = s.b; +ROLLBACK; + +-- fail (no DELETE) +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; + +-- grant delete privileges +SET SESSION AUTHORIZATION regress_priv_user1; +GRANT DELETE ON mtarget TO regress_priv_user4; +-- should be ok now +BEGIN; +MERGE INTO mtarget t USING msource s ON t.a = s.a +WHEN MATCHED AND t.b IS NOT NULL THEN + DELETE; +ROLLBACK; + -- check error reporting with column privs SET SESSION AUTHORIZATION regress_priv_user1; CREATE TABLE t1 (c1 int, c2 int, c3 int check (c3 < 5), primary key (c1, c2)); diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql index f3a31dbee0..6c75208998 100644 --- a/src/test/regress/sql/rowsecurity.sql +++ b/src/test/regress/sql/rowsecurity.sql @@ -812,6 +812,162 @@ INSERT INTO document VALUES (4, (SELECT cid from category WHERE cname = 'novel') INSERT INTO document VALUES (1, (SELECT cid from category WHERE cname = 'novel'), 1, 'regress_rls_bob', 'my first novel') ON CONFLICT (did) DO UPDATE SET dauthor = 'regress_rls_carol'; +-- +-- MERGE +-- +RESET SESSION AUTHORIZATION; +DROP POLICY p3_with_all ON document; + +ALTER TABLE document ADD COLUMN dnotes text DEFAULT ''; +-- all documents are readable +CREATE POLICY p1 ON document FOR SELECT USING (true); +-- one may insert documents only authored by them +CREATE POLICY p2 ON document FOR INSERT WITH CHECK (dauthor = current_user); +-- one may only update documents in 'novel' category +CREATE POLICY p3 ON document FOR UPDATE + USING (cid = (SELECT cid from category WHERE cname = 'novel')) + WITH CHECK (dauthor = current_user); +-- one may only delete documents in 'manga' category +CREATE POLICY p4 ON document FOR DELETE + USING (cid = (SELECT cid from category WHERE cname = 'manga')); + +SELECT * FROM document; + +SET SESSION AUTHORIZATION regress_rls_bob; + +-- Fails, since update violates WITH CHECK qual on dauthor +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge1 ', dauthor = 'regress_rls_alice'; + +-- Should be OK since USING and WITH CHECK quals pass +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge2 '; + +-- Even when dauthor is updated explicitly, but to the existing value +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge3 ', dauthor = 'regress_rls_bob'; + +-- There is a MATCH for did = 3, but UPDATE's USING qual does not allow +-- updating an item in category 'science fiction' +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge '; + +-- The same thing with DELETE action, but fails again because no permissions +-- to delete items in 'science fiction' category that did 3 belongs to. +MERGE INTO document d +USING (SELECT 3 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE; + +-- Document with did 4 belongs to 'manga' category which is allowed for +-- deletion. But this fails because the UPDATE action is matched first and +-- UPDATE policy does not allow updation in the category. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes = '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +-- UPDATE action is not matched this time because of the WHEN AND qual. +-- DELETE still fails because role regress_rls_bob does not have SELECT +-- privileges on 'manga' category row in the category table. +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +SELECT * FROM document WHERE did = 4; + +-- Switch to regress_rls_carol role and try the DELETE again. It should succeed +-- this time +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_carol; + +MERGE INTO document d +USING (SELECT 4 as sdid) s +ON did = s.sdid +WHEN MATCHED AND dnotes <> '' THEN + UPDATE SET dnotes = dnotes || ' notes added by merge ' +WHEN MATCHED THEN + DELETE; + +-- Switch back to regress_rls_bob role +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_rls_bob; + +-- Try INSERT action. This fails because we are trying to insert +-- dauthor = regress_rls_dave and INSERT's WITH CHECK does not allow +-- that +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_dave', 'another novel'); + +-- This should be fine +MERGE INTO document d +USING (SELECT 12 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + DELETE +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +-- ok +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge4 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +-- drop and create a new SELECT policy which prevents us from reading +-- any document except with category 'magna' +RESET SESSION AUTHORIZATION; +DROP POLICY p1 ON document; +CREATE POLICY p1 ON document FOR SELECT + USING (cid = (SELECT cid from category WHERE cname = 'manga')); + +SET SESSION AUTHORIZATION regress_rls_bob; + +-- MERGE can no longer see the matching row and hence attempts the +-- NOT MATCHED action, which results in unique key violation +MERGE INTO document d +USING (SELECT 1 as sdid) s +ON did = s.sdid +WHEN MATCHED THEN + UPDATE SET dnotes = dnotes || ' notes added by merge5 ' +WHEN NOT MATCHED THEN + INSERT VALUES (12, 11, 1, 'regress_rls_bob', 'another novel'); + +RESET SESSION AUTHORIZATION; +-- drop the restrictive SELECT policy so that we can look at the +-- final state of the table +DROP POLICY p1 ON document; +-- Just check everything went per plan +SELECT * FROM document; + -- -- ROLE/GROUP -- diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql index a82f52d154..b866268892 100644 --- a/src/test/regress/sql/rules.sql +++ b/src/test/regress/sql/rules.sql @@ -1191,6 +1191,39 @@ CREATE RULE rules_parted_table_insert AS ON INSERT to rules_parted_table ALTER RULE rules_parted_table_insert ON rules_parted_table RENAME TO rules_parted_table_insert_redirect; DROP TABLE rules_parted_table; +-- +-- test MERGE +-- +CREATE TABLE rule_merge1 (a int, b text); +CREATE TABLE rule_merge2 (a int, b text); +CREATE RULE rule1 AS ON INSERT TO rule_merge1 + DO INSTEAD INSERT INTO rule_merge2 VALUES (NEW.*); +CREATE RULE rule2 AS ON UPDATE TO rule_merge1 + DO INSTEAD UPDATE rule_merge2 SET a = NEW.a, b = NEW.b + WHERE a = OLD.a; +CREATE RULE rule3 AS ON DELETE TO rule_merge1 + DO INSTEAD DELETE FROM rule_merge2 WHERE a = OLD.a; + +-- MERGE not supported for table with rules +MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); + +-- should be ok with the other table though +MERGE INTO rule_merge2 t USING (SELECT 1 AS a) s + ON t.a = s.a + WHEN MATCHED AND t.a < 2 THEN + UPDATE SET b = b || ' updated by merge' + WHEN MATCHED AND t.a > 2 THEN + DELETE + WHEN NOT MATCHED THEN + INSERT VALUES (s.a, ''); + -- -- Test enabling/disabling -- diff --git a/src/test/regress/sql/triggers.sql b/src/test/regress/sql/triggers.sql index c6f31dd8c8..b51c884eee 100644 --- a/src/test/regress/sql/triggers.sql +++ b/src/test/regress/sql/triggers.sql @@ -2110,6 +2110,53 @@ delete from self_ref where a = 1; drop table self_ref; +-- +-- test transition tables with MERGE +-- +create table merge_target_table (a int primary key, b text); +create trigger merge_target_table_insert_trig + after insert on merge_target_table referencing new table as new_table + for each statement execute procedure dump_insert(); +create trigger merge_target_table_update_trig + after update on merge_target_table referencing old table as old_table new table as new_table + for each statement execute procedure dump_update(); +create trigger merge_target_table_delete_trig + after delete on merge_target_table referencing old table as old_table + for each statement execute procedure dump_delete(); + +create table merge_source_table (a int, b text); +insert into merge_source_table + values (1, 'initial1'), (2, 'initial2'), + (3, 'initial3'), (4, 'initial4'); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when not matched then + insert values (a, b); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); + +merge into merge_target_table t +using merge_source_table s +on t.a = s.a +when matched and s.a <= 2 then + update set b = t.b || ' updated again by merge' +when matched and s.a > 2 then + delete +when not matched then + insert values (a, b); + +drop table merge_source_table, merge_target_table; + -- cleanup drop function dump_insert(); drop function dump_update(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 17bf55c1f5..a7eb3524cf 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1228,6 +1228,8 @@ MemoryContextCallbackFunction MemoryContextCounters MemoryContextData MemoryContextMethods +MergeAction +MergeActionState MergeAppend MergeAppendPath MergeAppendState @@ -1235,6 +1237,7 @@ MergeJoin MergeJoinClause MergeJoinState MergePath +MergeStmt MergeScanSelCache MetaCommand MinMaxAggInfo -- 2.14.3 (Apple Git-98) ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:28 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-28 02:58 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-28 11:00 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> @ 2018-03-29 09:50 ` Simon Riggs <[email protected]> 2018-03-30 11:10 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Simon Riggs @ 2018-03-29 09:50 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 28 March 2018 at 12:00, Pavan Deolasee <[email protected]> wrote: > v27 attached, though review changes are in > the add-on 0005 patch. This all looks good now, thanks for making all of those changes. I propose [v27 patch1+patch3+patch5] as the initial commit candidate for MERGE, with other patches following later before end CF. I propose to commit this tomorrow, 30 March, about 26 hours from now. That will allow some time for buildfarm fixing/reversion before the Easter weekend, then other patches to follow starting 2 April. That then gives reasonable time to follow up on other issues that we will no doubt discover fairly soon after commit, such as additional runs by SQLsmith and more eyeballs. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 14:39 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-26 16:06 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-27 09:28 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-28 02:58 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-28 11:00 ` Re: [HACKERS] MERGE SQL Statement for PG11 Pavan Deolasee <[email protected]> 2018-03-29 09:50 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-30 11:10 ` Simon Riggs <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Simon Riggs @ 2018-03-30 11:10 UTC (permalink / raw) To: Pavan Deolasee <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 29 March 2018 at 10:50, Simon Riggs <[email protected]> wrote: > On 28 March 2018 at 12:00, Pavan Deolasee <[email protected]> wrote: > >> v27 attached, though review changes are in >> the add-on 0005 patch. > > This all looks good now, thanks for making all of those changes. > > I propose [v27 patch1+patch3+patch5] as the initial commit candidate > for MERGE, with other patches following later before end CF. > > I propose to commit this tomorrow, 30 March, about 26 hours from now. > That will allow some time for buildfarm fixing/reversion before the > Easter weekend, then other patches to follow starting 2 April. That > then gives reasonable time to follow up on other issues that we will > no doubt discover fairly soon after commit, such as additional runs by > SQLsmith and more eyeballs. No problems found, but moving proposed commit to 2 April pm -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-26 15:09 ` Robert Haas <[email protected]> 2018-03-26 16:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 1 sibling, 1 reply; 81+ messages in thread From: Robert Haas @ 2018-03-26 15:09 UTC (permalink / raw) To: Simon Riggs <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Mon, Mar 26, 2018 at 5:53 AM, Simon Riggs <[email protected]> wrote: > Since we now have MVCC catalog scans, all the name lookups are > performed using the same snapshot so in the above scenario the newly > created object would be invisible to the second name lookup. That's not true, because each lookup would be performed using a new snapshot -- not all under one snapshot. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 15:09 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> @ 2018-03-26 16:16 ` Simon Riggs <[email protected]> 2018-03-26 16:52 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Simon Riggs @ 2018-03-26 16:16 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 26 March 2018 at 16:09, Robert Haas <[email protected]> wrote: > On Mon, Mar 26, 2018 at 5:53 AM, Simon Riggs <[email protected]> wrote: >> Since we now have MVCC catalog scans, all the name lookups are >> performed using the same snapshot so in the above scenario the newly >> created object would be invisible to the second name lookup. > > That's not true, because each lookup would be performed using a new > snapshot -- not all under one snapshot. You're saying we take a separate snapshot for each table we lookup? Sounds weird to me. So this error could happen in SELECT, UPDATE, DELETE or INSERT as well. Or you see this as something related specifically to MERGE, if so how? Please explain what you see. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 15:09 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 16:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-26 16:52 ` Robert Haas <[email protected]> 2018-03-26 19:17 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Robert Haas @ 2018-03-26 16:52 UTC (permalink / raw) To: Simon Riggs <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Mon, Mar 26, 2018 at 12:16 PM, Simon Riggs <[email protected]> wrote: > On 26 March 2018 at 16:09, Robert Haas <[email protected]> wrote: >> On Mon, Mar 26, 2018 at 5:53 AM, Simon Riggs <[email protected]> wrote: >>> Since we now have MVCC catalog scans, all the name lookups are >>> performed using the same snapshot so in the above scenario the newly >>> created object would be invisible to the second name lookup. >> >> That's not true, because each lookup would be performed using a new >> snapshot -- not all under one snapshot. > > You're saying we take a separate snapshot for each table we lookup? > Sounds weird to me. I'm saying we take a separate snapshot for each and every catalog lookup, except when we know that no catalog changes can have occurred. See the commit message for 568d4138c646cd7cd8a837ac244ef2caf27c6bb8. If you do a lookup in pg_class and 3 lookups in pg_attribute each of the 4 can be done under a different snapshot, in the worst case. You're not the first person to believe that the MVCC catalog scan patch fixes that problem, but as the guy who wrote it, it definitely doesn't. What that patch fixed was, prior to that patch, a catalog scan might find the WRONG NUMBER OF ROWS, like you might do a lookup against a unique index for an object that existed and, if the row was concurrently updated, you might find 0 rows or 2 rows instead of 1 row. IOW, it guaranteed that we used a consistent snapshot for each individual lookup, not a consistent snapshot for the whole course of a command. > So this error could happen in SELECT, UPDATE, DELETE or INSERT as well. > > Or you see this as something related specifically to MERGE, if so how? > Please explain what you see. As I said before, the problem occurs if the same command looks up the same table name in more than one place. There is absolutely nothing to guarantee that we get the same answer every time. As far as I know, the proposed MERGE patch has that issue an existing DML commands don't; but someone else may have better information. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 15:09 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 16:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 16:52 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> @ 2018-03-26 19:17 ` Simon Riggs <[email protected]> 2018-03-26 22:10 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Simon Riggs @ 2018-03-26 19:17 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 26 March 2018 at 17:52, Robert Haas <[email protected]> wrote: > On Mon, Mar 26, 2018 at 12:16 PM, Simon Riggs <[email protected]> wrote: >> On 26 March 2018 at 16:09, Robert Haas <[email protected]> wrote: >>> On Mon, Mar 26, 2018 at 5:53 AM, Simon Riggs <[email protected]> wrote: >>>> Since we now have MVCC catalog scans, all the name lookups are >>>> performed using the same snapshot so in the above scenario the newly >>>> created object would be invisible to the second name lookup. >>> >>> That's not true, because each lookup would be performed using a new >>> snapshot -- not all under one snapshot. >> >> You're saying we take a separate snapshot for each table we lookup? >> Sounds weird to me. > > I'm saying we take a separate snapshot for each and every catalog > lookup, except when we know that no catalog changes can have occurred. > See the commit message for 568d4138c646cd7cd8a837ac244ef2caf27c6bb8. > If you do a lookup in pg_class and 3 lookups in pg_attribute each of > the 4 can be done under a different snapshot, in the worst case. > You're not the first person to believe that the MVCC catalog scan > patch fixes that problem, but as the guy who wrote it, it definitely > doesn't. What that patch fixed was, prior to that patch, a catalog > scan might find the WRONG NUMBER OF ROWS, like you might do a lookup > against a unique index for an object that existed and, if the row was > concurrently updated, you might find 0 rows or 2 rows instead of 1 > row. IOW, it guaranteed that we used a consistent snapshot for each > individual lookup, not a consistent snapshot for the whole course of a > command. That all makes sense, thanks for explaining. I spent a few more minutes, going "but", "but" though I can now see good reasons for everything to work this way. >> So this error could happen in SELECT, UPDATE, DELETE or INSERT as well. >> >> Or you see this as something related specifically to MERGE, if so how? >> Please explain what you see. > > As I said before, the problem occurs if the same command looks up the > same table name in more than one place. There is absolutely nothing > to guarantee that we get the same answer every time. > As far as I > know, the proposed MERGE patch has that issue an existing DML commands > don't; but someone else may have better information. I will look deeper and report back. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 15:09 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 16:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 16:52 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 19:17 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-26 22:10 ` Peter Geoghegan <[email protected]> 2018-03-27 08:15 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Peter Geoghegan @ 2018-03-26 22:10 UTC (permalink / raw) To: Simon Riggs <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Mon, Mar 26, 2018 at 12:17 PM, Simon Riggs <[email protected]> wrote: >> As far as I >> know, the proposed MERGE patch has that issue an existing DML commands >> don't; but someone else may have better information. > > I will look deeper and report back. It's quite clear that the problem exists with the MERGE patch; the simple fact that RangeVarGetRelidExtended() is called twice with the same RangeVar argument shows this. However, the Oid cross-check seems like a sufficient defense against an inconsistency that causes real trouble, since the cross-check will only error-out when a concurrent table creation (or maybe ALTER TABLE) makes a second table visible, in a schema that appears earlier in the user's search_path. It's hard to imagine any legitimate user truly preferring some alternative behavior in this particular scenario, which makes it okay. This cross-check workaround is ugly, but apparently there is a precedent in copy.c. I didn't know that detail until Robert pointed it out. That makes me feel a lot better about this general question of how the target relation is represented, having two RTEs, etc. -- Peter Geoghegan ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 15:09 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 16:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 16:52 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 19:17 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 22:10 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> @ 2018-03-27 08:15 ` Simon Riggs <[email protected]> 2018-03-27 18:14 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 0 siblings, 1 reply; 81+ messages in thread From: Simon Riggs @ 2018-03-27 08:15 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 26 March 2018 at 23:10, Peter Geoghegan <[email protected]> wrote: > On Mon, Mar 26, 2018 at 12:17 PM, Simon Riggs <[email protected]> wrote: >>> As far as I >>> know, the proposed MERGE patch has that issue an existing DML commands >>> don't; but someone else may have better information. >> >> I will look deeper and report back. > > It's quite clear that the problem exists with the MERGE patch Accepted, the only question is whether it affects UPDATE as well cos it looks like it should. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-24 12:27 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 09:53 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 15:09 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 16:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 16:52 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> 2018-03-26 19:17 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> 2018-03-26 22:10 ` Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-27 08:15 ` Re: [HACKERS] MERGE SQL Statement for PG11 Simon Riggs <[email protected]> @ 2018-03-27 18:14 ` Peter Geoghegan <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Peter Geoghegan @ 2018-03-27 18:14 UTC (permalink / raw) To: Simon Riggs <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On Tue, Mar 27, 2018 at 1:15 AM, Simon Riggs <[email protected]> wrote: > Accepted, the only question is whether it affects UPDATE as well cos > it looks like it should. If you mean an UPDATE FROM self-join, then I suppose that it does, in a very limited way. The difference is that there are no hard-coded assumptions about the relationship between those two RTEs. -- Peter Geoghegan ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: [HACKERS] MERGE SQL Statement for PG11 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Re: [HACKERS] MERGE SQL Statement for PG11 Robert Haas <[email protected]> @ 2018-03-24 12:28 ` Simon Riggs <[email protected]> 1 sibling, 0 replies; 81+ messages in thread From: Simon Riggs @ 2018-03-24 12:28 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andrew Dunstan <[email protected]>; Pavan Deolasee <[email protected]>; Stephen Frost <[email protected]>; pgsql-hackers On 24 March 2018 at 12:16, Robert Haas <[email protected]> wrote: > On Thu, Mar 22, 2018 at 7:13 PM, Peter Geoghegan <[email protected]> wrote: >> While I think this this particular HINT buglet is pretty harmless, I >> continue to be concerned about the unintended consequences of having >> multiple RTEs for MERGE's target table. Each RTE comes from a >> different lookup path -- the first one goes through setTargetTable()'s >> parserOpenTable() + addRangeTableEntryForRelation() calls. The second >> one goes through transformFromClauseItem(), for the join, which >> ultimately ends up calling transformTableEntry()/addRangeTableEntry(). >> Consider commit 5f173040, which fixed a privilege escalation security >> bug around multiple name lookup. Could the approach taken by MERGE >> here introduce a similar security issue? > > Yeah, that seems really bad. I don't think there is a huge problem > with having multiple RTEs; for example, we very commonly end up with > both rte->inh and !rte->inh RTEs for the same table, and that is OK. > However, generating those RTEs by doing multiple name lookups for the > same table is a big problem. Imagine, for example, that a user has a > search_path of a, b and that there is a table b.foo. The user does a > merge on foo. Between the first name lookup and the second, somebody > creates a.foo. Now, potentially, half of the MERGE statement is going > to be running against b.foo and the other half against a.foo. I don't > know whether that will crash or bomb out with a strange error or just > make some unexpected modification to one of those tables, but the > behavior, even if not insecure, will certainly be wrong. MERGE uses multiple RTEs in exactly the same way UPDATE does. I don't see a reason for specific concern wrt to MERGE. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. @ 2021-02-04 10:39 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-02-04 10:39 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index dd629668540..757b240a3c8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += nbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1087,7 +1047,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.0 --------------AD4B53627CEDABF9E8CC5B3C-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v3 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-04-01 20:22 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-04-01 20:22 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Reviewed-by: John Naylor Discussion: https://www.postgresql.org/message-id/89627a2a-c123-a8aa-267e-5d168feb73dd%40iki.fi --- src/backend/commands/copyfromparse.c | 111 +++++++++------------------ 1 file changed, 35 insertions(+), 76 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 0813424768..b52abc1520 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -90,21 +90,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (input_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - input_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -159,7 +144,7 @@ static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo, /* Low-level communications functions */ static int CopyGetData(CopyFromState cstate, void *databuf, - int minread, int maxread); + int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); static void CopyLoadInputBuf(CopyFromState cstate); @@ -230,9 +215,8 @@ ReceiveCopyBinaryHeader(CopyFromState cstate) /* * CopyGetData reads data from the source (file or frontend) * - * We attempt to read at least minread, and at most maxread, bytes from - * the source. The actual number of bytes read is returned; if this is - * less than minread, EOF was detected. + * We attempt to read at maxread bytes from the source. The actual + * number of bytes read is returned; if this is 0, EOF was detected. * * Note: when copying from the frontend, we expect a proper EOF mark per * protocol; if the frontend simply drops the connection, we raise error. @@ -241,7 +225,7 @@ ReceiveCopyBinaryHeader(CopyFromState cstate) * NB: no data conversion is applied here. */ static int -CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) +CopyGetData(CopyFromState cstate, void *databuf, int maxread) { int bytesread = 0; @@ -257,7 +241,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) cstate->raw_reached_eof = true; break; case COPY_FRONTEND: - while (maxread > 0 && bytesread < minread && !cstate->raw_reached_eof) + while (maxread > 0 && bytesread == 0 && !cstate->raw_reached_eof) { int avail; @@ -321,7 +305,9 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) } break; case COPY_CALLBACK: - bytesread = cstate->data_source_cb(databuf, minread, maxread); + bytesread = cstate->data_source_cb(databuf, 1, maxread); + if (bytesread == 0) + cstate->raw_reached_eof = true; break; } @@ -605,7 +591,7 @@ CopyLoadRawBuf(CopyFromState cstate) /* Load more data */ inbytes = CopyGetData(cstate, cstate->raw_buf + cstate->raw_buf_len, - 1, RAW_BUF_SIZE - cstate->raw_buf_len); + RAW_BUF_SIZE - cstate->raw_buf_len); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_len = nbytes; @@ -990,7 +976,7 @@ CopyReadLine(CopyFromState cstate) do { inbytes = CopyGetData(cstate, cstate->input_buf, - 1, INPUT_BUF_SIZE); + INPUT_BUF_SIZE); } while (inbytes > 0); cstate->input_buf_index = 0; cstate->input_buf_len = 0; @@ -1047,8 +1033,7 @@ CopyReadLineText(CopyFromState cstate) char *copy_input_buf; int input_buf_ptr; int copy_buf_len; - bool need_data = false; - bool hit_eof = false; + bool hit_eof; bool result = false; /* CSV variables */ @@ -1098,6 +1083,7 @@ CopyReadLineText(CopyFromState cstate) copy_input_buf = cstate->input_buf; input_buf_ptr = cstate->input_buf_index; copy_buf_len = cstate->input_buf_len; + hit_eof = cstate->input_reached_eof; for (;;) { @@ -1105,35 +1091,37 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (input_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (input_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - CopyLoadInputBuf(cstate); - /* update our local variables */ - hit_eof = cstate->input_reached_eof; - input_buf_ptr = cstate->input_buf_index; - copy_buf_len = cstate->input_buf_len; + CopyLoadInputBuf(cstate); + /* update our local variables */ + hit_eof = cstate->input_reached_eof; + input_buf_ptr = cstate->input_buf_index; + copy_buf_len = cstate->input_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (INPUT_BUF_BYTES(cstate) <= 0) + if (copy_buf_len - input_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -1142,20 +1130,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -1189,14 +1163,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of input_buf. + * Look at the next character. If we're at EOF, c will wind + * up as '\0' because of the guaranteed pad of input_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_input_buf[input_buf_ptr]; if (c == '\n') @@ -1262,7 +1231,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -1277,16 +1245,9 @@ CopyReadLineText(CopyFromState cstate) { input_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_input_buf[input_buf_ptr++]; if (c2 == '\n') @@ -1309,9 +1270,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_input_buf[input_buf_ptr++]; if (c2 != '\r' && c2 != '\n') -- 2.30.2 --------------11E1AF537C24312AF77A7698-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* [PATCH v4 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-04-06 17:48 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Heikki Linnakangas @ 2021-04-06 17:48 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). FIXME: This fails with sequence "\.<NL><some invalid bytes>". We should detect the end-of-copy marker \. and stop reading without complaining about the garbage after the end-of-copy marker. That doesn't work if we force 4 bytes of lookahead; the invalid byte sequence fits in the lookahead window, so we will try to convert it. Reviewed-by: John Naylor Discussion: https://www.postgresql.org/message-id/89627a2a-c123-a8aa-267e-5d168feb73dd%40iki.fi --- src/backend/commands/copyfromparse.c | 112 +++++++++------------------ 1 file changed, 36 insertions(+), 76 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 0813424768..6b140531ed 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -90,21 +90,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (input_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - input_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -159,7 +144,7 @@ static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo, /* Low-level communications functions */ static int CopyGetData(CopyFromState cstate, void *databuf, - int minread, int maxread); + int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); static void CopyLoadInputBuf(CopyFromState cstate); @@ -230,9 +215,8 @@ ReceiveCopyBinaryHeader(CopyFromState cstate) /* * CopyGetData reads data from the source (file or frontend) * - * We attempt to read at least minread, and at most maxread, bytes from - * the source. The actual number of bytes read is returned; if this is - * less than minread, EOF was detected. + * We attempt to read at most maxread bytes from the source. The actual + * number of bytes read is returned; if this is 0, EOF was detected. * * Note: when copying from the frontend, we expect a proper EOF mark per * protocol; if the frontend simply drops the connection, we raise error. @@ -241,7 +225,7 @@ ReceiveCopyBinaryHeader(CopyFromState cstate) * NB: no data conversion is applied here. */ static int -CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) +CopyGetData(CopyFromState cstate, void *databuf, int maxread) { int bytesread = 0; @@ -257,7 +241,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) cstate->raw_reached_eof = true; break; case COPY_FRONTEND: - while (maxread > 0 && bytesread < minread && !cstate->raw_reached_eof) + while (maxread > 0 && bytesread == 0 && !cstate->raw_reached_eof) { int avail; @@ -321,7 +305,9 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) } break; case COPY_CALLBACK: - bytesread = cstate->data_source_cb(databuf, minread, maxread); + bytesread = cstate->data_source_cb(databuf, 1, maxread); + if (bytesread == 0) + cstate->raw_reached_eof = true; break; } @@ -605,7 +591,7 @@ CopyLoadRawBuf(CopyFromState cstate) /* Load more data */ inbytes = CopyGetData(cstate, cstate->raw_buf + cstate->raw_buf_len, - 1, RAW_BUF_SIZE - cstate->raw_buf_len); + RAW_BUF_SIZE - cstate->raw_buf_len); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_len = nbytes; @@ -990,7 +976,7 @@ CopyReadLine(CopyFromState cstate) do { inbytes = CopyGetData(cstate, cstate->input_buf, - 1, INPUT_BUF_SIZE); + INPUT_BUF_SIZE); } while (inbytes > 0); cstate->input_buf_index = 0; cstate->input_buf_len = 0; @@ -1047,8 +1033,7 @@ CopyReadLineText(CopyFromState cstate) char *copy_input_buf; int input_buf_ptr; int copy_buf_len; - bool need_data = false; - bool hit_eof = false; + bool hit_eof; bool result = false; /* CSV variables */ @@ -1098,6 +1083,7 @@ CopyReadLineText(CopyFromState cstate) copy_input_buf = cstate->input_buf; input_buf_ptr = cstate->input_buf_index; copy_buf_len = cstate->input_buf_len; + hit_eof = cstate->input_reached_eof; for (;;) { @@ -1105,35 +1091,38 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead at most three bytes in one iteration of the + * loop (for the sequence \.<CR><NL>), so make sure we have at least + * four bytes in the buffer. Note that we always guarantee that there + * is one \0 in the buffer, after last valid byte; the lookaheads + * below rely on that. */ - if (input_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (input_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - CopyLoadInputBuf(cstate); - /* update our local variables */ - hit_eof = cstate->input_reached_eof; - input_buf_ptr = cstate->input_buf_index; - copy_buf_len = cstate->input_buf_len; + CopyLoadInputBuf(cstate); + /* update our local variables */ + input_buf_ptr = cstate->input_buf_index; + copy_buf_len = cstate->input_buf_len; + hit_eof = cstate->input_reached_eof; + continue; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (INPUT_BUF_BYTES(cstate) <= 0) + if (copy_buf_len - input_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -1142,20 +1131,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -1189,14 +1164,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of input_buf. + * Look at the next character. If we're at EOF, c will wind + * up as '\0' because of the guaranteed pad of input_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_input_buf[input_buf_ptr]; if (c == '\n') @@ -1262,7 +1232,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -1277,16 +1246,9 @@ CopyReadLineText(CopyFromState cstate) { input_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_input_buf[input_buf_ptr++]; if (c2 == '\n') @@ -1309,9 +1271,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_input_buf[input_buf_ptr++]; if (c2 != '\r' && c2 != '\n') -- 2.30.2 --------------1601CF8503AB8D78BE138DD6-- ^ permalink raw reply [nested|flat] 81+ messages in thread
* Re: Race condition in server-crash testing @ 2022-04-06 00:46 Tom Lane <[email protected]> 0 siblings, 0 replies; 81+ messages in thread From: Tom Lane @ 2022-04-06 00:46 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: [email protected] Andres Freund <[email protected]> writes: > On 2022-04-04 00:50:27 -0400, Tom Lane wrote: >> It's hard to be totally sure, but I think what happened is that >> gaur hit the in-hindsight-obvious race condition in this code: >> we managed to execute a successful iteration of poll_query_until >> before the postmaster had noticed its dead child and commenced >> the restart. The test lines after these are not prepared to see >> failure-to-connect. >> It's not obvious to me how to remove this race condition. >> Thoughts? > Maybe we can use pump_until() with the psql that's not getting killed? With a > non-matching regex? That'd only return once the backend was killed by > postmaster, afaics? Good idea. What I actually did was to borrow the recently-fixed code in 013_crash_restart.pl that checks for psql's "connection lost" report. regards, tom lane ^ permalink raw reply [nested|flat] 81+ messages in thread
end of thread, other threads:[~2022-04-06 00:46 UTC | newest] Thread overview: 81+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2018-03-22 23:13 Re: [HACKERS] MERGE SQL Statement for PG11 Peter Geoghegan <[email protected]> 2018-03-23 11:56 ` Pavan Deolasee <[email protected]> 2018-03-24 06:15 ` Peter Geoghegan <[email protected]> 2018-03-24 06:35 ` Peter Geoghegan <[email protected]> 2018-03-24 12:16 ` Robert Haas <[email protected]> 2018-03-24 12:27 ` Robert Haas <[email protected]> 2018-03-24 17:48 ` Peter Geoghegan <[email protected]> 2018-03-26 09:53 ` Simon Riggs <[email protected]> 2018-03-26 14:39 ` Pavan Deolasee <[email protected]> 2018-03-26 15:05 ` Simon Riggs <[email protected]> 2018-03-26 16:06 ` Simon Riggs <[email protected]> 2018-03-27 08:24 ` Simon Riggs <[email protected]> 2018-03-27 09:31 ` Pavan Deolasee <[email protected]> 2018-03-27 10:46 ` Simon Riggs <[email protected]> 2018-03-27 11:30 ` Simon Riggs <[email protected]> 2018-03-29 06:37 ` Pavan Deolasee <[email protected]> 2018-03-29 06:56 ` Simon Riggs <[email protected]> 2018-03-27 09:28 ` Pavan Deolasee <[email protected]> 2018-03-27 09:40 ` Simon Riggs <[email protected]> 2018-03-28 02:58 ` Peter Geoghegan <[email protected]> 2018-03-28 04:30 ` Pavan Deolasee <[email protected]> 2018-03-28 11:00 ` Pavan Deolasee <[email protected]> 2018-03-29 09:50 ` Simon Riggs <[email protected]> 2018-03-30 11:10 ` Simon Riggs <[email protected]> 2018-03-26 15:09 ` Robert Haas <[email protected]> 2018-03-26 16:16 ` Simon Riggs <[email protected]> 2018-03-26 16:52 ` Robert Haas <[email protected]> 2018-03-26 19:17 ` Simon Riggs <[email protected]> 2018-03-26 22:10 ` Peter Geoghegan <[email protected]> 2018-03-27 08:15 ` Simon Riggs <[email protected]> 2018-03-27 18:14 ` Peter Geoghegan <[email protected]> 2018-03-24 12:28 ` Simon Riggs <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-02-04 10:39 [PATCH 2/2] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-04-01 20:22 [PATCH v3 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2021-04-06 17:48 [PATCH v4 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2022-04-06 00:46 Re: Race condition in server-crash testing Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox