Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kHmpB-0002CH-T9 for pgsql-hackers@arkaria.postgresql.org; Mon, 14 Sep 2020 11:39:11 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kHmpA-0002o1-QS for pgsql-hackers@arkaria.postgresql.org; Mon, 14 Sep 2020 11:39:08 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kHmpA-0002nu-1R for pgsql-hackers@lists.postgresql.org; Mon, 14 Sep 2020 11:39:08 +0000 Received: from mail.postgrespro.ru ([93.174.131.139]) by makus.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kHmp1-0004xs-JU for pgsql-hackers@lists.postgresql.org; Mon, 14 Sep 2020 11:39:07 +0000 Received: from [192.168.1.64] (unknown [109.252.84.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mail.postgrespro.ru (Postfix) with ESMTPSA id 4E01C21C2BDE; Mon, 14 Sep 2020 14:38:56 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=postgrespro.ru; s=mail; t=1600083536; bh=zBUa6mV/4JK8UkPaRh5I+daggTEElcww+eaj1hIZGYI=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=eQ2MPljqiZ5Wl+zOQHsx5+yKnXYECkqfWoWIZgu3EnFsl7xy/ojEVBcCFgoymw2u2 vz3anehhD0wfvOGNTQ38ytKTk7q/McvX4KbPq/nHxvyNX3BX6x7xMzfESEcH1eQmYd U1rEamTCRZM+W/Bj5ZYonGFrgdMAVnBmYpkciFjM= Subject: Re: [PATCH] Automatic HASH and LIST partition creation To: Pavel Borisov Cc: PostgreSQL Hackers , Fabien COELHO , Justin Pryzby , Robert Haas , Amul Sul References: <7fec3abb-c663-c0d2-8452-a46141be6d4a@postgrespro.ru> <7c4013db-a628-c8bd-ce54-064389109e0f@postgrespro.ru> <726c63e2-ef34-8113-f3cf-90e1646e4c2b@postgrespro.ru> From: Anastasia Lubennikova Message-ID: <74a716a3-986d-c737-6217-6a3ffd488da3@postgrespro.ru> Date: Mon, 14 Sep 2020 14:38:56 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/mixed; boundary="------------FC4E4E20AFC6F93BE7445997" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------FC4E4E20AFC6F93BE7445997 Content-Type: multipart/alternative; boundary="------------E25B5B64E5C18199003833C4" --------------E25B5B64E5C18199003833C4 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 08.09.2020 17:03, Pavel Borisov wrote: > > The patch lacks documentation, because I expect some details may > change during discussion. Other than that, the feature is ready > for review. > > Hi, hackers! > > From what I've read I see there is much interest in automatic > partitions creation. (Overall discussion on the topic is partitioned > into two threads: (1) > https://www.postgresql.org/message-id/alpine.DEB.2.21.1907150711080.22273%40lancre and > (2) > https://www.postgresql.org/message-id/flat/7fec3abb-c663-c0d2-8452-a46141be6d4a@postgrespro.ru > (current thread) ) > > There were many syntax proposals and finally, there is a patch > realizing one of them. So I'd like to review it. > > The syntax proposed in the patch seems good enough for me and is in > accordance with one of the proposals in the discussion. Maybe I'd > prefer using the word AUTOMATICALLY/AUTO instead of CONFIGURATION with > explicit meaning that using this syntax we'd get already > (automatically) created partitions and don't need to create them > manually, as in the existing state of postgresql declarative > partitioning. > > CREATE TABLE tbl (iint) PARTITION BY HASH (i) AUTOMATICALLY (MODULUS 3); (partitions are created automatically) > vs > CREATE TABLE tbl (iint) PARTITION BY HASH (i); (partitions should be created manually by use of PARTITION OF) > CREATE TABLE tbl (i char) PARTITION BY LIST (i) AUTOMATICALLY (VALUES > IN ('a', 'b'), ('c', 'd'), ('e','f') DEFAULTPARTITION tbl_default); > vs > CREATE TABLE tbl (ichar) PARTITION BY LIST (i); (partitions should be created manually by use of PARTITION OF) > > I think this syntax can also be extended later with adding automatic > creation of RANGE partitions, with IMMEDIATE/DEFERRED for > dynamic/on-demand automatic partition creation, and with SUBPARTITION > possibility. > > But I don't have a strong preference for the word AUTOMATICALLY, > moreover I saw opposition to using AUTO at the top of the discussion. > I suppose we can go with the existing CONFIGURATION word. I agree that 'AUTOMATICALLY' keyword is more specific and probably less confusing for users. I've picked 'CONFIGURATION' simply because it is an already existing keyword. It would like to hear other opinions on that. > If compare with existing declarative partitions, I think automatic > creation simplifies the process for the end-user and  I'd vote for its > committing into Postgres. The patch is short and clean in code style. > It has enough comments Tests covering the new functionality are > included. Yet it doesn't have documentation and I'd suppose it's worth > adding it. Even if there will be syntax changes, I hope they will not > be more than the replacement of several words. Current syntax is > described in the text of a patch. > Fair enough. New patch contains a documentation draft. While writing it, I also noticed, that the syntax, introduced in the patch differs from greenpulm one. For now, list partitioning clause doesn't support 'PARTITION name' part, that is supported in greenplum. I don't think that we aim for 100% compatibility here. Still, the ability to provide table names is probably a good optional feature, especially for list partitions. What do you think? > The patch applies cleanly and installcheck-world is passed. > > Some minor things: > > I've got a compiler warning: > parse_utilcmd.c:4280:15: warning: unused variable 'lc' [-Wunused-variable] Fixed. This was also caught by cfbot. This version should pass it clean. > > When the number of partitions is over the maximum value of int32 the > output shows a generic syntax error. I don't think it is very > important as it is not the case someone will make deliberately, but > maybe it's better to output something like "Partitions number is more > than the maximum supported value" > create table test (i int, t text) partition by hash (i) configuration > (modulus 888888888888); > ERROR:  syntax error at or near "888888888888" This value is not a valid int32 number, thus parser throws the error before we have a chance to handle it more gracefully. > > One more piece of nitpicking. Probably we can go just with a mention > in documentation. > create table test (i int, t text) partition by hash (i) configuration > (modulus 8888); > ERROR:  out of shared memory > HINT:  You might need to increase max_locks_per_transaction. > Well, it looks like a legit error, when we try to lock a lot of objects in one transaction. I will double check if we don't release a lock somewhere. Do we need to restrict the number of partitions, that can be created by this statement? With what number?  As far as I see, there is no such restriction for now, just a recommendation about performance issues. With automatic creation it becomes easier to mess with it. Probably, it's enough to mention it in documentation and rely on users common sense. > Typo: > + /* Add statemets to create each partition after we create parent > table */ > Fixed. > Overall I see the patch almost ready for commit and I'd like to meet > this functionality in v14. I also hope that this patch will make it to v14, but for now, I don't see a consensus on the syntax and some details, so I wouldn't rush. Besides, it definitely needs more testing. I haven't thoroughly tested following cases yet: - how triggers and constraints are propagated to partitions; - how does it handle some tricky clauses in list partitioning expr_list; and so on. Also, there is an open question about partition naming. Currently, the patch implements dummy %tbl_%partnum name generation, which is far from user-friendly. I think we must provide some hook or trigger function to rename partitions after they were created. -- Anastasia Lubennikova Postgres Professional: http://www.postgrespro.com The Russian Postgres Company --------------E25B5B64E5C18199003833C4 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit
On 08.09.2020 17:03, Pavel Borisov wrote:

The patch lacks documentation, because I expect some details may change during discussion. Other than that, the feature is ready for review.

Hi, hackers!

From what I've read I see there is much interest in automatic partitions creation. (Overall discussion on the topic is partitioned into two threads: (1) https://www.postgresql.org/message-id/alpine.DEB.2.21.1907150711080.22273%40lancre and (2) https://www.postgresql.org/message-id/flat/7fec3abb-c663-c0d2-8452-a46141be6d4a@postgrespro.ru (current thread) ) 

There were many syntax proposals and finally, there is a patch realizing one of them. So I'd like to review it.

The syntax proposed in the patch seems good enough for me and is in accordance with one of the proposals in the discussion. Maybe I'd prefer using the word AUTOMATICALLY/AUTO instead of CONFIGURATION with explicit meaning that using this syntax we'd get already (automatically) created partitions and don't need to create them manually, as in the existing state of postgresql declarative partitioning. 

CREATE TABLE tbl (i int) PARTITION BY HASH (i) AUTOMATICALLY (MODULUS 3); (partitions are created automatically)
vs
CREATE TABLE tbl (i int) PARTITION BY HASH (i); (partitions should be created manually by use of PARTITION OF)

              
CREATE TABLE tbl (i char) PARTITION BY LIST (i) AUTOMATICALLY (VALUES IN ('a', 'b'), ('c', 'd'), ('e','f') DEFAULT PARTITION tbl_default);
vs
CREATE TABLE tbl (i char) PARTITION BY LIST (i); (partitions should be created manually by use of PARTITION OF)

I think this syntax can also be extended later with adding automatic creation of RANGE partitions, with IMMEDIATE/DEFERRED for dynamic/on-demand automatic partition creation, and with SUBPARTITION possibility.

But I don't have a strong preference for the word AUTOMATICALLY, moreover I saw opposition to using AUTO at the top of the discussion. I suppose we can go with the existing CONFIGURATION word.

I agree that 'AUTOMATICALLY' keyword is more specific and probably less confusing for users. I've picked 'CONFIGURATION' simply because it is an already existing keyword. It would like to hear other opinions on that.


If compare with existing declarative partitions, I think automatic creation simplifies the process for the end-user and  I'd vote for its committing into Postgres. The patch is short and clean in code style. It has enough comments Tests covering the new functionality are included. Yet it doesn't have documentation and I'd suppose it's worth adding it. Even if there will be syntax changes, I hope they will not be more than the replacement of several words. Current syntax is described in the text of a patch.


Fair enough. New patch contains a documentation draft. While writing it, I also noticed, that the syntax, introduced in the patch differs from greenpulm one. For now, list partitioning clause doesn't support 'PARTITION name' part, that is supported in greenplum. I don't think that we aim for 100% compatibility here. Still, the ability to provide table names is probably a good optional feature, especially for list partitions.

What do you think?

The patch applies cleanly and installcheck-world is passed. 

Some minor things:

I've got a compiler warning:
parse_utilcmd.c:4280:15: warning: unused variable 'lc' [-Wunused-variable]

Fixed. This was also caught by cfbot. This version should pass it clean.


When the number of partitions is over the maximum value of int32 the output shows a generic syntax error. I don't think it is very important as it is not the case someone will make deliberately, but maybe it's better to output something like "Partitions number is more than the maximum supported value"
create table test (i int, t text) partition by hash (i) configuration (modulus 888888888888);
ERROR:  syntax error at or near "888888888888"

This value is not a valid int32 number, thus parser throws the error before we have a chance to handle it more gracefully.


One more piece of nitpicking. Probably we can go just with a mention in documentation. 
create table test (i int, t text) partition by hash (i) configuration (modulus 8888);
ERROR:  out of shared memory
HINT:  You might need to increase max_locks_per_transaction.

Well, it looks like a legit error, when we try to lock a lot of objects in one transaction. I will double check if we don't release a lock somewhere.

Do we need to restrict the number of partitions, that can be created by this statement? With what number?  As far as I see, there is no such restriction for now, just a recommendation about performance issues. With automatic creation it becomes easier to mess with it.

Probably, it's enough to mention it in documentation and rely on users common sense.

Typo:
+ /* Add statemets to create each partition after we create parent table */

Fixed.

Overall I see the patch almost ready for commit and I'd like to meet this functionality in v14.
I also hope that this patch will make it to v14, but for now, I don't see a consensus on the syntax and some details, so I wouldn't rush.

Besides, it definitely needs more testing. I haven't thoroughly tested following cases yet:
- how triggers and constraints are propagated to partitions;
- how does it handle some tricky clauses in list partitioning expr_list;
and so on.

Also, there is an open question about partition naming. Currently, the patch implements dummy %tbl_%partnum name generation, which is far from user-friendly. I think we must provide some hook or trigger function to rename partitions after they were created.

-- 
Anastasia Lubennikova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--------------E25B5B64E5C18199003833C4-- --------------FC4E4E20AFC6F93BE7445997 Content-Type: text/x-patch; charset=UTF-8; name="auto_part_hash_list_v1.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="auto_part_hash_list_v1.patch" commit 08c5f8b35c4ffcaf09b8189cd8d0dc27ce76d715 Author: anastasia Date: Mon Sep 14 11:34:42 2020 +0300 Auto generated HASH and LIST partitions. New syntax: CREATE TABLE tbl_hash (i int) PARTITION BY HASH (i) CONFIGURATION (modulus 3); CREATE TABLE tbl_list (i int) PARTITION BY LIST (i) CONFIGURATION (values in (1, 2), (3, 4) DEFAULT PARTITION tbl_default); With documentation draft. diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 087cad184c..ff9a7eda09 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -29,6 +29,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI ] ) [ INHERITS ( parent_table [, ... ] ) ] [ PARTITION BY { RANGE | LIST | HASH } ( { column_name | ( expression ) } [ COLLATE collation ] [ opclass ] [, ... ] ) ] +[ CONFIGURATION ( partition_bound_auto_spec ) ] [ USING method ] [ WITH ( storage_parameter [= value] [, ... ] ) | WITHOUT OIDS ] [ ON COMMIT { PRESERVE ROWS | DELETE ROWS | DROP } ] @@ -41,6 +42,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI [, ... ] ) ] [ PARTITION BY { RANGE | LIST | HASH } ( { column_name | ( expression ) } [ COLLATE collation ] [ opclass ] [, ... ] ) ] +[ CONFIGURATION ( partition_bound_auto_spec ) ] [ USING method ] [ WITH ( storage_parameter [= value] [, ... ] ) | WITHOUT OIDS ] [ ON COMMIT { PRESERVE ROWS | DELETE ROWS | DROP } ] @@ -53,6 +55,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI [, ... ] ) ] { FOR VALUES partition_bound_spec | DEFAULT } [ PARTITION BY { RANGE | LIST | HASH } ( { column_name | ( expression ) } [ COLLATE collation ] [ opclass ] [, ... ] ) ] +[ CONFIGURATION ( partition_bound_auto_spec ) ] [ USING method ] [ WITH ( storage_parameter [= value] [, ... ] ) | WITHOUT OIDS ] [ ON COMMIT { PRESERVE ROWS | DELETE ROWS | DROP } ] @@ -96,6 +99,11 @@ FROM ( { partition_bound_expr | MIN TO ( { partition_bound_expr | MINVALUE | MAXVALUE } [, ...] ) | WITH ( MODULUS numeric_literal, REMAINDER numeric_literal ) +and partition_bound_auto_spec is: + +VALUES IN ( partition_bound_expr [, ...] ), [( partition_bound_expr [, ...] )] [, ...] [DEFAULT PARTITION defailt_part_name] +MODULUS numeric_literal + index_parameters in UNIQUE, PRIMARY KEY, and EXCLUDE constraints are: [ INCLUDE ( column_name [, ... ] ) ] @@ -383,6 +391,11 @@ WITH ( MODULUS numeric_literal, REM however, you can define these constraints on individual partitions. + + Range and list partitioning also support automatic creation of partitions + with an optional CONFIGURATION clause. + + See for more discussion on table partitioning. @@ -391,6 +404,38 @@ WITH ( MODULUS numeric_literal, REM + + CONFIGURATION ( partition_bound_auto_spec ) ] + + + The optional CONFIGURATION clause used together + with PARTITION BY specifies a rule of generating bounds + for partitions of the partitioned table. All partitions are created automatically + along with the parent table. + + Any indexes, constraints and user-defined row-level triggers that exist + in the parent table are cloned on the new partitions. + + + + The partition_bound_auto_spec + must correspond to the partitioning method and partition key of the + parent table, and must not overlap with any existing partition of that + parent. The form with VALUES IN is used for list partitioning + and the form with MODULUS is used for hash partitioning. + List partitioning can also provide a default partition using + DEFAULT PARTITION. + + + + Automatic range partitioning is not supported yet. + + + + + + + PARTITION OF parent_table { FOR VALUES partition_bound_spec | DEFAULT } diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 0409a40b82..6893fa5495 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -4628,6 +4628,7 @@ _copyPartitionSpec(const PartitionSpec *from) COPY_STRING_FIELD(strategy); COPY_NODE_FIELD(partParams); + COPY_NODE_FIELD(autopart); COPY_LOCATION_FIELD(location); return newnode; @@ -4650,6 +4651,19 @@ _copyPartitionBoundSpec(const PartitionBoundSpec *from) return newnode; } +static PartitionBoundAutoSpec * +_copyPartitionBoundAutoSpec(const PartitionBoundAutoSpec *from) +{ + PartitionBoundAutoSpec *newnode = makeNode(PartitionBoundAutoSpec); + + COPY_SCALAR_FIELD(strategy); + COPY_SCALAR_FIELD(modulus); + COPY_NODE_FIELD(listdatumsList); + COPY_NODE_FIELD(default_partition_rv); + + return newnode; +} + static PartitionRangeDatum * _copyPartitionRangeDatum(const PartitionRangeDatum *from) { @@ -5699,6 +5713,9 @@ copyObjectImpl(const void *from) case T_PartitionBoundSpec: retval = _copyPartitionBoundSpec(from); break; + case T_PartitionBoundAutoSpec: + retval = _copyPartitionBoundAutoSpec(from); + break; case T_PartitionRangeDatum: retval = _copyPartitionRangeDatum(from); break; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index e2d1b987bf..ccba2471de 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2897,6 +2897,7 @@ _equalPartitionSpec(const PartitionSpec *a, const PartitionSpec *b) COMPARE_STRING_FIELD(strategy); COMPARE_NODE_FIELD(partParams); COMPARE_LOCATION_FIELD(location); + COMPARE_NODE_FIELD(autopart); return true; } @@ -2916,6 +2917,19 @@ _equalPartitionBoundSpec(const PartitionBoundSpec *a, const PartitionBoundSpec * return true; } +static bool +_equalPartitionBoundAutoSpec(const PartitionBoundAutoSpec *a, + const PartitionBoundAutoSpec *b) +{ + COMPARE_SCALAR_FIELD(strategy); + COMPARE_SCALAR_FIELD(modulus); + COMPARE_NODE_FIELD(listdatumsList); + COMPARE_NODE_FIELD(default_partition_rv); + + return true; +} + + static bool _equalPartitionRangeDatum(const PartitionRangeDatum *a, const PartitionRangeDatum *b) { @@ -3751,6 +3765,9 @@ equal(const void *a, const void *b) case T_PartitionBoundSpec: retval = _equalPartitionBoundSpec(a, b); break; + case T_PartitionBoundAutoSpec: + retval = _equalPartitionBoundAutoSpec(a, b); + break; case T_PartitionRangeDatum: retval = _equalPartitionRangeDatum(a, b); break; diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..4fd12523d8 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -3643,6 +3643,7 @@ _outPartitionSpec(StringInfo str, const PartitionSpec *node) WRITE_STRING_FIELD(strategy); WRITE_NODE_FIELD(partParams); + WRITE_NODE_FIELD(autopart); WRITE_LOCATION_FIELD(location); } @@ -3661,6 +3662,18 @@ _outPartitionBoundSpec(StringInfo str, const PartitionBoundSpec *node) WRITE_LOCATION_FIELD(location); } +static void +_outPartitionBoundAutoSpec(StringInfo str, const PartitionBoundAutoSpec *node) +{ + WRITE_NODE_TYPE("PARTITIONBOUNDAUTOSPEC"); + + WRITE_CHAR_FIELD(strategy); + WRITE_INT_FIELD(modulus); + WRITE_NODE_FIELD(listdatumsList); + WRITE_NODE_FIELD(default_partition_rv); + +} + static void _outPartitionRangeDatum(StringInfo str, const PartitionRangeDatum *node) { @@ -4334,6 +4347,9 @@ outNode(StringInfo str, const void *obj) case T_PartitionBoundSpec: _outPartitionBoundSpec(str, obj); break; + case T_PartitionBoundAutoSpec: + _outPartitionBoundAutoSpec(str, obj); + break; case T_PartitionRangeDatum: _outPartitionRangeDatum(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..703b413f93 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -2602,6 +2602,19 @@ _readPartitionBoundSpec(void) READ_DONE(); } +static PartitionBoundAutoSpec * +_readPartitionBoundAutoSpec(void) +{ + READ_LOCALS(PartitionBoundAutoSpec); + + READ_CHAR_FIELD(strategy); + READ_INT_FIELD(modulus); + READ_NODE_FIELD(listdatumsList); + READ_NODE_FIELD(default_partition_rv); + + READ_DONE(); +} + /* * _readPartitionRangeDatum */ @@ -2880,6 +2893,8 @@ parseNodeString(void) return_value = _readPartitionBoundSpec(); else if (MATCH("PARTITIONRANGEDATUM", 19)) return_value = _readPartitionRangeDatum(); + else if (MATCH("PARTITIONBOUNDAUTOSPEC", 22)) + return_value = _readPartitionBoundAutoSpec(); else { elog(ERROR, "badly formatted node string \"%.32s\"...", token); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 9f47745ee2..a1f9db30d2 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -249,6 +249,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); PartitionElem *partelem; PartitionSpec *partspec; PartitionBoundSpec *partboundspec; + PartitionBoundAutoSpec *partboundautospec; RoleSpec *rolespec; struct SelectLimit *selectlimit; } @@ -599,6 +600,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type hash_partbound %type hash_partbound_elem +%type OptPartitionBoundAutoSpec values_in_clause p_desc +%type opt_default_partition_clause + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -3907,14 +3911,14 @@ OptPartitionSpec: PartitionSpec { $$ = $1; } | /*EMPTY*/ { $$ = NULL; } ; -PartitionSpec: PARTITION BY ColId '(' part_params ')' +PartitionSpec: PARTITION BY ColId '(' part_params ')' OptPartitionBoundAutoSpec { PartitionSpec *n = makeNode(PartitionSpec); n->strategy = $3; n->partParams = $5; n->location = @1; - + n->autopart = (Node *) $7; $$ = n; } ; @@ -3958,6 +3962,80 @@ part_elem: ColId opt_collate opt_class } ; +OptPartitionBoundAutoSpec: + CONFIGURATION '(' p_desc ')' + { + $$ = $3; + } + | /*EMPTY*/ { $$ = NULL; } + ; + +p_desc: + hash_partbound + { + ListCell *lc; + PartitionBoundAutoSpec *n = makeNode(PartitionBoundAutoSpec); + + n->modulus = -1; + + foreach (lc, $1) + { + DefElem *opt = lfirst_node(DefElem, lc); + + if (strcmp(opt->defname, "modulus") == 0) + { + n->strategy = PARTITION_STRATEGY_HASH; + if (n->modulus != -1) + ereport(ERROR, + (errcode(ERRCODE_DUPLICATE_OBJECT), + errmsg("modulus for hash partition provided more than once"), + parser_errposition(opt->location))); + n->modulus = defGetInt32(opt); + } + else + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unrecognized auto partition bound specification \"%s\"", + opt->defname), + parser_errposition(opt->location))); + } + + $$ = (PartitionBoundAutoSpec *) n; + } + | values_in_clause opt_default_partition_clause + { + PartitionBoundAutoSpec *n = $1; + n->default_partition_rv = $2; + $$ = (PartitionBoundAutoSpec *) n; + } + ; + +values_in_clause: + VALUES IN_P '(' expr_list ')' + { + PartitionBoundAutoSpec *n = makeNode(PartitionBoundAutoSpec); + n->strategy = PARTITION_STRATEGY_LIST; + n->listdatumsList = list_make1($4); + $$ = (PartitionBoundAutoSpec *) n; + } + | values_in_clause ',' '(' expr_list ')' + { + PartitionBoundAutoSpec *n = (PartitionBoundAutoSpec *) $1; + n->strategy = PARTITION_STRATEGY_LIST; + n->listdatumsList = lappend(n->listdatumsList, $4); + $$ = (PartitionBoundAutoSpec *) n; + } + ; + +opt_default_partition_clause: + DEFAULT PARTITION qualified_name + { + $$ = $3; + } + | /* EMPTY */ + { $$ = NULL; } + ; + table_access_method_clause: USING name { $$ = $2; } | /*EMPTY*/ { $$ = NULL; } diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index ec944371dd..a22bbb8ebc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -75,6 +75,7 @@ /* State shared by transformCreateStmt and its subroutines */ typedef struct { + CreateStmt *stmt; /* initial statement */ ParseState *pstate; /* overall parser state */ const char *stmtType; /* "CREATE [FOREIGN] TABLE" or "ALTER TABLE" */ RangeVar *relation; /* relation to create */ @@ -145,6 +146,7 @@ static Const *transformPartitionBoundValue(ParseState *pstate, Node *con, const char *colName, Oid colType, int32 colTypmod, Oid partCollation); +static void transformPartitionAutoCreate(CreateStmtContext *cxt, PartitionSpec* partspec); /* * transformCreateStmt - @@ -235,6 +237,7 @@ transformCreateStmt(CreateStmt *stmt, const char *queryString) cxt.stmtType = "CREATE TABLE"; cxt.isforeign = false; } + cxt.stmt = stmt; cxt.relation = stmt->relation; cxt.rel = NULL; cxt.inhRelations = stmt->inhRelations; @@ -324,6 +327,10 @@ transformCreateStmt(CreateStmt *stmt, const char *queryString) */ transformExtendedStatistics(&cxt); + /* Process partition definitions */ + if (stmt->partspec && stmt->partspec->autopart) + transformPartitionAutoCreate(&cxt, stmt->partspec); + /* * Output results. */ @@ -4258,3 +4265,133 @@ transformPartitionBoundValue(ParseState *pstate, Node *val, return (Const *) value; } + + +/* + * Transform configuration into a set of partition bounds. + * Generate extra statements to create partition tables. + */ +static void +transformPartitionAutoCreate(CreateStmtContext *cxt, PartitionSpec* partspec) +{ + CreateStmt *part; + List *partlist = NIL; + int i = 0; + PartitionBoundAutoSpec *bound = (PartitionBoundAutoSpec *) partspec->autopart; + + elog(DEBUG1, "transformPartitionAutoCreate \n %s \n ", nodeToString(bound)); + + /* + * Generate regular partbounds based on autopart rule. + * and form create table statements from these partbounds + */ + if (pg_strcasecmp(partspec->strategy, "hash") == 0) + { + if (bound->strategy != PARTITION_STRATEGY_HASH) + ereport(ERROR, + (errcode(ERRCODE_INVALID_TABLE_DEFINITION), + errmsg("invalid bound specification for a hash partition"), + parser_errposition(cxt->pstate, exprLocation((Node *) partspec)))); + + for (i = 0; i < bound->modulus; i++) + { + char *part_relname; + + /* + * Generate partition name in the format: + * $relname_$partnum + * + * TODO: Add checks on relname length. + */ + part_relname = psprintf("%s_%d", cxt->relation->relname, i); + + part = copyObject(cxt->stmt); + + part->relation = makeRangeVar(cxt->relation->schemaname, + part_relname, cxt->relation->location); + /* set table as a parent */ + part->inhRelations = lappend(part->inhRelations, cxt->relation); + + /* child table is not partitioned */ + part->partspec = NULL; + + /* Actual partbound generation is here */ + part->partbound = makeNode(PartitionBoundSpec); + part->partbound->strategy = PARTITION_STRATEGY_HASH; + part->partbound->modulus = bound->modulus; + part->partbound->remainder = i; + part->partbound->is_default = false; + + elog(DEBUG1,"stransformPartitionAutoCreate HASH i %d MODULUS %d \n %s\n", + i, bound->modulus, nodeToString(part)); + + partlist = lappend(partlist, part); + } + } + else if (pg_strcasecmp(partspec->strategy, "list") == 0) + { + + int n_list_parts = list_length(bound->listdatumsList); + + if (bound->strategy != PARTITION_STRATEGY_LIST) + ereport(ERROR, + (errcode(ERRCODE_INVALID_TABLE_DEFINITION), + errmsg("invalid bound specification for a list partition"), + parser_errposition(cxt->pstate, exprLocation((Node *) partspec)))); + + for (i = 0; i < n_list_parts; i++) + { + char *part_relname; + List *listdatums = (List *) + list_nth(bound->listdatumsList, i); + + part_relname = psprintf("%s_%d", cxt->relation->relname, i); + + part = copyObject(cxt->stmt); + + part->relation = makeRangeVar(cxt->relation->schemaname, + part_relname, cxt->relation->location); + /* set table as a parent */ + part->inhRelations = lappend(part->inhRelations, cxt->relation); + + /* child table is not partitioned */ + part->partspec = NULL; + + /* Actual partbound generation is here */ + part->partbound = makeNode(PartitionBoundSpec); + part->partbound->strategy = PARTITION_STRATEGY_LIST; + part->partbound->listdatums = list_copy(listdatums); + part->partbound->is_default = false; + + elog(DEBUG1,"Debug transformPartitionAutoCreate LIST i %d \n %s\n", + i, nodeToString(part)); + + partlist = lappend(partlist, part); + } + + if (bound->default_partition_rv) + { + part = copyObject(cxt->stmt); + + part->relation = bound->default_partition_rv; + /* set table as a parent */ + part->inhRelations = lappend(part->inhRelations, cxt->relation); + + /* child table is not partitioned */ + part->partspec = NULL; + + part->partbound = makeNode(PartitionBoundSpec); + part->partbound->strategy = PARTITION_STRATEGY_LIST; + part->partbound->listdatums = NULL; + part->partbound->is_default = true; + + elog(DEBUG1,"Debug transformPartitionAutoCreate LIST default partition \n %s\n", + nodeToString(part)); + + partlist = lappend(partlist, part); + } + } + + /* Add statements to create each partition after we create parent table */ + cxt->alist = list_concat(cxt->alist, partlist); +} diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..ac6fcc029e 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -478,6 +478,7 @@ typedef enum NodeTag T_PartitionElem, T_PartitionSpec, T_PartitionBoundSpec, + T_PartitionBoundAutoSpec, T_PartitionRangeDatum, T_PartitionCmd, T_VacuumRelation, diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index e83329fd6d..271632f0fe 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -794,6 +794,9 @@ typedef struct PartitionSpec * 'range') */ List *partParams; /* List of PartitionElems */ int location; /* token location, or -1 if unknown */ + + Node *autopart; /* PartitionBoundAutoSpec - + * spec to generate bounds automatically */ } PartitionSpec; /* Internal codes for partitioning strategies */ @@ -828,6 +831,26 @@ struct PartitionBoundSpec int location; /* token location, or -1 if unknown */ }; +/* + * PartitionBoundAutoSpec - a partition bound specification + * for auto generated partitions. + * + * This represents the rule of generating partition bounds + */ +struct PartitionBoundAutoSpec +{ + NodeTag type; + + char strategy; /* see PARTITION_STRATEGY codes above */ + + /* Partitioning info for HASH strategy: */ + int modulus; + + /* Partitioning info for LIST strategy: */ + List *listdatumsList; /* List of lists of Consts (or A_Consts in raw tree) */ + RangeVar *default_partition_rv; /* Name of default list partition */ +}; + /* * PartitionRangeDatum - one of the values in a range partition bound * diff --git a/src/include/partitioning/partdefs.h b/src/include/partitioning/partdefs.h index 6414e2c116..25ecfbd1de 100644 --- a/src/include/partitioning/partdefs.h +++ b/src/include/partitioning/partdefs.h @@ -19,6 +19,8 @@ typedef struct PartitionKeyData *PartitionKey; typedef struct PartitionBoundSpec PartitionBoundSpec; +typedef struct PartitionBoundAutoSpec PartitionBoundAutoSpec; + typedef struct PartitionDescData *PartitionDesc; typedef struct PartitionDirectoryData *PartitionDirectory; diff --git a/src/test/regress/expected/create_table.out b/src/test/regress/expected/create_table.out index 1c72f23bc9..3047ed9007 100644 --- a/src/test/regress/expected/create_table.out +++ b/src/test/regress/expected/create_table.out @@ -1283,3 +1283,40 @@ Indexes: "part_column_drop_1_10_expr_idx1" btree ((d = 2)) drop table part_column_drop; +-- Auto generated partitions +-- must fail because of wrong configuration +CREATE TABLE tbl_hash_fail (i int) PARTITION BY HASH (i) +CONFIGURATION (values in (1, 2), (3, 4) default partition tbl_default); +ERROR: invalid bound specification for a hash partition +LINE 1: CREATE TABLE tbl_hash_fail (i int) PARTITION BY HASH (i) + ^ +-- must fail because of wrong configuration +CREATE TABLE tbl_list_fail (i int) PARTITION BY LIST (i) +CONFIGURATION (values in (1, 2), (1, 3)); +ERROR: partition "tbl_list_fail_1" would overlap partition "tbl_list_fail_0" +CREATE TABLE tbl_list (i int) PARTITION BY LIST (i) +CONFIGURATION (values in (1, 2), (3, 4) default partition tbl_default); +\d+ tbl_list + Partitioned table "public.tbl_list" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + i | integer | | | | plain | | +Partition key: LIST (i) +Partitions: tbl_list_0 FOR VALUES IN (1, 2), + tbl_list_1 FOR VALUES IN (3, 4), + tbl_default DEFAULT + +CREATE TABLE tbl_hash (i int) PARTITION BY HASH (i) +CONFIGURATION (modulus 3); +\d+ tbl_hash + Partitioned table "public.tbl_hash" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + i | integer | | | | plain | | +Partition key: HASH (i) +Partitions: tbl_hash_0 FOR VALUES WITH (modulus 3, remainder 0), + tbl_hash_1 FOR VALUES WITH (modulus 3, remainder 1), + tbl_hash_2 FOR VALUES WITH (modulus 3, remainder 2) + +DROP TABLE tbl_list; +DROP TABLE tbl_hash; diff --git a/src/test/regress/sql/create_table.sql b/src/test/regress/sql/create_table.sql index 9b1adcb8ad..c82fca0a9a 100644 --- a/src/test/regress/sql/create_table.sql +++ b/src/test/regress/sql/create_table.sql @@ -971,3 +971,26 @@ create table part_column_drop_1_10 partition of \d part_column_drop \d part_column_drop_1_10 drop table part_column_drop; + +-- Auto generated partitions + +-- must fail because of wrong configuration +CREATE TABLE tbl_hash_fail (i int) PARTITION BY HASH (i) +CONFIGURATION (values in (1, 2), (3, 4) default partition tbl_default); + +-- must fail because of wrong configuration +CREATE TABLE tbl_list_fail (i int) PARTITION BY LIST (i) +CONFIGURATION (values in (1, 2), (1, 3)); + +CREATE TABLE tbl_list (i int) PARTITION BY LIST (i) +CONFIGURATION (values in (1, 2), (3, 4) default partition tbl_default); + +\d+ tbl_list + +CREATE TABLE tbl_hash (i int) PARTITION BY HASH (i) +CONFIGURATION (modulus 3); + +\d+ tbl_hash + +DROP TABLE tbl_list; +DROP TABLE tbl_hash; --------------FC4E4E20AFC6F93BE7445997--