public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning 50+ messages / 2 participants [nested] [flat]
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------19C79A137EF2D0DB3BE72F79 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1473 ++++++++++++++++++++++++------------------------- 1 file changed, 722 insertions(+), 751 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index d1e915c11a..2cd75a9673 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,123 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + <productname>PostgreSQL</productname> offers built-in support for the + following forms of partitioning: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + + If your application needs to use other forms of partitioning not listed + above, alternative methods such as inheritance and + <literal>UNION ALL</literal> views can be used instead. Such methods + offer flexibility but do not have some of the performance benefits + of built-in declarative partitioning. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,74 +2899,72 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> - Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + Partitions may themselves be defined as partitioned tables, referred to as + <firstterm>sub-partitioning</firstterm>. Partitions may have their own + indexes, constraints and default values, distinct from other partitions. + They do not inherit indexes from the partitioned table. See + <xref linkend="sql-createtable"> for more details on creating partitioned + tables and partitions. </para> <para> - Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see - <xref linkend="sql-altertable"> to learn more about the - <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. + It is not possible to turn a regular table into a partitioned table or + vice versa. However, it is possible to add a regular or partitioned table + containing data as a partition of a partitioned table, or remove a + partition from a partitioned table turning it into a standalone table; + see <xref linkend="sql-altertable"> to learn more about the + <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> + sub-commands. </para> <para> Individual partitions are linked to the partitioned table with inheritance - behind-the-scenes, however it is not possible to use some of the inheritance - features discussed in the previous section with partitioned tables and - partitions. For example, partitions cannot have any other parents than - the partitioned table it is a partition of, nor can a regular table inherit - from a partitioned table making the latter its parent. That means - partitioned table and partitions do not participate in inheritance with - regular tables. Since a partition hierarchy consisting of the - partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + behind-the-scenes, however it is not possible to use some of the + inheritance features discussed in the previous section with partitioned + tables and partitions. For example, a partition cannot have any parents + other than the partitioned table it is a partition of, nor can a regular + table inherit from a partitioned table making the latter its parent. + That means partitioned table and partitions do not participate in + inheritance with regular tables. Since a partition hierarchy consisting + of the partitioned table and its partitions is still an inheritance + hierarchy, all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> <para> Both <literal>CHECK</literal> and <literal>NOT NULL</literal> constraints of a partitioned table are always inherited by all its - partitions. There cannot be any <literal>CHECK</literal> constraints - that are marked <literal>NO INHERIT</literal>. + partitions. <literal>CHECK</literal> constraints that are marked + <literal>NO INHERIT</literal> are not allowed. </para> </listitem> <listitem> <para> The <literal>ONLY</literal> notation used to exclude child tables - would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + will cause an error for partitioned tables in the case of + schema-modifying commands such as most <literal>ALTER TABLE</literal> + commands. For example, dropping a column from only the parent does + not make sense for partitioned tables. </para> </listitem> <listitem> <para> - Partitions cannot have columns that are not present in the parent. - It is neither possible to specify columns when creating partitions - with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + Partitions cannot have columns that are not present in the parent. It + is neither possible to specify columns when creating partitions with + <command>CREATE TABLE</> nor is it possible to add columns to + partitions after-the-fact using <command>ALTER TABLE</>. Tables may be + added as a partition with <command>ALTER TABLE ... ATTACH PARTITION</> + only if their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,487 +2978,505 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), - although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), + although certain limitations exist in their usage. For example, data + inserted into the partitioned table is not routed to foreign table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> in this + case) and the list of column(s) to use as the partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + You may decide to use multiple columns in the partition key for range + partitioning if it's known that each of the selected columns will + divide the incoming data using successively more granular partition + criteria. Whereas using fewer columns may lead to coarser-grained + partitioning causing each partition to accept bigger set of data than + might be desirable. A query accessing the partitioned table will have + to scan fewer partitions if the conditions involve some or all of these + columns. For example, consider a table range partitioned using columns + <structfield>lastname</> and <structfield>firstname</> (in that order) + as the partition key. </para> - </listitem> - <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into the parent table that does not map + to one of the existing partitions will cause an error; appropriate + partition must be added manually. </para> - </listitem> - </itemizedlist> - </para> - - <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. - </para> - - </sect1> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> - - <indexterm> - <primary>partitioning</primary> - </indexterm> - - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible to specify + tablespace, storage parameters for each partition separately. + </para> - <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> + </para> - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> (or data that is directly inserted + into <structname>measurement_y2006m02</>, provided it satisfies its + partition constraint) will be further redirected to one of its + partitions based on the <structfield>peaktemp</> column. Partition + key specified may overlap with the parent's partition key, although + care must be taken when specifying the bounds of a sub-partition + such that the set of data it accepts constitutes a subset of what + the partition's own bounds allows; the system does not try to check + if that's really the case. + </para> + </listitem> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + Create an index on the key column(s), as well as any other indexes you + might want for every partition. - <varlistentry> - <term>Using Partitioned Tables</term> +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> + </para> + </listitem> <listitem> <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> - </varlistentry> - </variablelist> + </orderedlist> </para> <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: - - <variablelist> - <varlistentry> - <term>Range Partitioning</term> - - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> - - <varlistentry> - <term>List Partitioning</term> - - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - </sect2> + </sect3> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> <para> - To set up a partitioned table using inheritance, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <quote>master</quote> table, from which all of the - partitions will inherit. - </para> - <para> - This table will contain no data. Do not define any check - constraints on this table, unless you intend them to - be applied equally to all partitions. There is no point - in defining any indexes or unique constraints on it, either. - </para> - </listitem> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <listitem> - <para> - Create several <quote>child</quote> tables that each inherit from - the master table. Normally, these tables will not add any columns - to the set inherited from the master. - </para> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> - <para> - We will refer to the child tables as partitions, though they - are in every way normal <productname>PostgreSQL</> tables - (or, possibly, foreign tables). - </para> - </listitem> + <para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <listitem> - <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. - </para> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> + + <para> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <para> - Typical examples would be: <programlisting> -CHECK ( x = 1 ) -CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) -CHECK ( outletID >= 100 AND outletID < 200 ) +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; </programlisting> - Ensure that the constraints guarantee that there is no overlap - between the key values permitted in different partitions. A common - mistake is to set up range constraints like: + + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: + <programlisting> -CHECK ( outletID BETWEEN 100 AND 200 ) -CHECK ( outletID BETWEEN 200 AND 300 ) +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; + +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work + +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); </programlisting> - This is wrong since it is not clear which partition the key value - 200 belongs in. - </para> + </para> - <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may then drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </sect3> - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> + <para> + The following limitations apply to partitioned tables: + <itemizedlist> + <listitem> + <para> + It is not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. This also means that there is no way to create a primary + key, unique constraint, or exclusion constraint spanning all + partitions; it is only possible to constrain each leaf partition + individually. + </para> + </listitem> - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, foreign + keys referencing partitioned tables are not supported, nor are foreign + key references from a partitioned table to some other table. + </para> + </listitem> + + <listitem> + <para> + Using the <literal>ON CONFLICT</literal> clause with partitioned tables + will cause an error if <literal>DO UPDATE</literal> is specified as the + alternative action, because unique or exclusion constraints can only be + created on individual partitions. There is no support for enforcing + uniqueness (or an exclusion constraint) across an entire partitioning + hierarchy. + </para> + </listitem> + + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. + </para> + </listitem> - </orderedlist> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, + not the partitioned table as it is not supported. + </para> + </listitem> + </itemizedlist> </para> + </sect3> + </sect2> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> + While the built-in declarative partitioning is suitable for most + common use cases, there are some circumstances where a more flexible + approach may be useful. Partitioning can be implemented using table + inheritance, which allows for several features which are not supported + by declarative partitioning, such as: + + <itemizedlist> <listitem> <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. + Partitioning enforces a rule that all partitions must have exactly + the same set of columns as the parent, but table inheritance allows + children to have extra columns not present in the parent. </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> </listitem> <listitem> <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. + Table inheritance allows for multiple inheritance. </para> </listitem> <listitem> <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. + Declarative partitioning only supports list and range partitioning, + whereas table inheritance allows data to be divided in a manner of + the user's choosing. (Note, however, that if constraint exclusion is + unable to prune partitions effectively, query performance will be very + poor.) </para> </listitem> <listitem> <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. + Some operations require a stronger lock when using declarative + partitioning than when using table inheritance. For example, adding + or removing a partition to or from a partitioned table requires taking + an <literal>ACCESS EXCLUSIVE</literal> lock on the parent table, + whereas a <literal>SHARE UPDATE EXCLUSIVE</literal> lock is enough + in the case of regular inheritance. </para> </listitem> - - </orderedlist> + </itemizedlist> </para> - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> + <para> + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, use + the following steps: - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> + <orderedlist spacing="compact"> + <listitem> + <para> + Create the <quote>master</quote> table, from which all of the + partitions will inherit. This table will contain no data. Do not + define any check constraints on this table, unless you intend them + to be applied equally to all partitions. There is no point in + defining any indexes or unique constraints on it, either. For our + example, master table is the <structname>measurement</structname> + table as originally defined. + </para> + </listitem> - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> + <listitem> + <para> + Create several <quote>child</quote> tables that each inherit from + the master table. Normally, these tables will not add any columns + to the set inherited from the master. + </para> - <listitem> - <para> - Next we create one partition for each active month: + <para> + We will refer to the child tables as partitions, though they are + in every way normal <productname>PostgreSQL</> tables (or, possibly, + foreign tables). + </para> + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. <programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); ... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); </programlisting> + </para> + </listitem> - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> + <listitem> + <para> + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. + </para> - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> + <para> + Typical examples would be: +<programlisting> +CHECK ( x = 1 ) +CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' )) +CHECK ( outletID >= 100 AND outletID < 200 ) +</programlisting> + Ensure that the constraints guarantee that there is no overlap + between the key values permitted in different partitions. A common + mistake is to set up range constraints like: +<programlisting> +CHECK ( outletID BETWEEN 100 AND 200 ) +CHECK ( outletID BETWEEN 200 AND 300 ) +</programlisting> + This is wrong since it is not clear which partition the key value + 200 belongs in. + </para> - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + <para> + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> - </para> - </listitem> + </para> - <listitem> - <para> - We probably need indexes on the key columns too: + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> + </listitem> + <listitem> + <para> + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> + </para> + </listitem> - We choose not to add further indexes at this time. - </para> - </listitem> - - <listitem> - <para> - We want our application to be able to say <literal>INSERT INTO - measurement ...</> and have the data be redirected into the - appropriate partition table. We can arrange that by attaching - a suitable trigger function to the master table. - If data will be added only to the latest partition, we can - use a very simple trigger function: + <listitem> + <para> + We want our application to be able to say <literal>INSERT INTO + measurement ...</> and have the data be redirected into the + appropriate partition table. We can arrange that by attaching + a suitable trigger function to the master table. + If data will be added only to the latest partition, we can + use a very simple trigger function: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3363,9 +3488,11 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> - After creating the function, we create a trigger which - calls the trigger function: + <para> + After creating the function, we create a trigger which + calls the trigger function: <programlisting> CREATE TRIGGER insert_measurement_trigger @@ -3373,15 +3500,15 @@ CREATE TRIGGER insert_measurement_trigger FOR EACH ROW EXECUTE PROCEDURE measurement_insert_trigger(); </programlisting> - We must redefine the trigger function each month so that it always - points to the current partition. The trigger definition does - not need to be updated, however. - </para> + We must redefine the trigger function each month so that it always + points to the current partition. The trigger definition does + not need to be updated, however. + </para> - <para> - We might want to insert data and have the server automatically - locate the partition into which the row should be added. We - could do this with a more complex trigger function, for example: + <para> + We might want to insert data and have the server automatically + locate the partition into which the row should be added. We + could do this with a more complex trigger function, for example: <programlisting> CREATE OR REPLACE FUNCTION measurement_insert_trigger() @@ -3393,183 +3520,120 @@ BEGIN ELSIF ( NEW.logdate >= DATE '2006-03-01' AND NEW.logdate < DATE '2006-04-01' ) THEN INSERT INTO measurement_y2006m03 VALUES (NEW.*); - ... - ELSIF ( NEW.logdate >= DATE '2008-01-01' AND - NEW.logdate < DATE '2008-02-01' ) THEN - INSERT INTO measurement_y2008m01 VALUES (NEW.*); - ELSE - RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; - END IF; - RETURN NULL; -END; -$$ -LANGUAGE plpgsql; -</programlisting> - - The trigger definition is the same as before. - Note that each <literal>IF</literal> test must exactly match the - <literal>CHECK</literal> constraint for its partition. - </para> - - <para> - While this function is more complex than the single-month case, - it doesn't need to be updated as often, since branches can be - added in advance of being needed. - </para> - - <note> - <para> - In practice it might be best to check the newest partition first, - if most inserts go into that partition. For simplicity we have - shown the trigger's tests in the same order as in other parts - of this example. - </para> - </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - Create the <structname>measurement</> table as a partitioned table: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); + ... + ELSIF ( NEW.logdate >= DATE '2008-01-01' AND + NEW.logdate < DATE '2008-02-01' ) THEN + INSERT INTO measurement_y2008m01 VALUES (NEW.*); + ELSE + RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!'; + END IF; + RETURN NULL; +END; +$$ +LANGUAGE plpgsql; </programlisting> - </para> - </listitem> - <listitem> - <para> - Then create partitions as follows: + The trigger definition is the same as before. + Note that each <literal>IF</literal> test must exactly match the + <literal>CHECK</literal> constraint for its partition. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> - </para> - </listitem> + <para> + While this function is more complex than the single-month case, + it doesn't need to be updated as often, since branches can be + added in advance of being needed. + </para> - <listitem> - <para> - Create indexes on the key columns just like in case of inheritance - partitions. - </para> - </listitem> - </orderedlist> + <note> + <para> + In practice it might be best to check the newest partition first, + if most inserts go into that partition. For simplicity we have + shown the trigger's tests in the same order as in other parts + of this example. + </para> + </note> - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: + <para> + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> - </para> - - <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> + A rule has significantly more overhead than a trigger, but the + overhead is paid once per query rather than once per row, so this + method might be advantageous for bulk-insert situations. In most + cases, however, the trigger method will offer better performance. + </para> - </sect2> + <para> + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. + </para> + </listitem> - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> + </para> - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + <para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> - - <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> + </para> - When using a partitioned table: + <para> + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> + </para> - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. - </para> - - <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + <para> + To add a new partition to handle new data, create an empty partition + just as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,17 +3641,9 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may want to create the new table outside the partition + structure, and make it a partition after the data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 @@ -3598,31 +3654,64 @@ ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 -- possibly some other data preparation work ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> - The last of the above commands when using a partitioned table would be: + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> + + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> - <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. - </para> - </tip> - </sect2> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3721,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,160 +3805,15 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - Using the <literal>ON CONFLICT</literal> clause with partitioned tables - will cause an error if <literal>DO UPDATE</literal> is specified as the - alternative action, because unique or exclusion constraints can only be - created on individual partitions. There is no support for enforcing - uniqueness (or an exclusion constraint) across an entire partitioning - hierarchy. - </para> - </listitem> - - </itemizedlist> + Constraint exclusion is also used for declarative partitioning, however + it is not required to create <literal>CHECK</literal> constraints for + individual partitions as when using table inheritance. </para> <para> - The following caveats apply to constraint exclusion, which is currently - used by both inheritance and partitioned tables: + The following caveats apply to constraint exclusion, which is used by + both inheritance and partitioned tables: <itemizedlist> <listitem> @@ -3909,6 +3854,32 @@ ANALYZE measurement; </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------4AB4E704D39F2BF05E79CD67 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------4AB4E704D39F2BF05E79CD67-- ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning @ 2017-03-03 07:39 amit <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: amit @ 2017-03-03 07:39 UTC (permalink / raw) Merge sections Partitioned Tables and Partitioning into one section called Table Partitioning and Related Solutions. --- doc/src/sgml/ddl.sgml | 1359 +++++++++++++++++++++++++------------------------ 1 file changed, 707 insertions(+), 652 deletions(-) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 09b5b3ff70..a2dd39df54 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -2772,14 +2772,181 @@ VALUES ('Albany', NULL, NULL, 'NY'); </sect2> </sect1> - <sect1 id="ddl-partitioned-tables"> - <title>Partitioned Tables</title> + <sect1 id="ddl-partitioning"> + <title>Table Partitioning and Related Solutions</title> + + <indexterm> + <primary>partitioning</primary> + </indexterm> + + <indexterm> + <primary>table</primary> + <secondary>partitioning</secondary> + </indexterm> <indexterm> <primary>partitioned table</primary> </indexterm> <para> + <productname>PostgreSQL</productname> supports basic table + partitioning. This section describes why and how to implement + partitioning as part of your database design. + </para> + + <sect2 id="ddl-partitioning-overview"> + <title>Overview</title> + + <para> + Partitioning refers to splitting what is logically one large table into + smaller physical pieces. Partitioning can provide several benefits: + <itemizedlist> + <listitem> + <para> + Query performance can be improved dramatically in certain situations, + particularly when most of the heavily accessed rows of the table are in a + single partition or a small number of partitions. The partitioning + substitutes for leading columns of indexes, reducing index size and + making it more likely that the heavily-used parts of the indexes + fit in memory. + </para> + </listitem> + + <listitem> + <para> + When queries or updates access a large percentage of a single + partition, performance can be improved by taking advantage + of sequential scan of that partition instead of using an + index and random access reads scattered across the whole table. + </para> + </listitem> + + <listitem> + <para> + Bulk loads and deletes can be accomplished by adding or removing + partitions, if that requirement is planned into the partitioning design. + Doing <command>ALTER TABLE DETACH PARTITION</> followed by + <command>DROP TABLE</> is far faster than a bulk operation. These + commands also entirely avoid the <command>VACUUM</command> overhead + caused by a bulk <command>DELETE</>. + </para> + </listitem> + + <listitem> + <para> + Seldom-used data can be migrated to cheaper and slower storage media. + </para> + </listitem> + </itemizedlist> + + The benefits will normally be worthwhile only when a table would + otherwise be very large. The exact point at which a table will + benefit from partitioning depends on the application, although a + rule of thumb is that the size of the table should exceed the physical + memory of the database server. + </para> + + <para> + The following forms of partitioning can be implemented in + <productname>PostgreSQL</productname>: + + <variablelist> + <varlistentry> + <term>Range Partitioning</term> + + <listitem> + <para> + The table is partitioned into <quote>ranges</quote> defined + by a key column or set of columns, with no overlap between + the ranges of values assigned to different partitions. For + example one might partition by date ranges, or by ranges of + identifiers for particular business objects. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>List Partitioning</term> + + <listitem> + <para> + The table is partitioned by explicitly listing which key values + appear in each partition. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + The following partitioning methods are currently supported: + + <variablelist> + <varlistentry> + <term>Declarative Partitioning</term> + + <listitem> + <para> + One creates a <firstterm>partitioned table</firstterm> by specifying + the partitioning method and a set of columns as the partition key. + <firstterm>Partitions</firstterm>, which contain actual data inserted + into the table, are created by specifying what subset of the data it + accepts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using inheritance</term> + + <listitem> + <para> + Each partition must be created as a child table of a single parent + table. The parent table itself is normally empty; it exists just to + represent the entire data set. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Using UNION ALL views</term> + + <listitem> + <para> + One can define a <literal>UNION ALL</literal> view over + <literal>SELECT</literal> on individual tables, each of which + contains a partition of data. Partitions are added or removed + by updating the view definition. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>Accessing Tables using BRIN Indexes</term> + + <listitem> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index is, + designed for handling very large tables in which certain columns + have some natural physical location within the table. Scanning + a large table using a <acronym>BRIN</acronym> index results in + reading only a portion of the table, which is often why partitioning + is implemented. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Each of the above mentioned methods is described below. + </para> + </sect2> + + <sect2 id="ddl-partitioning-declarative"> + <title>Declarative Partitioning</title> + + <para> PostgreSQL offers a way to specify how to divide a table into pieces called partitions. The table that is divided is referred to as a <firstterm>partitioned table</firstterm>. The specification consists @@ -2790,25 +2957,29 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> All rows inserted into a partitioned table will be routed to one of the <firstterm>partitions</firstterm> based on the value of the partition - key. Each partition has a subset defined by its <firstterm>partition - bounds</firstterm>. Currently supported partitioning methods include - range and list, wherein each partition is assigned a range of keys or - a list of keys, respectively. + key. Each partition has a subset of the data defined by its + <firstterm>partition bounds</firstterm>. Currently supported + partitioning methods include range and list, where each partition is + assigned a range of keys and a list of keys, respectively. </para> <para> Partitions may have their own indexes, constraints and default values, - distinct from other partitions. Partitions do not inherit indexes from - the partitioned table. + distinct from other partitions. Partitions do not currently inherit + indexes from the partitioned table. + </para> + + <para> + See <xref linkend="sql-createtable"> for more details creating partitioned + tables and partitions. </para> <para> Partitions may themselves be defined as partitioned tables, referred to as - <firstterm>sub-partitioning</firstterm>. See <xref linkend="sql-createtable"> - for more details creating partitioned tables and partitions. It is not - currently possible to alter a regular table into a partitioned table or - vice versa. However, it is possible to add a regular table containing - data into a partition of a partitioned table, or remove a partition; see + <firstterm>sub-partitioning</firstterm>. It is not currently possible to + alter a regular table into a partitioned table or vice versa. However, + it is possible to add a regular or partitioned table containing data into + a partition of a partitioned table, or remove a partition; see <xref linkend="sql-altertable"> to learn more about the <command>ATTACH PARTITION</> and <command>DETACH PARTITION</> sub-commands. </para> @@ -2823,8 +2994,8 @@ VALUES ('Albany', NULL, NULL, 'NY'); partitioned table and partitions do not participate in inheritance with regular tables. Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, - all the normal rules of inheritance apply as described in the previous - section (<xref linkend="ddl-inherit">) with some exceptions, most notably: + all the normal rules of inheritance apply as described in + <xref linkend="ddl-inherit"> with some exceptions, most notably: <itemizedlist> <listitem> @@ -2840,13 +3011,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); <para> The <literal>ONLY</literal> notation used to exclude child tables would either cause error or will be ignored in some cases for - partitioned tables. For example, specifying <literal>ONLY</literal> - when querying data from a partitioned table would not make much sense, - because all the data is contained in partitions, so this raises an - error. Specifying <literal>ONLY</literal> when modifying schema is - not desirable in certain cases with partitioned tables where it may be - fine for regular inheritance parents (for example, dropping a column - from only the parent); an error will be thrown in that case. + partitioned tables. Specifying <literal>ONLY</literal> when modifying + schema is not desirable in certain cases with partitioned tables + whereas it may be fine for regular inheritance parents (for example, + dropping a column from only the parent); an error will be thrown in + that case. </para> </listitem> @@ -2855,9 +3024,9 @@ VALUES ('Albany', NULL, NULL, 'NY'); Partitions cannot have columns that are not present in the parent. It is neither possible to specify columns when creating partitions with <command>CREATE TABLE</> nor is it possible to add columns to - partitions using <command>ALTER TABLE</>. Tables may be added with - <command>ALTER TABLE ... ATTACH PARTITION</> if their columns exactly - match the parent, including oids. + partitions using <command>ALTER TABLE</>. Tables may be added as a + partition with <command>ALTER TABLE ... ATTACH PARTITION</> only if + their columns exactly match the parent, including oids. </para> </listitem> @@ -2871,199 +3040,353 @@ VALUES ('Albany', NULL, NULL, 'NY'); </para> <para> - Partitions can also be foreign tables (see <xref linkend="ddl-foreign-data">), + Partitions can also be foreign tables (see <xref linkend="sql-createforeigntable">), although certain limitations exist currently in their usage. For example, - data inserted into the partitioned table cannot be routed to foreign table - partitions. + data inserted into the partitioned table is currently not routed to foreign + table partitions. </para> + <sect3 id="ddl-partitioning-declarative-example"> + <title>Example</title> + <para> - There are currently the following limitations of using partitioned tables: - <itemizedlist> + Suppose we are constructing a database for a large ice cream company. + The company measures peak temperatures every day as well as ice cream + sales in each region. Conceptually, we want a table like: + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +); +</programlisting> + + We know that most queries will access just the last week's, month's or + quarter's data, since the main use of this table will be to prepare + online reports for management. To reduce the amount of old data that + needs to be stored, we decide to only keep the most recent 3 years + worth of data. At the beginning of each month we will remove the oldest + month's data. In this situation we can use partitioning to help us meet + all of our different requirements for the measurements table. + </para> + + <para> + To use declarative partitioning in this case, use the following steps: + + <orderedlist spacing="compact"> <listitem> <para> - It is currently not possible to add same set of indexes on all partitions - automatically. Indexes must be added to each partition with separate - commands. + Create <structname>measurement</structname> table as a partitioned + table by specifying the <literal>PARTITION BY</literal> clause, which + includes the partitioning method (<literal>RANGE</literal> or + <literal>LIST</literal>) and the list of column(s) to use as the + partition key. + +<programlisting> +CREATE TABLE measurement ( + city_id int not null, + logdate date not null, + peaktemp int, + unitsales int +) PARTITION BY RANGE (logdate); +</programlisting> </para> - </listitem> - <listitem> + <note> + <para> + To decide when to use multiple columns in the partition key for range + partitioning, consider whether queries accessing the partitioned + in question will include conditions that involve multiple columns, + especially the columns being considered to be the partition key. + If so, the optimizer can create a plan that will scan fewer partitions + if a query's conditions are such that there is equality constraint on + leading partition key columns, because they limit the number of + partitions of interest. The first partition key column with + inequality constraint also further eliminates some partitions of + those chosen by equality constraints on earlier columns. + </para> + </note> + <para> - It is currently not possible to define indexes on partitioned tables - that include all rows from all partitions in one global index. - Consequently, it is not possible to create constraints that are realized - using an index such as <literal>UNIQUE</>. + To be able to insert data into this table, one must create partitions, + as described below. </para> </listitem> <listitem> <para> - Since primary keys are not supported on partitioned tables, - foreign keys referencing partitioned tables are not supported, nor - are foreign key references from a partitioned table to some other table. + Create partitions. Each partition's definition must specify the bounds + that correspond to the partitioning method and partition key of the + parent. Note that specifying bounds such that the new partition's + values will overlap with those in one or more existing partitions will + cause an error. Inserting data into into the parent table that does + not map to one of the existing partitions will cause an error; + appropriate partition must be added manually. + </para> + + <para> + Partitions thus created are in every way normal <productname>PostgreSQL</> + tables (or, possibly, foreign tables). It is possible, for example, to + specify tablespace, storage parameters for each partition separately. + </para> + + <para> + It is not necessary to create table constraints describing partition + boundary condition for partitions. Instead, partition constraints are + generated implicitly from the partition bound specification whenever + there is need to refer to them. Also, since any data inserted into the + parent table is automatically inserted into the appropriate partition, + it is not necessary to create triggers for the same. + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + +CREATE TABLE measurement_y2006m03 PARTITION OF measurement + FOR VALUES FROM ('2006-03-01') TO ('2006-04-01') + +... +CREATE TABLE measurement_y2007m11 PARTITION OF measurement + FOR VALUES FROM ('2007-11-01') TO ('2007-12-01') + +CREATE TABLE measurement_y2007m12 PARTITION OF measurement + FOR VALUES FROM ('2007-12-01') TO ('2008-01-01') + TABLESPACE fasttablespace; + +CREATE TABLE measurement_y2008m01 PARTITION OF measurement + FOR VALUES FROM ('2008-01-01') TO ('2008-02-01') + TABLESPACE fasttablespace + WITH (parallel_workers = 4); +</programlisting> </para> + + <note> + <para> + To implement sub-partitioning, specify the + <literal>PARTITION BY</literal> clause in the commands used to create + individual partitions, for example: + +<programlisting> +CREATE TABLE measurement_y2006m02 PARTITION OF measurement + FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') + PARTITION BY RANGE (peaktemp); +</programlisting> + + After creating partitions of <structname>measurement_y2006m02</>, + any data inserted into <structname>measurement</> that is mapped to + <structname>measurement_y2006m02</> will be further redirected to one + of its partitions based on the <structfield>peaktemp</> column. + Partition key specified may overlap with the parent's partition key, + although care must be taken when specifying the bounds of + sub-partitions such that the accepted set of data constitutes a + subset of what a partition's own bounds allows; the system does not + try to check if that's really the case. + </para> + </note> </listitem> <listitem> <para> - Row triggers, if necessary, must be defined on individual partitions, not - the partitioned table as it is currently not supported. + Create an index on the key column(s), + as well as any other indexes you might want for every partition. + Note that it is currently not supported to propagate index definition + from the master partitioned table to its partitions; in fact, it is + not possible to define indexes on partitioned tables in the first + place. This might change in future releases. + +<programlisting> +CREATE INDEX ON measurement_y2006m02 (logdate); +CREATE INDEX ON measurement_y2006m03 (logdate); +... +CREATE INDEX ON measurement_y2007m11 (logdate); +CREATE INDEX ON measurement_y2007m12 (logdate); +CREATE INDEX ON measurement_y2008m01 (logdate); +</programlisting> </para> </listitem> - </itemizedlist> + + <listitem> + <para> + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. + </para> + </listitem> + </orderedlist> </para> <para> - A detailed example that shows how to use partitioned tables is discussed in - the next chapter. + In the above example we would be creating a new partition each month, so + it might be wise to write a script that generates the required DDL + automatically. </para> - - </sect1> + </sect3> - <sect1 id="ddl-partitioning"> - <title>Partitioning</title> + <sect3 id="ddl-partitioning-declarative-maintenance"> + <title>Partition Maintenance</title> - <indexterm> - <primary>partitioning</primary> - </indexterm> + <para> + Normally the set of partitions established when initially defining the + the table are not intended to remain static. It is common to want to + remove old partitions of data and periodically add new partitions for + new data. One of the most important advantages of partitioning is + precisely that it allows this otherwise painful task to be executed + nearly instantaneously by manipulating the partition structure, rather + than physically moving large amounts of data around. + </para> - <indexterm> - <primary>table</primary> - <secondary>partitioning</secondary> - </indexterm> + <para> + The simplest option for removing old data is simply to drop the partition + that is no longer necessary: +<programlisting> +DROP TABLE measurement_y2006m02; +</programlisting> + This can very quickly delete millions of records because it doesn't have + to individually delete every record. Note however that the above command + requires taking an <literal>ACCESS EXCLUSIVE</literal> lock on the parent + table. + </para> <para> - <productname>PostgreSQL</productname> supports basic table - partitioning. This section describes why and how to implement - partitioning as part of your database design. - </para> + Another option that is often preferable is to remove the partition from + the partitioned table but retain access to it as a table in its own + right: - <sect2 id="ddl-partitioning-overview"> - <title>Overview</title> +<programlisting> +ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +</programlisting> + + This allows further operations to be performed on the data before + it is dropped. For example, this is often a useful time to back up + the data using <command>COPY</>, <application>pg_dump</>, or + similar tools. It might also be a useful time to aggregate data + into smaller formats, perform other data manipulations, or run + reports. + </para> <para> - Partitioning refers to splitting what is logically one large table - into smaller physical pieces. - Partitioning can provide several benefits: - <itemizedlist> - <listitem> - <para> - Query performance can be improved dramatically in certain situations, - particularly when most of the heavily accessed rows of the table are in a - single partition or a small number of partitions. The partitioning - substitutes for leading columns of indexes, reducing index size and - making it more likely that the heavily-used parts of the indexes - fit in memory. - </para> - </listitem> + Similarly we can add a new partition to handle new data. We can create an + empty partition in the partitioned table just as the original partitions + were created above: - <listitem> - <para> - When queries or updates access a large percentage of a single - partition, performance can be improved by taking advantage - of sequential scan of that partition instead of using an - index and random access reads scattered across the whole table. - </para> - </listitem> +<programlisting> +CREATE TABLE measurement_y2008m02 PARTITION OF measurement + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01') + TABLESPACE fasttablespace; +</programlisting> - <listitem> - <para> - Bulk loads and deletes can be accomplished by adding or removing - partitions, if that requirement is planned into the partitioning design. - <command>ALTER TABLE NO INHERIT</> or <command>ALTER TABLE DETACH PARTITION</> - and <command>DROP TABLE</> are both far faster than a bulk operation. - These commands also entirely avoid the <command>VACUUM</command> - overhead caused by a bulk <command>DELETE</>. - </para> - </listitem> + As an alternative, it is sometimes more convenient to create the + new table outside the partition structure, and make it a proper + partition later. This allows the data to be loaded, checked, and + transformed prior to it appearing in the partitioned table: - <listitem> - <para> - Seldom-used data can be migrated to cheaper and slower storage media. - </para> - </listitem> - </itemizedlist> +<programlisting> +CREATE TABLE measurement_y2008m02 + (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS) + TABLESPACE fasttablespace; - The benefits will normally be worthwhile only when a table would - otherwise be very large. The exact point at which a table will - benefit from partitioning depends on the application, although a - rule of thumb is that the size of the table should exceed the physical - memory of the database server. - </para> +ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 + CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); - <para> - Currently, <productname>PostgreSQL</productname> supports partitioning - using two methods: +\copy measurement_y2008m02 from 'measurement_y2008m02' +-- possibly some other data preparation work - <variablelist> - <varlistentry> - <term>Using Table Inheritance</term> +ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 + FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +</programlisting> + </para> - <listitem> - <para> - Each partition must be created as a child table of a single parent - table. The parent table itself is normally empty; it exists just to - represent the entire data set. You should be familiar with - inheritance (see <xref linkend="ddl-inherit">) before attempting to - set up partitioning with it. This was the only method to implement - partitioning in older versions. - </para> - </listitem> - </varlistentry> + <tip> + <para> + Before running the <command>ATTACH PARTITION</> command, it is + recommended to create a <literal>CHECK</> constraint on the table to + be attached describing the desired partition constraint. Using the + same, system is able to skip the scan to validate the implicit + partition constraint. Without such a constraint, the table will be + scanned to validate the partition constraint, while holding an + <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. + One may want to drop the constraint after <command>ATTACH PARTITION</> + is finished, because it is no longer necessary. + </para> + </tip> + </sect3> - <varlistentry> - <term>Using Partitioned Tables</term> + <sect3 id="ddl-partitioning-declarative-limitations"> + <title>Limitations</title> - <listitem> - <para> - See last section for some general information: - <xref linkend="ddl-partitioned-tables"> - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <para> + There are currently the following limitations of using partitioned tables: + <itemizedlist> + <listitem> + <para> + It is currently not possible to add same set of indexes on all partitions + automatically. Indexes must be added to each partition with separate + commands. + </para> + </listitem> - <para> - The following forms of partitioning can be implemented in - <productname>PostgreSQL</productname> using either of the above mentioned - methods, although the latter provides dedicated syntax for each: + <listitem> + <para> + It is currently not possible to define indexes on partitioned tables + that include all rows from all partitions in one global index. + Consequently, it is not possible to create constraints that are realized + using an index such as <literal>UNIQUE</>. + </para> + </listitem> - <variablelist> - <varlistentry> - <term>Range Partitioning</term> + <listitem> + <para> + Since primary keys are not supported on partitioned tables, + foreign keys referencing partitioned tables are not supported, nor + are foreign key references from a partitioned table to some other table. + </para> + </listitem> - <listitem> - <para> - The table is partitioned into <quote>ranges</quote> defined - by a key column or set of columns, with no overlap between - the ranges of values assigned to different partitions. For - example one might partition by date ranges, or by ranges of - identifiers for particular business objects. - </para> - </listitem> - </varlistentry> + <listitem> + <para> + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clause are currently not allowed on partitioned tables. + </para> + </listitem> - <varlistentry> - <term>List Partitioning</term> + <listitem> + <para> + An <command>UPDATE</> that causes a row to move from one partition to + another fails, because the new value of the row fails to satisfy the + implicit partition constraint of the original partition. This might + change in future releases. + </para> + </listitem> - <listitem> - <para> - The table is partitioned by explicitly listing which key values - appear in each partition. - </para> - </listitem> - </varlistentry> - </variablelist> - </para> + <listitem> + <para> + Row triggers, if necessary, must be defined on individual partitions, not + the partitioned table as it is currently not supported. + </para> + </listitem> + </itemizedlist> + </para> + </sect3> </sect2> - <sect2 id="ddl-partitioning-implementation"> - <title>Implementing Partitioning</title> + <sect2 id="ddl-partitioning-implementation-inheritance"> + <title>Implementation Using Inheritance</title> + <para> + In some cases, one may want to add columns to partitions that are not + present in the parent table which is not possible to do with the above + method. For such cases, partitioning can be implemented using + inheritance (see <xref linkend="ddl-inherit">). + </para> + + <sect3 id="ddl-partitioning-inheritance-example"> + <title>Example</title> <para> - To set up a partitioned table using inheritance, do the following: + We use the same <structname>measurement</structname> table we used + above. To implement it as a partitioned table using inheritance, do the + following: <orderedlist spacing="compact"> <listitem> <para> @@ -3076,6 +3399,11 @@ VALUES ('Albany', NULL, NULL, 'NY'); be applied equally to all partitions. There is no point in defining any indexes or unique constraints on it, either. </para> + + <para> + In case of our example, master table is the original + <structname>measurement</structname> as originally defined. + </para> </listitem> <listitem> @@ -3090,12 +3418,27 @@ VALUES ('Albany', NULL, NULL, 'NY'); are in every way normal <productname>PostgreSQL</> tables (or, possibly, foreign tables). </para> + + <para> + This solves one of our problems: deleting old data. Each + month, all we will need to do is perform a <command>DROP + TABLE</command> on the oldest child table and create a new + child table for the new month's data. +<programlisting> +CREATE TABLE measurement_y2006m02 () INHERITS (measurement); +CREATE TABLE measurement_y2006m03 () INHERITS (measurement); +... +CREATE TABLE measurement_y2007m11 () INHERITS (measurement); +CREATE TABLE measurement_y2007m12 () INHERITS (measurement); +CREATE TABLE measurement_y2008m01 () INHERITS (measurement); +</programlisting> + </para> </listitem> <listitem> <para> - Add table constraints to the partition tables to define the - allowed key values in each partition. + Add non-overlapping table constraints to the partition tables to + define the allowed key values in each partition. </para> <para> @@ -3117,230 +3460,53 @@ CHECK ( outletID BETWEEN 200 AND 300 ) </para> <para> - Note that there is no difference in - syntax between range and list partitioning; those terms are - descriptive only. - </para> - </listitem> - - <listitem> - <para> - For each partition, create an index on the key column(s), - as well as any other indexes you might want. (The key index is - not strictly necessary, but in most scenarios it is helpful. - If you intend the key values to be unique then you should - always create a unique or primary-key constraint for each - partition.) - </para> - </listitem> - - <listitem> - <para> - Optionally, define a trigger or rule to redirect data inserted into - the master table to the appropriate partition. - </para> - </listitem> - - <listitem> - <para> - Ensure that the <xref linkend="guc-constraint-exclusion"> - configuration parameter is not disabled in - <filename>postgresql.conf</>. - If it is, queries will not be optimized as desired. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - To use partitioned tables, do the following: - <orderedlist spacing="compact"> - <listitem> - <para> - Create <quote>master</quote> table as a partitioned table by - specifying the <literal>PARTITION BY</literal> clause, which includes - the partitioning method (<literal>RANGE</literal> or - <literal>LIST</literal>) and the list of column(s) to use as the - partition key. To be able to insert data into the table, one must - create partitions, as described below. - </para> - - <note> - <para> - To decide when to use multiple columns in the partition key for range - partitioning, consider whether queries accessing the partitioned - in question will include conditions that involve multiple columns, - especially the columns being considered to be the partition key. - If so, the optimizer can create a plan that will scan fewer partitions - if a query's conditions are such that there is equality constraint on - leading partition key columns, because they limit the number of - partitions of interest. The first partition key column with - inequality constraint also further eliminates some partitions of - those chosen by equality constraints on earlier columns. - </para> - </note> - </listitem> - - <listitem> - <para> - Create partitions of the master partitioned table, with the partition - bounds specified for each partition matching the partitioning method - and partition key of the master table. Note that specifying partition - bounds such that the new partition's values will overlap with one or - more existing partitions will cause an error. It is only after - creating partitions that one is able to insert data into the master - partitioned table, provided it maps to one of the existing partitions. - If a data row does not map to any of the existing partitions, it will - cause an error. - </para> - - <para> - Partitions thus created are also in every way normal - <productname>PostgreSQL</> tables (or, possibly, foreign tables), - whereas partitioned tables differ in a number of ways. - </para> - - <para> - It is not necessary to create table constraints for partitions. - Instead, partition constraints are generated implicitly whenever - there is a need to refer to them. Also, since any data inserted into - the master partitioned table is automatically inserted into the - appropriate partition, it is not necessary to create triggers for the - same. - </para> - </listitem> - - <listitem> - <para> - Just like with inheritance, create an index on the key column(s), - as well as any other indexes you might want for every partition. - Note that it is currently not supported to propagate index definition - from the master partitioned table to its partitions; in fact, it is - not possible to define indexes on partitioned tables in the first - place. This might change in future releases. - </para> - </listitem> - - <listitem> - <para> - Currently, partitioned tables also depend on constraint exclusion - for query optimization, so ensure that the - <xref linkend="guc-constraint-exclusion"> configuration parameter is - not disabled in <filename>postgresql.conf</>. This might change in - future releases. - </para> - </listitem> - - </orderedlist> - </para> - - <para> - For example, suppose we are constructing a database for a large - ice cream company. The company measures peak temperatures every - day as well as ice cream sales in each region. Conceptually, - we want a table like: - -<programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -); -</programlisting> - - We know that most queries will access just the last week's, month's or - quarter's data, since the main use of this table will be to prepare - online reports for management. - To reduce the amount of old data that needs to be stored, we - decide to only keep the most recent 3 years worth of data. At the - beginning of each month we will remove the oldest month's data. - </para> - - <para> - In this situation we can use partitioning to help us meet all of our - different requirements for the measurements table. Following the - steps outlined above for both methods, partitioning can be set up as - follows: - </para> - - <para> - <orderedlist spacing="compact"> - <listitem> - <para> - The master table is the <structname>measurement</> table, declared - exactly as above. - </para> - </listitem> - - <listitem> - <para> - Next we create one partition for each active month: - -<programlisting> -CREATE TABLE measurement_y2006m02 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2006m03 ( ) INHERITS (measurement); -... -CREATE TABLE measurement_y2007m11 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2007m12 ( ) INHERITS (measurement); -CREATE TABLE measurement_y2008m01 ( ) INHERITS (measurement); -</programlisting> - - Each of the partitions are complete tables in their own right, - but they inherit their definitions from the - <structname>measurement</> table. - </para> - - <para> - This solves one of our problems: deleting old data. Each - month, all we will need to do is perform a <command>DROP - TABLE</command> on the oldest child table and create a new - child table for the new month's data. - </para> - </listitem> - - <listitem> - <para> - We must provide non-overlapping table constraints. Rather than - just creating the partition tables as above, the table creation - script should really be: + It would be better to instead create partitions as follows: <programlisting> CREATE TABLE measurement_y2006m02 ( CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2006m03 ( CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' ) ) INHERITS (measurement); + ... CREATE TABLE measurement_y2007m11 ( CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2007m12 ( CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' ) ) INHERITS (measurement); + CREATE TABLE measurement_y2008m01 ( CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) ) INHERITS (measurement); </programlisting> </para> + + <para> + Note that there is no difference in syntax between range and list + partitioning; those terms are descriptive only. + </para> </listitem> <listitem> <para> - We probably need indexes on the key columns too: - + For each partition, create an index on the key column(s), + as well as any other indexes you might want. (The key index is + not strictly necessary, but in most scenarios it is helpful. + If you intend the key values to be unique then you should + always create a unique or primary-key constraint for each + partition.) <programlisting> CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate); CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate); -... CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate); CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate); CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate); </programlisting> - - We choose not to add further indexes at this time. </para> </listitem> @@ -3363,7 +3529,9 @@ END; $$ LANGUAGE plpgsql; </programlisting> + </para> + <para> After creating the function, we create a trigger which calls the trigger function: @@ -3425,151 +3593,88 @@ LANGUAGE plpgsql; of this example. </para> </note> - </listitem> - </orderedlist> - </para> - - <para> - Steps when using a partitioned table are as follows: - </para> - <para> - <orderedlist spacing="compact"> - <listitem> <para> - Create the <structname>measurement</> table as a partitioned table: + A different approach to redirecting inserts into the appropriate + partition table is to set up rules, instead of a trigger, on the + master table. For example: <programlisting> -CREATE TABLE measurement ( - city_id int not null, - logdate date not null, - peaktemp int, - unitsales int -) PARTITION BY RANGE (logdate); +CREATE RULE measurement_insert_y2006m02 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) +DO INSTEAD + INSERT INTO measurement_y2006m02 VALUES (NEW.*); +... +CREATE RULE measurement_insert_y2008m01 AS +ON INSERT TO measurement WHERE + ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) +DO INSTEAD + INSERT INTO measurement_y2008m01 VALUES (NEW.*); </programlisting> + + A rule has significantly more overhead than a trigger, but the overhead + is paid once per query rather than once per row, so this method might + be advantageous for bulk-insert situations. In most cases, however, + the trigger method will offer better performance. </para> - </listitem> - <listitem> <para> - Then create partitions as follows: + Be aware that <command>COPY</> ignores rules. If you want to + use <command>COPY</> to insert data, you'll need to copy into the + correct partition table rather than into the master. <command>COPY</> + does fire triggers, so you can use it normally if you use the trigger + approach. + </para> -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01'); -CREATE TABLE measurement_y2006m03 PARTITION OF measurement - FOR VALUES FROM ('2006-03-01') TO ('2006-04-01'); -... -CREATE TABLE measurement_y2007m11 PARTITION OF measurement - FOR VALUES FROM ('2007-11-01') TO ('2007-12-01'); -CREATE TABLE measurement_y2007m12 PARTITION OF measurement - FOR VALUES FROM ('2007-12-01') TO ('2008-01-01'); -CREATE TABLE measurement_y2008m01 PARTITION OF measurement - FOR VALUES FROM ('2008-01-01') TO ('2008-02-01'); -</programlisting> + <para> + Another disadvantage of the rule approach is that there is no simple + way to force an error if the set of rules doesn't cover the insertion + date; the data will silently go into the master table instead. </para> </listitem> <listitem> <para> - Create indexes on the key columns just like in case of inheritance - partitions. + Ensure that the <xref linkend="guc-constraint-exclusion"> + configuration parameter is not disabled in + <filename>postgresql.conf</>. + If it is, queries will not be optimized as desired. </para> </listitem> </orderedlist> - - <note> - <para> - To implement sub-partitioning, specify the - <literal>PARTITION BY</literal> clause in the commands used to create - individual partitions, for example: - -<programlisting> -CREATE TABLE measurement_y2006m02 PARTITION OF measurement - FOR VALUES FROM ('2006-02-01') TO ('2006-03-01') - PARTITION BY RANGE (peaktemp); -</programlisting> - - After creating partitions of <structname>measurement_y2006m02</>, any - data inserted into <structname>measurement</> that is mapped to - <structname>measurement_y2006m02</> will be further redirected to one - of its partitions based on the <structfield>peaktemp</> column. - Partition key specified may overlap with the parent's partition key, - although care must be taken when specifying the bounds of sub-partitions - such that the accepted set of data constitutes a subset of what a - partition's own bounds allows; the system does not try to check if - that's really the case. - </para> - </note> </para> <para> - As we can see, a complex partitioning scheme could require a - substantial amount of DDL, although significantly less when using - partitioned tables. In the above example we would be creating a new - partition each month, so it might be wise to write a script that - generates the required DDL automatically. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-managing-partitions"> - <title>Managing Partitions</title> - - <para> - Normally the set of partitions established when initially - defining the table are not intended to remain static. It is - common to want to remove old partitions of data and periodically - add new partitions for new data. One of the most important - advantages of partitioning is precisely that it allows this - otherwise painful task to be executed nearly instantaneously by - manipulating the partition structure, rather than physically moving large - amounts of data around. - </para> - - <para> - Both the inheritance-based and partitioned table methods allow this to - be done, although the latter requires taking an <literal>ACCESS EXCLUSIVE</literal> - lock on the master table for various commands mentioned below. - </para> + As we can see, a complex partitioning scheme could require a + substantial amount of DDL. In the above example we would be creating + a new partition each month, so it might be wise to write a script that + generates the required DDL automatically. + </para> + </sect3> - <para> - The simplest option for removing old data is simply to drop the partition - that is no longer necessary, which works using both methods of - partitioning: + <sect3 id="ddl-partitioning-inheritance-maintenance"> + <title>Partition Maintenance</title> + <para> + To remove old data quickly, simply to drop the partition that is no + longer necessary: <programlisting> DROP TABLE measurement_y2006m02; </programlisting> - This can very quickly delete millions of records because it doesn't have - to individually delete every record. - </para> + </para> <para> - Another option that is often preferable is to remove the partition from - the partitioned table but retain access to it as a table in its own - right: -<programlisting> -ALTER TABLE measurement_y2006m02 NO INHERIT measurement; -</programlisting> - - When using a partitioned table: + To remove the partition from the partitioned table but retain access to + it as a table in its own right: <programlisting> -ALTER TABLE measurement DETACH PARTITION measurement_y2006m02; +ALTER TABLE measurement_y2006m02 NO INHERIT measurement; </programlisting> - - This allows further operations to be performed on the data before - it is dropped. For example, this is often a useful time to back up - the data using <command>COPY</>, <application>pg_dump</>, or - similar tools. It might also be a useful time to aggregate data - into smaller formats, perform other data manipulations, or run - reports. </para> <para> - Similarly we can add a new partition to handle new data. We can create an - empty partition in the partitioned table just as the original partitions - were created above: + To add a new partition to handle new data, create an empty partition just + as the original partitions were created above: <programlisting> CREATE TABLE measurement_y2008m02 ( @@ -3577,52 +3682,80 @@ CREATE TABLE measurement_y2008m02 ( ) INHERITS (measurement); </programlisting> - When using a partitioned table: - -<programlisting> -CREATE TABLE measurement_y2008m02 PARTITION OF measurement - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01'); -</programlisting> - - As an alternative, it is sometimes more convenient to create the - new table outside the partition structure, and make it a proper - partition later. This allows the data to be loaded, checked, and - transformed prior to it appearing in the partitioned table: + Alternatively, one may created the new table outside the partition + structure, and make it a partition after data is loaded, checked, + and transformed. <programlisting> CREATE TABLE measurement_y2008m02 (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS); + ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02 CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' ); + \copy measurement_y2008m02 from 'measurement_y2008m02' -- possibly some other data preparation work + ALTER TABLE measurement_y2008m02 INHERIT measurement; </programlisting> + </para> + </sect3> + + <sect3 id="ddl-partitioning-inheritance-caveats"> + <title>Caveats</title> + + <para> + The following caveats apply to partitioned tables implemented using + inheritance: + <itemizedlist> + <listitem> + <para> + There is no automatic way to verify that all of the + <literal>CHECK</literal> constraints are mutually + exclusive. It is safer to create code that generates + partitions and creates and/or modifies associated objects than + to write each by hand. + </para> + </listitem> - The last of the above commands when using a partitioned table would be: + <listitem> + <para> + The schemes shown here assume that the partition key column(s) + of a row never change, or at least do not change enough to require + it to move to another partition. An <command>UPDATE</> that attempts + to do that will fail because of the <literal>CHECK</> constraints. + If you need to handle such cases, you can put suitable update triggers + on the partition tables, but it makes management of the structure + much more complicated. + </para> + </listitem> + <listitem> + <para> + If you are using manual <command>VACUUM</command> or + <command>ANALYZE</command> commands, don't forget that + you need to run them on each partition individually. A command like: <programlisting> -ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 - FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' ); +ANALYZE measurement; </programlisting> - </para> + will only process the master table. + </para> + </listitem> - <tip> + <listitem> <para> - Before running the <command>ATTACH PARTITION</> command, it is - recommended to create a <literal>CHECK</> constraint on the table to - be attached describing the desired partition constraint. Using the - same, system is able to skip the scan to validate the implicit - partition constraint. Without such a constraint, the table will be - scanned to validate the partition constraint, while holding an - <literal>ACCESS EXCLUSIVE</literal> lock on the parent table. - One may want to drop the constraint after <command>ATTACH PARTITION</> - is finished, because it is no longer necessary. + <command>INSERT</command> statements with <literal>ON CONFLICT</> + clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> + action is only taken in case of unique violations on the specified + target relation, not its child relations. </para> - </tip> - </sect2> + </listitem> + </itemizedlist> + </para> + </sect3> + </sect2> - <sect2 id="ddl-partitioning-constraint-exclusion"> + <sect2 id="ddl-partitioning-constraint-exclusion"> <title>Partitioning and Constraint Exclusion</title> <indexterm> @@ -3632,7 +3765,8 @@ ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02 <para> <firstterm>Constraint exclusion</> is a query optimization technique that improves performance for partitioned tables defined in the - fashion described above. As an example: + fashion described above (both declarative partitioned tables and those + implemented using inheritance). As an example: <programlisting> SET constraint_exclusion = on; @@ -3715,153 +3849,6 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; are unlikely to benefit. </para> - <note> - <para> - Currently, constraint exclusion is also used for partitioned tables. - However, we did not create any <literal>CHECK</literal> constraints - for individual partitions as seen above. In this case, the optimizer - uses internally generated constraint for every partition. - </para> - </note> - - </sect2> - - <sect2 id="ddl-partitioning-alternatives"> - <title>Alternative Partitioning Methods</title> - - <para> - A different approach to redirecting inserts into the appropriate - partition table is to set up rules, instead of a trigger, on the - master table (unless it is a partitioned table). For example: - -<programlisting> -CREATE RULE measurement_insert_y2006m02 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' ) -DO INSTEAD - INSERT INTO measurement_y2006m02 VALUES (NEW.*); -... -CREATE RULE measurement_insert_y2008m01 AS -ON INSERT TO measurement WHERE - ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' ) -DO INSTEAD - INSERT INTO measurement_y2008m01 VALUES (NEW.*); -</programlisting> - - A rule has significantly more overhead than a trigger, but the overhead - is paid once per query rather than once per row, so this method might be - advantageous for bulk-insert situations. In most cases, however, the - trigger method will offer better performance. - </para> - - <para> - Be aware that <command>COPY</> ignores rules. If you want to - use <command>COPY</> to insert data, you'll need to copy into the correct - partition table rather than into the master. <command>COPY</> does fire - triggers, so you can use it normally if you use the trigger approach. - </para> - - <para> - Another disadvantage of the rule approach is that there is no simple - way to force an error if the set of rules doesn't cover the insertion - date; the data will silently go into the master table instead. - </para> - - <para> - Partitioning can also be arranged using a <literal>UNION ALL</literal> - view, instead of table inheritance. For example, - -<programlisting> -CREATE VIEW measurement AS - SELECT * FROM measurement_y2006m02 -UNION ALL SELECT * FROM measurement_y2006m03 -... -UNION ALL SELECT * FROM measurement_y2007m11 -UNION ALL SELECT * FROM measurement_y2007m12 -UNION ALL SELECT * FROM measurement_y2008m01; -</programlisting> - - However, the need to recreate the view adds an extra step to adding and - dropping individual partitions of the data set. In practice this - method has little to recommend it compared to using inheritance. - </para> - - </sect2> - - <sect2 id="ddl-partitioning-caveats"> - <title>Caveats</title> - - <para> - The following caveats apply to using inheritance to implement partitioning: - <itemizedlist> - <listitem> - <para> - There is no automatic way to verify that all of the - <literal>CHECK</literal> constraints are mutually - exclusive. It is safer to create code that generates - partitions and creates and/or modifies associated objects than - to write each by hand. - </para> - </listitem> - - <listitem> - <para> - The schemes shown here assume that the partition key column(s) - of a row never change, or at least do not change enough to require - it to move to another partition. An <command>UPDATE</> that attempts - to do that will fail because of the <literal>CHECK</> constraints. - If you need to handle such cases, you can put suitable update triggers - on the partition tables, but it makes management of the structure - much more complicated. - </para> - </listitem> - - <listitem> - <para> - If you are using manual <command>VACUUM</command> or - <command>ANALYZE</command> commands, don't forget that - you need to run them on each partition individually. A command like: -<programlisting> -ANALYZE measurement; -</programlisting> - will only process the master table. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clauses are unlikely to work as expected, as the <literal>ON CONFLICT</> - action is only taken in case of unique violations on the specified - target relation, not its child relations. - </para> - </listitem> - </itemizedlist> - </para> - - <para> - The following caveats apply to partitioned tables created with the - explicit syntax: - <itemizedlist> - <listitem> - <para> - An <command>UPDATE</> that causes a row to move from one partition to - another fails, because the new value of the row fails to satisfy the - implicit partition constraint of the original partition. This might - change in future releases. - </para> - </listitem> - - <listitem> - <para> - <command>INSERT</command> statements with <literal>ON CONFLICT</> - clause are currently not allowed on partitioned tables. - </para> - </listitem> - - </itemizedlist> - </para> - <para> The following caveats apply to constraint exclusion, which is currently used by both inheritance and partitioned tables: @@ -3901,10 +3888,78 @@ ANALYZE measurement; don't try to use many thousands of partitions. </para> </listitem> - </itemizedlist> </para> </sect2> + + <sect2 id="ddl-partitioning-alternatives"> + <title>Alternative Partitioning Methods</title> + + <sect3 id="ddl-partitioning-alternatives-union-all"> + <title>Using UNION ALL view</title> + <para> + Partitioning can also be arranged using a <literal>UNION ALL</literal> + view, instead of table inheritance. For example, + +<programlisting> +CREATE VIEW measurement AS + SELECT * FROM measurement_y2006m02 +UNION ALL SELECT * FROM measurement_y2006m03 +... +UNION ALL SELECT * FROM measurement_y2007m11 +UNION ALL SELECT * FROM measurement_y2007m12 +UNION ALL SELECT * FROM measurement_y2008m01; +</programlisting> + + However, the need to recreate the view adds an extra step to adding and + dropping individual partitions of the data set. In practice this + method has little to recommend it compared to using inheritance. + </para> + </sect3> + + <sect3 id="ddl-partitioning-alternatives-brin-index"> + <title>Accessing Tables Using BRIN Index</title> + <para> + <acronym>BRIN</acronym>, which stands for Block Range Index, is + designed for handling very large tables in which certain columns + have some natural physical location within the table. For example, + in the <structname>measurement</structname> table, the entries for + earlier times (<structfield>logdate</structfield> column) will appear + earlier in the table most of the time. A table storing a ZIP code + column might have all codes for a city grouped together naturally. + </para> + + <para> + In case of <structname>measurement</structname> table, one may consider + adding a minmax <acronym>BRIN</acronym> index on the + <structfield>logdate</structfield> column. + +<programlisting> +CREATE INDEX ON measurement USING brin (logdate date_minmax_ops); +</programlisting> + + In this case, specifying <literal>date_minmax_ops</literal> is not + necessary; it is shown for clarity. + </para> + + <para> + <acronym>BRIN</acronym> indexes leverage this locality of data and + store summary information for a range of consecutive pages and keep + it updated as the data is added or removed. Because a + <acronym>BRIN</acronym> index is very small, scanning the index adds + adds little overhead compared to a sequential scan, but may avoid + scanning large parts of the table that are known not to contain + matching tuples. That is often why table partitioning is used. Thus, + <acronym>BRIN</acronym> indexes provide a subset of benefits that + parttioning provides with much less upfront setup. + </para> + + <para> + See <xref linkend="brin"> for more details. + </para> + </sect3> + + </sect2> </sect1> <sect1 id="ddl-foreign-data"> -- 2.11.0 --------------434C61868FD2E7C005F82712 Content-Type: text/x-diff; name="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Add-a-note-about-DROP-NOT-NULL-and-partitions.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v7 1/3] Add stats tests related to rewrite @ 2025-10-27 14:54 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 50+ messages in thread From: Bertrand Drouvot @ 2025-10-27 14:54 UTC (permalink / raw) While there are existing rewrite tests, the stats behavior during rewrites doesn't have a good coverage. This patch adds some tests to record some stats after different rewrite scenarios. That is useful for a following patch where relation statistics will be keyed by relfilenode. We'll be able to test that the stats are still the ones we expect after rewrites. --- src/test/regress/expected/stats.out | 321 ++++ src/test/regress/expected/stats_1.out | 2255 +++++++++++++++++++++++++ src/test/regress/sql/stats.sql | 186 ++ 3 files changed, 2762 insertions(+) 91.8% src/test/regress/expected/ 8.1% src/test/regress/sql/ diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out index 67e1860e984..06487e367d8 100644 --- a/src/test/regress/expected/stats.out +++ b/src/test/regress/expected/stats.out @@ -1910,4 +1910,325 @@ SELECT * FROM check_estimated_rows('SELECT * FROM table_fillfactor'); (1 row) DROP TABLE table_fillfactor; +-- Test some rewrites +CREATE TABLE test_2pc_timestamp (a int) WITH (autovacuum_enabled = false); +VACUUM ANALYZE test_2pc_timestamp; +SELECT last_analyze AS last_vacuum_analyze FROM pg_stat_all_tables WHERE relname = 'test_2pc_timestamp' \gset +BEGIN; +ALTER TABLE test_2pc_timestamp ALTER COLUMN a TYPE int; +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT last_analyze = :'last_vacuum_analyze'::timestamptz FROM pg_stat_all_tables WHERE relname = 'test_2pc_timestamp'; + ?column? +---------- + t +(1 row) + +DROP TABLE test_2pc_timestamp; +CREATE TABLE test_2pc_rewrite_alone (a int); +INSERT INTO test_2pc_rewrite_alone VALUES (1); +BEGIN; +ALTER TABLE test_2pc_rewrite_alone ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_rewrite_alone'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_2pc_rewrite_alone; +CREATE TABLE test_2pc (a int); +INSERT INTO test_2pc VALUES (1); +BEGIN; +INSERT INTO test_2pc VALUES (1); +INSERT INTO test_2pc VALUES (2); +INSERT INTO test_2pc VALUES (3); +ALTER TABLE test_2pc ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 4 | 4 | 0 +(1 row) + +DROP TABLE test_2pc; +CREATE TABLE test_2pc_multi (a int); +INSERT INTO test_2pc_multi VALUES (1); +BEGIN; +INSERT INTO test_2pc_multi VALUES (1); +INSERT INTO test_2pc_multi VALUES (2); +ALTER TABLE test_2pc_multi ALTER COLUMN a TYPE bigint; +INSERT INTO test_2pc_multi VALUES (3); +INSERT INTO test_2pc_multi VALUES (4); +ALTER TABLE test_2pc_multi ALTER COLUMN a TYPE int; +INSERT INTO test_2pc_multi VALUES (5); +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_multi'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 6 | 6 | 0 +(1 row) + +DROP TABLE test_2pc_multi; +CREATE TABLE test_2pc_rewrite_alone_abort (a int); +INSERT INTO test_2pc_rewrite_alone_abort VALUES (1); +BEGIN; +ALTER TABLE test_2pc_rewrite_alone_abort ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +ROLLBACK PREPARED 'test'; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_rewrite_alone_abort'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_2pc_rewrite_alone_abort; +CREATE TABLE test_2pc_abort (a int); +INSERT INTO test_2pc_abort VALUES (1); +BEGIN; +INSERT INTO test_2pc_abort VALUES (1); +INSERT INTO test_2pc_abort VALUES (2); +ALTER TABLE test_2pc_abort ALTER COLUMN a TYPE bigint; +INSERT INTO test_2pc_abort VALUES (3); +PREPARE TRANSACTION 'test'; +ROLLBACK PREPARED 'test'; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_abort'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 4 | 1 | 3 +(1 row) + +DROP TABLE test_2pc_abort; +CREATE TABLE test_2pc_savepoint (a int); +INSERT INTO test_2pc_savepoint VALUES (1); +BEGIN; +SAVEPOINT a; +INSERT INTO test_2pc_savepoint VALUES (1); +INSERT INTO test_2pc_savepoint VALUES (2); +ALTER TABLE test_2pc_savepoint ALTER COLUMN a TYPE bigint; +SAVEPOINT b; +INSERT INTO test_2pc_savepoint VALUES (3); +ALTER TABLE test_2pc_savepoint ALTER COLUMN a TYPE int; +SAVEPOINT c; +INSERT INTO test_2pc_savepoint VALUES (4); +INSERT INTO test_2pc_savepoint VALUES (5); +ROLLBACK TO SAVEPOINT b; +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_savepoint'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 6 | 3 | 3 +(1 row) + +DROP TABLE test_2pc_savepoint; +-- Rewrite without 2PC +CREATE TABLE test_timestamp (a int) WITH (autovacuum_enabled = false); +VACUUM ANALYZE test_timestamp; +SELECT last_analyze AS last_vacuum_analyze FROM pg_stat_all_tables WHERE relname = 'test_timestamp' \gset +ALTER TABLE test_timestamp ALTER COLUMN a TYPE bigint; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT last_analyze = :'last_vacuum_analyze'::timestamptz FROM pg_stat_all_tables WHERE relname = 'test_timestamp'; + ?column? +---------- + t +(1 row) + +DROP TABLE test_timestamp; +CREATE TABLE test_alone (a int); +INSERT INTO test_alone VALUES (1); +BEGIN; +ALTER TABLE test_alone ALTER COLUMN a TYPE bigint; +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_alone'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_alone; +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +BEGIN; +INSERT INTO test VALUES (1); +INSERT INTO test VALUES (2); +INSERT INTO test VALUES (3); +ALTER TABLE test ALTER COLUMN a TYPE bigint; +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 4 | 4 | 0 +(1 row) + +DROP TABLE test; +CREATE TABLE test_multi (a int); +INSERT INTO test_multi VALUES (1); +BEGIN; +INSERT INTO test_multi VALUES (1); +INSERT INTO test_multi VALUES (2); +ALTER TABLE test_multi ALTER COLUMN a TYPE bigint; +INSERT INTO test_multi VALUES (3); +INSERT INTO test_multi VALUES (4); +ALTER TABLE test_multi ALTER COLUMN a TYPE int; +INSERT INTO test_multi VALUES (5); +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_multi'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 6 | 6 | 0 +(1 row) + +DROP TABLE test_multi; +CREATE TABLE test_rewrite_alone_abort (a int); +INSERT INTO test_rewrite_alone_abort VALUES (1); +BEGIN; +ALTER TABLE test_rewrite_alone_abort ALTER COLUMN a TYPE bigint; +ROLLBACK; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_rewrite_alone_abort'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_rewrite_alone_abort; +CREATE TABLE test_abort (a int); +INSERT INTO test_abort VALUES (1); +BEGIN; +INSERT INTO test_abort VALUES (1); +INSERT INTO test_abort VALUES (2); +ALTER TABLE test_abort ALTER COLUMN a TYPE bigint; +INSERT INTO test_abort VALUES (3); +ROLLBACK; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_abort'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 4 | 1 | 3 +(1 row) + +DROP TABLE test_abort; +CREATE TABLE test_savepoint (a int); +INSERT INTO test_savepoint VALUES (1); +BEGIN; +SAVEPOINT a; +INSERT INTO test_savepoint VALUES (1); +INSERT INTO test_savepoint VALUES (2); +ALTER TABLE test_savepoint ALTER COLUMN a TYPE bigint; +SAVEPOINT b; +INSERT INTO test_savepoint VALUES (3); +ALTER TABLE test_savepoint ALTER COLUMN a TYPE int; +SAVEPOINT c; +INSERT INTO test_savepoint VALUES (4); +INSERT INTO test_savepoint VALUES (5); +ROLLBACK TO SAVEPOINT b; +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_savepoint'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 6 | 3 | 3 +(1 row) + +DROP TABLE test_savepoint; +CREATE TABLE test_tbs (a int); +INSERT INTO test_tbs VALUES (1); +ALTER TABLE test_tbs SET TABLESPACE pg_default; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_tbs'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_tbs; -- End of Stats Test diff --git a/src/test/regress/expected/stats_1.out b/src/test/regress/expected/stats_1.out new file mode 100644 index 00000000000..629e71fce0d --- /dev/null +++ b/src/test/regress/expected/stats_1.out @@ -0,0 +1,2255 @@ +-- +-- Test cumulative stats system +-- +-- Must be run after tenk2 has been created (by create_table), +-- populated (by create_misc) and indexed (by create_index). +-- +-- conditio sine qua non +SHOW track_counts; -- must be on + track_counts +-------------- + on +(1 row) + +-- List of backend types, contexts and objects tracked in pg_stat_io. +\a +SELECT backend_type, object, context FROM pg_stat_io + ORDER BY backend_type COLLATE "C", object COLLATE "C", context COLLATE "C"; +backend_type|object|context +autovacuum launcher|relation|bulkread +autovacuum launcher|relation|init +autovacuum launcher|relation|normal +autovacuum launcher|wal|init +autovacuum launcher|wal|normal +autovacuum worker|relation|bulkread +autovacuum worker|relation|init +autovacuum worker|relation|normal +autovacuum worker|relation|vacuum +autovacuum worker|wal|init +autovacuum worker|wal|normal +background worker|relation|bulkread +background worker|relation|bulkwrite +background worker|relation|init +background worker|relation|normal +background worker|relation|vacuum +background worker|temp relation|normal +background worker|wal|init +background worker|wal|normal +background writer|relation|init +background writer|relation|normal +background writer|wal|init +background writer|wal|normal +checkpointer|relation|init +checkpointer|relation|normal +checkpointer|wal|init +checkpointer|wal|normal +client backend|relation|bulkread +client backend|relation|bulkwrite +client backend|relation|init +client backend|relation|normal +client backend|relation|vacuum +client backend|temp relation|normal +client backend|wal|init +client backend|wal|normal +io worker|relation|bulkread +io worker|relation|bulkwrite +io worker|relation|init +io worker|relation|normal +io worker|relation|vacuum +io worker|temp relation|normal +io worker|wal|init +io worker|wal|normal +slotsync worker|relation|bulkread +slotsync worker|relation|bulkwrite +slotsync worker|relation|init +slotsync worker|relation|normal +slotsync worker|relation|vacuum +slotsync worker|temp relation|normal +slotsync worker|wal|init +slotsync worker|wal|normal +standalone backend|relation|bulkread +standalone backend|relation|bulkwrite +standalone backend|relation|init +standalone backend|relation|normal +standalone backend|relation|vacuum +standalone backend|wal|init +standalone backend|wal|normal +startup|relation|bulkread +startup|relation|bulkwrite +startup|relation|init +startup|relation|normal +startup|relation|vacuum +startup|wal|init +startup|wal|normal +walreceiver|wal|init +walreceiver|wal|normal +walsender|relation|bulkread +walsender|relation|bulkwrite +walsender|relation|init +walsender|relation|normal +walsender|relation|vacuum +walsender|temp relation|normal +walsender|wal|init +walsender|wal|normal +walsummarizer|wal|init +walsummarizer|wal|normal +walwriter|wal|init +walwriter|wal|normal +(79 rows) +\a +-- ensure that both seqscan and indexscan plans are allowed +SET enable_seqscan TO on; +SET enable_indexscan TO on; +-- for the moment, we don't want index-only scans here +SET enable_indexonlyscan TO off; +-- not enabled by default, but we want to test it... +SET track_functions TO 'all'; +-- record dboid for later use +SELECT oid AS dboid from pg_database where datname = current_database() \gset +-- save counters +BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; +CREATE TABLE prevstats AS +SELECT t.seq_scan, t.seq_tup_read, t.idx_scan, t.idx_tup_fetch, + (b.heap_blks_read + b.heap_blks_hit) AS heap_blks, + (b.idx_blks_read + b.idx_blks_hit) AS idx_blks, + pg_stat_get_snapshot_timestamp() as snap_ts + FROM pg_catalog.pg_stat_user_tables AS t, + pg_catalog.pg_statio_user_tables AS b + WHERE t.relname='tenk2' AND b.relname='tenk2'; +COMMIT; +-- test effects of TRUNCATE on n_live_tup/n_dead_tup counters +CREATE TABLE trunc_stats_test(id serial); +CREATE TABLE trunc_stats_test1(id serial, stuff text); +CREATE TABLE trunc_stats_test2(id serial); +CREATE TABLE trunc_stats_test3(id serial, stuff text); +CREATE TABLE trunc_stats_test4(id serial); +-- check that n_live_tup is reset to 0 after truncate +INSERT INTO trunc_stats_test DEFAULT VALUES; +INSERT INTO trunc_stats_test DEFAULT VALUES; +INSERT INTO trunc_stats_test DEFAULT VALUES; +TRUNCATE trunc_stats_test; +-- test involving a truncate in a transaction; 4 ins but only 1 live +INSERT INTO trunc_stats_test1 DEFAULT VALUES; +INSERT INTO trunc_stats_test1 DEFAULT VALUES; +INSERT INTO trunc_stats_test1 DEFAULT VALUES; +UPDATE trunc_stats_test1 SET id = id + 10 WHERE id IN (1, 2); +DELETE FROM trunc_stats_test1 WHERE id = 3; +BEGIN; +UPDATE trunc_stats_test1 SET id = id + 100; +TRUNCATE trunc_stats_test1; +INSERT INTO trunc_stats_test1 DEFAULT VALUES; +COMMIT; +-- use a savepoint: 1 insert, 1 live +BEGIN; +INSERT INTO trunc_stats_test2 DEFAULT VALUES; +INSERT INTO trunc_stats_test2 DEFAULT VALUES; +SAVEPOINT p1; +INSERT INTO trunc_stats_test2 DEFAULT VALUES; +TRUNCATE trunc_stats_test2; +INSERT INTO trunc_stats_test2 DEFAULT VALUES; +RELEASE SAVEPOINT p1; +COMMIT; +-- rollback a savepoint: this should count 4 inserts and have 2 +-- live tuples after commit (and 2 dead ones due to aborted subxact) +BEGIN; +INSERT INTO trunc_stats_test3 DEFAULT VALUES; +INSERT INTO trunc_stats_test3 DEFAULT VALUES; +SAVEPOINT p1; +INSERT INTO trunc_stats_test3 DEFAULT VALUES; +INSERT INTO trunc_stats_test3 DEFAULT VALUES; +TRUNCATE trunc_stats_test3; +INSERT INTO trunc_stats_test3 DEFAULT VALUES; +ROLLBACK TO SAVEPOINT p1; +COMMIT; +-- rollback a truncate: this should count 2 inserts and produce 2 dead tuples +BEGIN; +INSERT INTO trunc_stats_test4 DEFAULT VALUES; +INSERT INTO trunc_stats_test4 DEFAULT VALUES; +TRUNCATE trunc_stats_test4; +INSERT INTO trunc_stats_test4 DEFAULT VALUES; +ROLLBACK; +-- do a seqscan +SELECT count(*) FROM tenk2; + count +------- + 10000 +(1 row) + +-- do an indexscan +-- make sure it is not a bitmap scan, which might skip fetching heap tuples +SET enable_bitmapscan TO off; +SELECT count(*) FROM tenk2 WHERE unique1 = 1; + count +------- + 1 +(1 row) + +RESET enable_bitmapscan; +-- ensure pending stats are flushed +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +-- check effects +BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; +SELECT relname, n_tup_ins, n_tup_upd, n_tup_del, n_live_tup, n_dead_tup + FROM pg_stat_user_tables + WHERE relname like 'trunc_stats_test%' order by relname; + relname | n_tup_ins | n_tup_upd | n_tup_del | n_live_tup | n_dead_tup +-------------------+-----------+-----------+-----------+------------+------------ + trunc_stats_test | 3 | 0 | 0 | 0 | 0 + trunc_stats_test1 | 4 | 2 | 1 | 1 | 0 + trunc_stats_test2 | 1 | 0 | 0 | 1 | 0 + trunc_stats_test3 | 4 | 0 | 0 | 2 | 2 + trunc_stats_test4 | 2 | 0 | 0 | 0 | 2 +(5 rows) + +SELECT st.seq_scan >= pr.seq_scan + 1, + st.seq_tup_read >= pr.seq_tup_read + cl.reltuples, + st.idx_scan >= pr.idx_scan + 1, + st.idx_tup_fetch >= pr.idx_tup_fetch + 1 + FROM pg_stat_user_tables AS st, pg_class AS cl, prevstats AS pr + WHERE st.relname='tenk2' AND cl.relname='tenk2'; + ?column? | ?column? | ?column? | ?column? +----------+----------+----------+---------- + t | t | t | t +(1 row) + +SELECT st.heap_blks_read + st.heap_blks_hit >= pr.heap_blks + cl.relpages, + st.idx_blks_read + st.idx_blks_hit >= pr.idx_blks + 1 + FROM pg_statio_user_tables AS st, pg_class AS cl, prevstats AS pr + WHERE st.relname='tenk2' AND cl.relname='tenk2'; + ?column? | ?column? +----------+---------- + t | t +(1 row) + +SELECT pr.snap_ts < pg_stat_get_snapshot_timestamp() as snapshot_newer +FROM prevstats AS pr; + snapshot_newer +---------------- + t +(1 row) + +COMMIT; +---- +-- Basic tests for track_functions +--- +CREATE FUNCTION stats_test_func1() RETURNS VOID LANGUAGE plpgsql AS $$BEGIN END;$$; +SELECT 'stats_test_func1()'::regprocedure::oid AS stats_test_func1_oid \gset +CREATE FUNCTION stats_test_func2() RETURNS VOID LANGUAGE plpgsql AS $$BEGIN END;$$; +SELECT 'stats_test_func2()'::regprocedure::oid AS stats_test_func2_oid \gset +-- test that stats are accumulated +BEGIN; +SET LOCAL stats_fetch_consistency = none; +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); + pg_stat_get_xact_function_calls +--------------------------------- + +(1 row) + +SELECT stats_test_func1(); + stats_test_func1 +------------------ + +(1 row) + +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); + pg_stat_get_xact_function_calls +--------------------------------- + 1 +(1 row) + +SELECT stats_test_func1(); + stats_test_func1 +------------------ + +(1 row) + +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); + pg_stat_get_xact_function_calls +--------------------------------- + 2 +(1 row) + +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + 0 +(1 row) + +COMMIT; +-- Verify that function stats are not transactional +-- rolled back savepoint in committing transaction +BEGIN; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +SAVEPOINT foo; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +ROLLBACK TO SAVEPOINT foo; +SELECT pg_stat_get_xact_function_calls(:stats_test_func2_oid); + pg_stat_get_xact_function_calls +--------------------------------- + 2 +(1 row) + +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +COMMIT; +-- rolled back transaction +BEGIN; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +ROLLBACK; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +-- check collected stats +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +------------------+------- + stats_test_func1 | 2 +(1 row) + +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func2_oid; + funcname | calls +------------------+------- + stats_test_func2 | 4 +(1 row) + +-- check that a rolled back drop function stats leaves stats alive +BEGIN; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +------------------+------- + stats_test_func1 | 2 +(1 row) + +DROP FUNCTION stats_test_func1(); +-- shouldn't be visible via view +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +----------+------- +(0 rows) + +-- but still via oid access +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + 2 +(1 row) + +ROLLBACK; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +------------------+------- + stats_test_func1 | 2 +(1 row) + +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + 2 +(1 row) + +-- check that function dropped in main transaction leaves no stats behind +BEGIN; +DROP FUNCTION stats_test_func1(); +COMMIT; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +----------+------- +(0 rows) + +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +-- check that function dropped in a subtransaction leaves no stats behind +BEGIN; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +SAVEPOINT a; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +SAVEPOINT b; +DROP FUNCTION stats_test_func2(); +COMMIT; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func2_oid; + funcname | calls +----------+------- +(0 rows) + +SELECT pg_stat_get_function_calls(:stats_test_func2_oid); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +-- Check that stats for relations are dropped. For that we need to access stats +-- by oid after the DROP TABLE. Save oids. +CREATE TABLE drop_stats_test(); +INSERT INTO drop_stats_test DEFAULT VALUES; +SELECT 'drop_stats_test'::regclass::oid AS drop_stats_test_oid \gset +CREATE TABLE drop_stats_test_xact(); +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +SELECT 'drop_stats_test_xact'::regclass::oid AS drop_stats_test_xact_oid \gset +CREATE TABLE drop_stats_test_subxact(); +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SELECT 'drop_stats_test_subxact'::regclass::oid AS drop_stats_test_subxact_oid \gset +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT pg_stat_get_live_tuples(:drop_stats_test_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +DROP TABLE drop_stats_test; +SELECT pg_stat_get_live_tuples(:drop_stats_test_oid); + pg_stat_get_live_tuples +------------------------- + 0 +(1 row) + +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 0 +(1 row) + +-- check that rollback protects against having stats dropped and that local +-- modifications don't pose a problem +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_tuples_inserted +----------------------------- + 1 +(1 row) + +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 0 +(1 row) + +BEGIN; +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 1 +(1 row) + +DROP TABLE drop_stats_test_xact; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 0 +(1 row) + +ROLLBACK; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_tuples_inserted +----------------------------- + 2 +(1 row) + +-- transactional drop +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_tuples_inserted +----------------------------- + 2 +(1 row) + +BEGIN; +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 1 +(1 row) + +DROP TABLE drop_stats_test_xact; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 0 +(1 row) + +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); + pg_stat_get_live_tuples +------------------------- + 0 +(1 row) + +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_tuples_inserted +----------------------------- + 0 +(1 row) + +-- savepoint rollback (2 levels) +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +BEGIN; +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SAVEPOINT sp1; +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_subxact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 2 +(1 row) + +SAVEPOINT sp2; +DROP TABLE drop_stats_test_subxact; +ROLLBACK TO SAVEPOINT sp2; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_subxact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 2 +(1 row) + +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 3 +(1 row) + +-- savepoint rolback (1 level) +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 3 +(1 row) + +BEGIN; +SAVEPOINT sp1; +DROP TABLE drop_stats_test_subxact; +SAVEPOINT sp2; +ROLLBACK TO SAVEPOINT sp1; +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 3 +(1 row) + +-- and now actually drop +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 3 +(1 row) + +BEGIN; +SAVEPOINT sp1; +DROP TABLE drop_stats_test_subxact; +SAVEPOINT sp2; +RELEASE SAVEPOINT sp1; +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 0 +(1 row) + +DROP TABLE trunc_stats_test, trunc_stats_test1, trunc_stats_test2, trunc_stats_test3, trunc_stats_test4; +DROP TABLE prevstats; +----- +-- Test that last_seq_scan, last_idx_scan are correctly maintained +-- +-- Perform test using a temporary table. That way autovacuum etc won't +-- interfere. To be able to check that timestamps increase, we sleep for 100ms +-- between tests, assuming that there aren't systems with a coarser timestamp +-- granularity. +----- +BEGIN; +CREATE TEMPORARY TABLE test_last_scan(idx_col int primary key, noidx_col int); +INSERT INTO test_last_scan(idx_col, noidx_col) VALUES(1, 1); +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT last_seq_scan, last_idx_scan FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass; + last_seq_scan | last_idx_scan +---------------+--------------- + | +(1 row) + +COMMIT; +SELECT stats_reset IS NOT NULL AS has_stats_reset + FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass; + has_stats_reset +----------------- + f +(1 row) + +SELECT pg_stat_reset_single_table_counters('test_last_scan'::regclass); + pg_stat_reset_single_table_counters +------------------------------------- + +(1 row) + +SELECT seq_scan, idx_scan, stats_reset IS NOT NULL AS has_stats_reset + FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass; + seq_scan | idx_scan | has_stats_reset +----------+----------+----------------- + 0 | 0 | t +(1 row) + +-- ensure we start out with exactly one index and sequential scan +BEGIN; +SET LOCAL enable_seqscan TO on; +SET LOCAL enable_indexscan TO on; +SET LOCAL enable_bitmapscan TO off; +EXPLAIN (COSTS off) SELECT count(*) FROM test_last_scan WHERE noidx_col = 1; + QUERY PLAN +---------------------------------- + Aggregate + -> Seq Scan on test_last_scan + Filter: (noidx_col = 1) +(3 rows) + +SELECT count(*) FROM test_last_scan WHERE noidx_col = 1; + count +------- + 1 +(1 row) + +SET LOCAL enable_seqscan TO off; +EXPLAIN (COSTS off) SELECT count(*) FROM test_last_scan WHERE idx_col = 1; + QUERY PLAN +-------------------------------------------------------------- + Aggregate + -> Index Scan using test_last_scan_pkey on test_last_scan + Index Cond: (idx_col = 1) +(3 rows) + +SELECT count(*) FROM test_last_scan WHERE idx_col = 1; + count +------- + 1 +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +COMMIT; +-- fetch timestamps from before the next test +SELECT last_seq_scan AS test_last_seq, last_idx_scan AS test_last_idx +FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass \gset +SELECT pg_sleep(0.1); -- assume a minimum timestamp granularity of 100ms + pg_sleep +---------- + +(1 row) + +-- cause one sequential scan +BEGIN; +SET LOCAL enable_seqscan TO on; +SET LOCAL enable_indexscan TO off; +SET LOCAL enable_bitmapscan TO off; +EXPLAIN (COSTS off) SELECT count(*) FROM test_last_scan WHERE noidx_col = 1; + QUERY PLAN +---------------------------------- + Aggregate + -> Seq Scan on test_last_scan + Filter: (noidx_col = 1) +(3 rows) + +SELECT count(*) FROM test_last_scan WHERE noidx_col = 1; + count +------- + 1 +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +COMMIT; +-- check that just sequential scan stats were incremented +SELECT seq_scan, :'test_last_seq' < last_seq_scan AS seq_ok, idx_scan, :'test_last_idx' = last_idx_scan AS idx_ok +FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass; + seq_scan | seq_ok | idx_scan | idx_ok +----------+--------+----------+-------- + 2 | t | 1 | t +(1 row) + +-- fetch timestamps from before the next test +SELECT last_seq_scan AS test_last_seq, last_idx_scan AS test_last_idx +FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass \gset +SELECT pg_sleep(0.1); + pg_sleep +---------- + +(1 row) + +-- cause one index scan +BEGIN; +SET LOCAL enable_seqscan TO off; +SET LOCAL enable_indexscan TO on; +SET LOCAL enable_bitmapscan TO off; +EXPLAIN (COSTS off) SELECT count(*) FROM test_last_scan WHERE idx_col = 1; + QUERY PLAN +-------------------------------------------------------------- + Aggregate + -> Index Scan using test_last_scan_pkey on test_last_scan + Index Cond: (idx_col = 1) +(3 rows) + +SELECT count(*) FROM test_last_scan WHERE idx_col = 1; + count +------- + 1 +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +COMMIT; +-- check that just index scan stats were incremented +SELECT seq_scan, :'test_last_seq' = last_seq_scan AS seq_ok, idx_scan, :'test_last_idx' < last_idx_scan AS idx_ok +FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass; + seq_scan | seq_ok | idx_scan | idx_ok +----------+--------+----------+-------- + 2 | t | 2 | t +(1 row) + +-- fetch timestamps from before the next test +SELECT last_seq_scan AS test_last_seq, last_idx_scan AS test_last_idx +FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass \gset +SELECT pg_sleep(0.1); + pg_sleep +---------- + +(1 row) + +-- cause one bitmap index scan +BEGIN; +SET LOCAL enable_seqscan TO off; +SET LOCAL enable_indexscan TO off; +SET LOCAL enable_bitmapscan TO on; +EXPLAIN (COSTS off) SELECT count(*) FROM test_last_scan WHERE idx_col = 1; + QUERY PLAN +------------------------------------------------------ + Aggregate + -> Bitmap Heap Scan on test_last_scan + Recheck Cond: (idx_col = 1) + -> Bitmap Index Scan on test_last_scan_pkey + Index Cond: (idx_col = 1) +(5 rows) + +SELECT count(*) FROM test_last_scan WHERE idx_col = 1; + count +------- + 1 +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +COMMIT; +-- check that just index scan stats were incremented +SELECT seq_scan, :'test_last_seq' = last_seq_scan AS seq_ok, idx_scan, :'test_last_idx' < last_idx_scan AS idx_ok +FROM pg_stat_all_tables WHERE relid = 'test_last_scan'::regclass; + seq_scan | seq_ok | idx_scan | idx_ok +----------+--------+----------+-------- + 2 | t | 3 | t +(1 row) + +-- check the stats in pg_stat_all_indexes +SELECT idx_scan, :'test_last_idx' < last_idx_scan AS idx_ok, + stats_reset IS NOT NULL AS has_stats_reset + FROM pg_stat_all_indexes WHERE indexrelid = 'test_last_scan_pkey'::regclass; + idx_scan | idx_ok | has_stats_reset +----------+--------+----------------- + 3 | t | f +(1 row) + +-- check that the stats in pg_stat_all_indexes are reset +SELECT pg_stat_reset_single_table_counters('test_last_scan_pkey'::regclass); + pg_stat_reset_single_table_counters +------------------------------------- + +(1 row) + +SELECT idx_scan, stats_reset IS NOT NULL AS has_stats_reset + FROM pg_stat_all_indexes WHERE indexrelid = 'test_last_scan_pkey'::regclass; + idx_scan | has_stats_reset +----------+----------------- + 0 | t +(1 row) + +----- +-- Test reset of some stats for shared table +----- +-- This updates the comment of the database currently in use in +-- pg_shdescription with a fake value, then sets it back to its +-- original value. +SELECT shobj_description(d.oid, 'pg_database') as description_before + FROM pg_database d WHERE datname = current_database() \gset +-- force some stats in pg_shdescription. +BEGIN; +SELECT current_database() as datname \gset +COMMENT ON DATABASE :"datname" IS 'This is a test comment'; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +COMMIT; +-- check that the stats are reset. +SELECT (n_tup_ins + n_tup_upd) > 0 AS has_data FROM pg_stat_all_tables + WHERE relid = 'pg_shdescription'::regclass; + has_data +---------- + t +(1 row) + +SELECT pg_stat_reset_single_table_counters('pg_shdescription'::regclass); + pg_stat_reset_single_table_counters +------------------------------------- + +(1 row) + +SELECT (n_tup_ins + n_tup_upd) > 0 AS has_data FROM pg_stat_all_tables + WHERE relid = 'pg_shdescription'::regclass; + has_data +---------- + f +(1 row) + +-- set back comment +\if :{?description_before} + COMMENT ON DATABASE :"datname" IS :'description_before'; +\else + COMMENT ON DATABASE :"datname" IS NULL; +\endif +----- +-- Test that various stats views are being properly populated +----- +-- Test that sessions is incremented when a new session is started in pg_stat_database +SELECT sessions AS db_stat_sessions FROM pg_stat_database WHERE datname = (SELECT current_database()) \gset +\c +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT sessions > :db_stat_sessions FROM pg_stat_database WHERE datname = (SELECT current_database()); + ?column? +---------- + t +(1 row) + +-- Test pg_stat_checkpointer checkpointer-related stats, together with pg_stat_wal +SELECT num_requested AS rqst_ckpts_before FROM pg_stat_checkpointer \gset +-- Test pg_stat_wal +SELECT wal_bytes AS wal_bytes_before FROM pg_stat_wal \gset +-- Test pg_stat_get_backend_wal() +SELECT wal_bytes AS backend_wal_bytes_before from pg_stat_get_backend_wal(pg_backend_pid()) \gset +-- Make a temp table so our temp schema exists +CREATE TEMP TABLE test_stats_temp AS SELECT 17; +DROP TABLE test_stats_temp; +-- Checkpoint twice: The checkpointer reports stats after reporting completion +-- of the checkpoint. But after a second checkpoint we'll see at least the +-- results of the first. +-- +-- While at it, test checkpoint options. Note that we don't test MODE SPREAD +-- because it would prolong the test. +CHECKPOINT (WRONG); +ERROR: unrecognized CHECKPOINT option "wrong" +LINE 1: CHECKPOINT (WRONG); + ^ +CHECKPOINT (MODE WRONG); +ERROR: unrecognized MODE option "wrong" +LINE 1: CHECKPOINT (MODE WRONG); + ^ +CHECKPOINT (MODE FAST, FLUSH_UNLOGGED FALSE); +CHECKPOINT (FLUSH_UNLOGGED); +SELECT num_requested > :rqst_ckpts_before FROM pg_stat_checkpointer; + ?column? +---------- + t +(1 row) + +SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal; + ?column? +---------- + t +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_backend_pid()); + ?column? +---------- + t +(1 row) + +-- Test pg_stat_get_backend_idset() and some allied functions. +-- In particular, verify that their notion of backend ID matches +-- our temp schema index. +SELECT (current_schemas(true))[1] = ('pg_temp_' || beid::text) AS match +FROM pg_stat_get_backend_idset() beid +WHERE pg_stat_get_backend_pid(beid) = pg_backend_pid(); + match +------- + t +(1 row) + +----- +-- Test that resetting stats works for reset timestamp +----- +-- Test that reset_slru with a specified SLRU works. +SELECT stats_reset AS slru_commit_ts_reset_ts FROM pg_stat_slru WHERE name = 'commit_timestamp' \gset +SELECT stats_reset AS slru_notify_reset_ts FROM pg_stat_slru WHERE name = 'notify' \gset +SELECT pg_stat_reset_slru('commit_timestamp'); + pg_stat_reset_slru +-------------------- + +(1 row) + +SELECT stats_reset > :'slru_commit_ts_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'commit_timestamp'; + ?column? +---------- + t +(1 row) + +SELECT stats_reset AS slru_commit_ts_reset_ts FROM pg_stat_slru WHERE name = 'commit_timestamp' \gset +-- Test that multiple SLRUs are reset when no specific SLRU provided to reset function +SELECT pg_stat_reset_slru(); + pg_stat_reset_slru +-------------------- + +(1 row) + +SELECT stats_reset > :'slru_commit_ts_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'commit_timestamp'; + ?column? +---------- + t +(1 row) + +SELECT stats_reset > :'slru_notify_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'notify'; + ?column? +---------- + t +(1 row) + +-- Test that reset_shared with archiver specified as the stats type works +SELECT stats_reset AS archiver_reset_ts FROM pg_stat_archiver \gset +SELECT pg_stat_reset_shared('archiver'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT stats_reset > :'archiver_reset_ts'::timestamptz FROM pg_stat_archiver; + ?column? +---------- + t +(1 row) + +-- Test that reset_shared with bgwriter specified as the stats type works +SELECT stats_reset AS bgwriter_reset_ts FROM pg_stat_bgwriter \gset +SELECT pg_stat_reset_shared('bgwriter'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT stats_reset > :'bgwriter_reset_ts'::timestamptz FROM pg_stat_bgwriter; + ?column? +---------- + t +(1 row) + +-- Test that reset_shared with checkpointer specified as the stats type works +SELECT stats_reset AS checkpointer_reset_ts FROM pg_stat_checkpointer \gset +SELECT pg_stat_reset_shared('checkpointer'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT stats_reset > :'checkpointer_reset_ts'::timestamptz FROM pg_stat_checkpointer; + ?column? +---------- + t +(1 row) + +-- Test that reset_shared with recovery_prefetch specified as the stats type works +SELECT stats_reset AS recovery_prefetch_reset_ts FROM pg_stat_recovery_prefetch \gset +SELECT pg_stat_reset_shared('recovery_prefetch'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT stats_reset > :'recovery_prefetch_reset_ts'::timestamptz FROM pg_stat_recovery_prefetch; + ?column? +---------- + t +(1 row) + +-- Test that reset_shared with slru specified as the stats type works +SELECT max(stats_reset) AS slru_reset_ts FROM pg_stat_slru \gset +SELECT pg_stat_reset_shared('slru'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT max(stats_reset) > :'slru_reset_ts'::timestamptz FROM pg_stat_slru; + ?column? +---------- + t +(1 row) + +-- Test that reset_shared with wal specified as the stats type works +SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset +SELECT pg_stat_reset_shared('wal'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal; + ?column? +---------- + t +(1 row) + +-- Test error case for reset_shared with unknown stats type +SELECT pg_stat_reset_shared('unknown'); +ERROR: unrecognized reset target: "unknown" +HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", or "wal". +-- Test that reset works for pg_stat_database +-- Since pg_stat_database stats_reset starts out as NULL, reset it once first so we have something to compare it to +SELECT pg_stat_reset(); + pg_stat_reset +--------------- + +(1 row) + +SELECT stats_reset AS db_reset_ts FROM pg_stat_database WHERE datname = (SELECT current_database()) \gset +SELECT pg_stat_reset(); + pg_stat_reset +--------------- + +(1 row) + +SELECT stats_reset > :'db_reset_ts'::timestamptz FROM pg_stat_database WHERE datname = (SELECT current_database()); + ?column? +---------- + t +(1 row) + +---- +-- pg_stat_get_snapshot_timestamp behavior +---- +BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; +-- no snapshot yet, return NULL +SELECT pg_stat_get_snapshot_timestamp(); + pg_stat_get_snapshot_timestamp +-------------------------------- + +(1 row) + +-- any attempt at accessing stats will build snapshot +SELECT pg_stat_get_function_calls(0); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +SELECT pg_stat_get_snapshot_timestamp() >= NOW(); + ?column? +---------- + t +(1 row) + +-- shows NULL again after clearing +SELECT pg_stat_clear_snapshot(); + pg_stat_clear_snapshot +------------------------ + +(1 row) + +SELECT pg_stat_get_snapshot_timestamp(); + pg_stat_get_snapshot_timestamp +-------------------------------- + +(1 row) + +COMMIT; +---- +-- Changing stats_fetch_consistency in a transaction. +---- +BEGIN; +-- Stats filled under the cache mode +SET LOCAL stats_fetch_consistency = cache; +SELECT pg_stat_get_function_calls(0); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +SELECT pg_stat_get_snapshot_timestamp() IS NOT NULL AS snapshot_ok; + snapshot_ok +------------- + f +(1 row) + +-- Success in accessing pre-existing snapshot data. +SET LOCAL stats_fetch_consistency = snapshot; +SELECT pg_stat_get_snapshot_timestamp() IS NOT NULL AS snapshot_ok; + snapshot_ok +------------- + f +(1 row) + +SELECT pg_stat_get_function_calls(0); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +SELECT pg_stat_get_snapshot_timestamp() IS NOT NULL AS snapshot_ok; + snapshot_ok +------------- + t +(1 row) + +-- Snapshot cleared. +SET LOCAL stats_fetch_consistency = none; +SELECT pg_stat_get_snapshot_timestamp() IS NOT NULL AS snapshot_ok; + snapshot_ok +------------- + f +(1 row) + +SELECT pg_stat_get_function_calls(0); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +SELECT pg_stat_get_snapshot_timestamp() IS NOT NULL AS snapshot_ok; + snapshot_ok +------------- + f +(1 row) + +ROLLBACK; +---- +-- pg_stat_have_stats behavior +---- +-- fixed-numbered stats exist +SELECT pg_stat_have_stats('bgwriter', 0, 0); + pg_stat_have_stats +-------------------- + t +(1 row) + +-- unknown stats kinds error out +SELECT pg_stat_have_stats('zaphod', 0, 0); +ERROR: invalid statistics kind: "zaphod" +-- db stats have objid 0 +SELECT pg_stat_have_stats('database', :dboid, 1); + pg_stat_have_stats +-------------------- + f +(1 row) + +SELECT pg_stat_have_stats('database', :dboid, 0); + pg_stat_have_stats +-------------------- + t +(1 row) + +-- pg_stat_have_stats returns true for committed index creation +CREATE table stats_test_tab1 as select generate_series(1,10) a; +CREATE index stats_test_idx1 on stats_test_tab1(a); +SELECT 'stats_test_idx1'::regclass::oid AS stats_test_idx1_oid \gset +SET enable_seqscan TO off; +select a from stats_test_tab1 where a = 3; + a +--- + 3 +(1 row) + +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + t +(1 row) + +-- pg_stat_have_stats returns false for dropped index with stats +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + t +(1 row) + +DROP index stats_test_idx1; +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + f +(1 row) + +-- pg_stat_have_stats returns false for rolled back index creation +BEGIN; +CREATE index stats_test_idx1 on stats_test_tab1(a); +SELECT 'stats_test_idx1'::regclass::oid AS stats_test_idx1_oid \gset +select a from stats_test_tab1 where a = 3; + a +--- + 3 +(1 row) + +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + t +(1 row) + +ROLLBACK; +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + f +(1 row) + +-- pg_stat_have_stats returns true for reindex CONCURRENTLY +CREATE index stats_test_idx1 on stats_test_tab1(a); +SELECT 'stats_test_idx1'::regclass::oid AS stats_test_idx1_oid \gset +select a from stats_test_tab1 where a = 3; + a +--- + 3 +(1 row) + +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + t +(1 row) + +REINDEX index CONCURRENTLY stats_test_idx1; +-- false for previous oid +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + f +(1 row) + +-- true for new oid +SELECT 'stats_test_idx1'::regclass::oid AS stats_test_idx1_oid \gset +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + t +(1 row) + +-- pg_stat_have_stats returns true for a rolled back drop index with stats +BEGIN; +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + t +(1 row) + +DROP index stats_test_idx1; +ROLLBACK; +SELECT pg_stat_have_stats('relation', :dboid, :stats_test_idx1_oid); + pg_stat_have_stats +-------------------- + t +(1 row) + +-- put enable_seqscan back to on +SET enable_seqscan TO on; +-- ensure that stats accessors handle NULL input correctly +SELECT pg_stat_get_replication_slot(NULL); + pg_stat_get_replication_slot +------------------------------ + +(1 row) + +SELECT pg_stat_get_subscription_stats(NULL); + pg_stat_get_subscription_stats +-------------------------------- + +(1 row) + +-- Test that the following operations are tracked in pg_stat_io and in +-- backend stats: +-- - reads of target blocks into shared buffers +-- - writes of shared buffers to permanent storage +-- - extends of relations using shared buffers +-- - fsyncs done to ensure the durability of data dirtying shared buffers +-- - shared buffer hits +-- - WAL writes and fsyncs in IOContext IOCONTEXT_NORMAL +-- There is no test for blocks evicted from shared buffers, because we cannot +-- be sure of the state of shared buffers at the point the test is run. +-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL +-- extends. +SELECT pid AS checkpointer_pid FROM pg_stat_activity + WHERE backend_type = 'checkpointer' \gset +SELECT sum(extends) AS io_sum_shared_before_extends + FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset +SELECT sum(extends) AS my_io_sum_shared_before_extends + FROM pg_stat_get_backend_io(pg_backend_pid()) + WHERE context = 'normal' AND object = 'relation' \gset +SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs + FROM pg_stat_io + WHERE object = 'relation' \gset io_sum_shared_before_ +SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs + FROM pg_stat_get_backend_io(pg_backend_pid()) + WHERE object = 'relation' \gset my_io_sum_shared_before_ +SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs + FROM pg_stat_io + WHERE context = 'normal' AND object = 'wal' \gset io_sum_wal_normal_before_ +CREATE TABLE test_io_shared(a int); +INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT sum(extends) AS io_sum_shared_after_extends + FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset +SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends; + ?column? +---------- + t +(1 row) + +SELECT sum(extends) AS my_io_sum_shared_after_extends + FROM pg_stat_get_backend_io(pg_backend_pid()) + WHERE context = 'normal' AND object = 'relation' \gset +SELECT :my_io_sum_shared_after_extends > :my_io_sum_shared_before_extends; + ?column? +---------- + t +(1 row) + +-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes +-- and fsyncs in the global stats (usually not for the backend). +-- See comment above for rationale for two explicit CHECKPOINTs. +CHECKPOINT; +CHECKPOINT; +SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs + FROM pg_stat_io + WHERE object = 'relation' \gset io_sum_shared_after_ +SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes; + ?column? +---------- + t +(1 row) + +SELECT current_setting('fsync') = 'off' + OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs; + ?column? +---------- + t +(1 row) + +SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs + FROM pg_stat_get_backend_io(pg_backend_pid()) + WHERE object = 'relation' \gset my_io_sum_shared_after_ +SELECT :my_io_sum_shared_after_writes >= :my_io_sum_shared_before_writes; + ?column? +---------- + t +(1 row) + +SELECT current_setting('fsync') = 'off' + OR :my_io_sum_shared_after_fsyncs >= :my_io_sum_shared_before_fsyncs; + ?column? +---------- + t +(1 row) + +SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs + FROM pg_stat_io + WHERE context = 'normal' AND object = 'wal' \gset io_sum_wal_normal_after_ +SELECT current_setting('synchronous_commit') = 'on'; + ?column? +---------- + t +(1 row) + +SELECT :io_sum_wal_normal_after_writes > :io_sum_wal_normal_before_writes; + ?column? +---------- + t +(1 row) + +SELECT current_setting('fsync') = 'off' + OR current_setting('wal_sync_method') IN ('open_sync', 'open_datasync') + OR :io_sum_wal_normal_after_fsyncs > :io_sum_wal_normal_before_fsyncs; + ?column? +---------- + t +(1 row) + +-- Change the tablespace so that the table is rewritten directly, then SELECT +-- from it to cause it to be read back into shared buffers. +SELECT sum(reads) AS io_sum_shared_before_reads + FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset +-- Do this in a transaction to prevent spurious failures due to concurrent accesses to our newly +-- rewritten table, e.g. by autovacuum. +BEGIN; +ALTER TABLE test_io_shared SET TABLESPACE regress_tblspace; +-- SELECT from the table so that the data is read into shared buffers and +-- context 'normal', object 'relation' reads are counted. +SELECT COUNT(*) FROM test_io_shared; + count +------- + 100 +(1 row) + +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT sum(reads) AS io_sum_shared_after_reads + FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset +SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads; + ?column? +---------- + t +(1 row) + +SELECT sum(hits) AS io_sum_shared_before_hits + FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset +-- Select from the table again to count hits. +-- Ensure we generate hits by forcing a nested loop self-join with no +-- materialize node. The outer side's buffer will stay pinned, preventing its +-- eviction, while we loop through the inner side and generate hits. +BEGIN; +SET LOCAL enable_nestloop TO on; SET LOCAL enable_mergejoin TO off; +SET LOCAL enable_hashjoin TO off; SET LOCAL enable_material TO off; +-- ensure plan stays as we expect it to +EXPLAIN (COSTS OFF) SELECT COUNT(*) FROM test_io_shared t1 INNER JOIN test_io_shared t2 USING (a); + QUERY PLAN +------------------------------------------- + Aggregate + -> Nested Loop + Join Filter: (t1.a = t2.a) + -> Seq Scan on test_io_shared t1 + -> Seq Scan on test_io_shared t2 +(5 rows) + +SELECT COUNT(*) FROM test_io_shared t1 INNER JOIN test_io_shared t2 USING (a); + count +------- + 100 +(1 row) + +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT sum(hits) AS io_sum_shared_after_hits + FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset +SELECT :io_sum_shared_after_hits > :io_sum_shared_before_hits; + ?column? +---------- + t +(1 row) + +DROP TABLE test_io_shared; +-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io: +-- - eviction of local buffers in order to reuse them +-- - reads of temporary table blocks into local buffers +-- - writes of local buffers to permanent storage +-- - extends of temporary tables +-- Set temp_buffers to its minimum so that we can trigger writes with fewer +-- inserted tuples. Do so in a new session in case temporary tables have been +-- accessed by previous tests in this session. +\c +SET temp_buffers TO 100; +CREATE TEMPORARY TABLE test_io_local(a int, b TEXT); +SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes + FROM pg_stat_io + WHERE context = 'normal' AND object = 'temp relation' \gset io_sum_local_before_ +-- Insert tuples into the temporary table, generating extends in the stats. +-- Insert enough values that we need to reuse and write out dirty local +-- buffers, generating evictions and writes. +INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200); +-- Ensure the table is large enough to exceed our temp_buffers setting. +SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100; + ?column? +---------- + t +(1 row) + +SELECT sum(reads) AS io_sum_local_before_reads + FROM pg_stat_io WHERE context = 'normal' AND object = 'temp relation' \gset +-- Read in evicted buffers, generating reads. +SELECT COUNT(*) FROM test_io_local; + count +------- + 5000 +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT sum(evictions) AS evictions, + sum(reads) AS reads, + sum(writes) AS writes, + sum(extends) AS extends + FROM pg_stat_io + WHERE context = 'normal' AND object = 'temp relation' \gset io_sum_local_after_ +SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions, + :io_sum_local_after_reads > :io_sum_local_before_reads, + :io_sum_local_after_writes > :io_sum_local_before_writes, + :io_sum_local_after_extends > :io_sum_local_before_extends; + ?column? | ?column? | ?column? | ?column? +----------+----------+----------+---------- + t | t | t | t +(1 row) + +-- Change the tablespaces so that the temporary table is rewritten to other +-- local buffers, exercising a different codepath than standard local buffer +-- writes. +ALTER TABLE test_io_local SET TABLESPACE regress_tblspace; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT sum(writes) AS io_sum_local_new_tblspc_writes + FROM pg_stat_io WHERE context = 'normal' AND object = 'temp relation' \gset +SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes; + ?column? +---------- + t +(1 row) + +RESET temp_buffers; +-- Test that reuse of strategy buffers and reads of blocks into these reused +-- buffers while VACUUMing are tracked in pg_stat_io. If there is sufficient +-- demand for shared buffers from concurrent queries, some buffers may be +-- pinned by other backends before they can be reused. In such cases, the +-- backend will evict a buffer from outside the ring and add it to the +-- ring. This is considered an eviction and not a reuse. +-- Set wal_skip_threshold smaller than the expected size of +-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will +-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL. +-- Writing it to WAL will result in the newly written relation pages being in +-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy +-- reads. +SET wal_skip_threshold = '1 kB'; +SELECT sum(reuses) AS reuses, sum(reads) AS reads, sum(evictions) AS evictions + FROM pg_stat_io WHERE context = 'vacuum' \gset io_sum_vac_strategy_before_ +CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false'); +INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 4500)i; +-- Ensure that the next VACUUM will need to perform IO by rewriting the table +-- first with VACUUM (FULL). +VACUUM (FULL) test_io_vac_strategy; +-- Use the minimum BUFFER_USAGE_LIMIT to cause reuses or evictions with the +-- smallest table possible. +VACUUM (PARALLEL 0, BUFFER_USAGE_LIMIT 128) test_io_vac_strategy; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT sum(reuses) AS reuses, sum(reads) AS reads, sum(evictions) AS evictions + FROM pg_stat_io WHERE context = 'vacuum' \gset io_sum_vac_strategy_after_ +SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads; + ?column? +---------- + t +(1 row) + +SELECT (:io_sum_vac_strategy_after_reuses + :io_sum_vac_strategy_after_evictions) > + (:io_sum_vac_strategy_before_reuses + :io_sum_vac_strategy_before_evictions); + ?column? +---------- + t +(1 row) + +RESET wal_skip_threshold; +-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE +-- BufferAccessStrategy, are tracked in pg_stat_io. +SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before + FROM pg_stat_io WHERE context = 'bulkwrite' \gset +CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after + FROM pg_stat_io WHERE context = 'bulkwrite' \gset +SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before; + ?column? +---------- + t +(1 row) + +-- Test IO stats reset +SELECT pg_stat_have_stats('io', 0, 0); + pg_stat_have_stats +-------------------- + t +(1 row) + +SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS io_stats_pre_reset + FROM pg_stat_io \gset +SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_pre_reset + FROM pg_stat_get_backend_io(pg_backend_pid()) \gset +SELECT pg_stat_reset_shared('io'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS io_stats_post_reset + FROM pg_stat_io \gset +SELECT :io_stats_post_reset < :io_stats_pre_reset; + ?column? +---------- + t +(1 row) + +SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset + FROM pg_stat_get_backend_io(pg_backend_pid()) \gset +-- pg_stat_reset_shared() did not reset backend IO stats +SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset; + ?column? +---------- + t +(1 row) + +-- but pg_stat_reset_backend_stats() does +SELECT pg_stat_reset_backend_stats(pg_backend_pid()); + pg_stat_reset_backend_stats +----------------------------- + +(1 row) + +SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_backend_reset + FROM pg_stat_get_backend_io(pg_backend_pid()) \gset +SELECT :my_io_stats_pre_reset > :my_io_stats_post_backend_reset; + ?column? +---------- + t +(1 row) + +-- Check invalid input for pg_stat_get_backend_io() +SELECT pg_stat_get_backend_io(NULL); + pg_stat_get_backend_io +------------------------ +(0 rows) + +SELECT pg_stat_get_backend_io(0); + pg_stat_get_backend_io +------------------------ +(0 rows) + +-- Auxiliary processes return no data. +SELECT pg_stat_get_backend_io(:checkpointer_pid); + pg_stat_get_backend_io +------------------------ +(0 rows) + +-- test BRIN index doesn't block HOT update +CREATE TABLE brin_hot ( + id integer PRIMARY KEY, + val integer NOT NULL +) WITH (autovacuum_enabled = off, fillfactor = 70); +INSERT INTO brin_hot SELECT *, 0 FROM generate_series(1, 235); +CREATE INDEX val_brin ON brin_hot using brin(val); +CREATE FUNCTION wait_for_hot_stats() RETURNS void AS $$ +DECLARE + start_time timestamptz := clock_timestamp(); + updated bool; +BEGIN + -- we don't want to wait forever; loop will exit after 30 seconds + FOR i IN 1 .. 300 LOOP + SELECT (pg_stat_get_tuples_hot_updated('brin_hot'::regclass::oid) > 0) INTO updated; + EXIT WHEN updated; + + -- wait a little + PERFORM pg_sleep_for('100 milliseconds'); + -- reset stats snapshot so we can test again + PERFORM pg_stat_clear_snapshot(); + END LOOP; + -- report time waited in postmaster log (where it won't change test output) + RAISE log 'wait_for_hot_stats delayed % seconds', + EXTRACT(epoch FROM clock_timestamp() - start_time); +END +$$ LANGUAGE plpgsql; +UPDATE brin_hot SET val = -3 WHERE id = 42; +-- We can't just call wait_for_hot_stats() at this point, because we only +-- transmit stats when the session goes idle, and we probably didn't +-- transmit the last couple of counts yet thanks to the rate-limiting logic +-- in pgstat_report_stat(). But instead of waiting for the rate limiter's +-- timeout to elapse, let's just start a new session. The old one will +-- then send its stats before dying. +\c - +SELECT wait_for_hot_stats(); + wait_for_hot_stats +-------------------- + +(1 row) + +SELECT pg_stat_get_tuples_hot_updated('brin_hot'::regclass::oid); + pg_stat_get_tuples_hot_updated +-------------------------------- + 1 +(1 row) + +DROP TABLE brin_hot; +DROP FUNCTION wait_for_hot_stats(); +-- Test handling of index predicates - updating attributes in precicates +-- should not block HOT when summarizing indexes are involved. We update +-- a row that was not indexed due to the index predicate, and becomes +-- indexable - the HOT-updated tuple is forwarded to the BRIN index. +CREATE TABLE brin_hot_2 (a int, b int); +INSERT INTO brin_hot_2 VALUES (1, 100); +CREATE INDEX ON brin_hot_2 USING brin (b) WHERE a = 2; +UPDATE brin_hot_2 SET a = 2; +EXPLAIN (COSTS OFF) SELECT * FROM brin_hot_2 WHERE a = 2 AND b = 100; + QUERY PLAN +----------------------------------- + Seq Scan on brin_hot_2 + Filter: ((a = 2) AND (b = 100)) +(2 rows) + +SELECT COUNT(*) FROM brin_hot_2 WHERE a = 2 AND b = 100; + count +------- + 1 +(1 row) + +SET enable_seqscan = off; +EXPLAIN (COSTS OFF) SELECT * FROM brin_hot_2 WHERE a = 2 AND b = 100; + QUERY PLAN +--------------------------------------------- + Bitmap Heap Scan on brin_hot_2 + Recheck Cond: ((b = 100) AND (a = 2)) + -> Bitmap Index Scan on brin_hot_2_b_idx + Index Cond: (b = 100) +(4 rows) + +SELECT COUNT(*) FROM brin_hot_2 WHERE a = 2 AND b = 100; + count +------- + 1 +(1 row) + +DROP TABLE brin_hot_2; +-- Test that updates to indexed columns are still propagated to the +-- BRIN column. +-- https://postgr.es/m/[email protected] +CREATE TABLE brin_hot_3 (a int, filler text) WITH (fillfactor = 10); +INSERT INTO brin_hot_3 SELECT 1, repeat(' ', 500) FROM generate_series(1, 20); +CREATE INDEX ON brin_hot_3 USING brin (a) WITH (pages_per_range = 1); +UPDATE brin_hot_3 SET a = 2; +EXPLAIN (COSTS OFF) SELECT * FROM brin_hot_3 WHERE a = 2; + QUERY PLAN +--------------------------------------------- + Bitmap Heap Scan on brin_hot_3 + Recheck Cond: (a = 2) + -> Bitmap Index Scan on brin_hot_3_a_idx + Index Cond: (a = 2) +(4 rows) + +SELECT COUNT(*) FROM brin_hot_3 WHERE a = 2; + count +------- + 20 +(1 row) + +DROP TABLE brin_hot_3; +SET enable_seqscan = on; +-- Test that estimation of relation size works with tuples wider than the +-- relation fillfactor. We create a table with wide inline attributes and +-- low fillfactor, insert rows and then see how many rows EXPLAIN shows +-- before running analyze. We disable autovacuum so that it does not +-- interfere with the test. +CREATE TABLE table_fillfactor ( + n char(1000) +) with (fillfactor=10, autovacuum_enabled=off); +INSERT INTO table_fillfactor +SELECT 'x' FROM generate_series(1,1000); +SELECT * FROM check_estimated_rows('SELECT * FROM table_fillfactor'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +DROP TABLE table_fillfactor; +-- Test some rewrites +CREATE TABLE test_2pc_timestamp (a int) WITH (autovacuum_enabled = false); +VACUUM ANALYZE test_2pc_timestamp; +SELECT last_analyze AS last_vacuum_analyze FROM pg_stat_all_tables WHERE relname = 'test_2pc_timestamp' \gset +BEGIN; +ALTER TABLE test_2pc_timestamp ALTER COLUMN a TYPE int; +PREPARE TRANSACTION 'test'; +ERROR: prepared transactions are disabled +HINT: Set "max_prepared_transactions" to a nonzero value. +COMMIT PREPARED 'test'; +ERROR: prepared transaction with identifier "test" does not exist +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT last_analyze = :'last_vacuum_analyze'::timestamptz FROM pg_stat_all_tables WHERE relname = 'test_2pc_timestamp'; + ?column? +---------- + t +(1 row) + +DROP TABLE test_2pc_timestamp; +CREATE TABLE test_2pc_rewrite_alone (a int); +INSERT INTO test_2pc_rewrite_alone VALUES (1); +BEGIN; +ALTER TABLE test_2pc_rewrite_alone ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +ERROR: prepared transactions are disabled +HINT: Set "max_prepared_transactions" to a nonzero value. +COMMIT PREPARED 'test'; +ERROR: prepared transaction with identifier "test" does not exist +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_rewrite_alone'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_2pc_rewrite_alone; +CREATE TABLE test_2pc (a int); +INSERT INTO test_2pc VALUES (1); +BEGIN; +INSERT INTO test_2pc VALUES (1); +INSERT INTO test_2pc VALUES (2); +INSERT INTO test_2pc VALUES (3); +ALTER TABLE test_2pc ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +ERROR: prepared transactions are disabled +HINT: Set "max_prepared_transactions" to a nonzero value. +COMMIT PREPARED 'test'; +ERROR: prepared transaction with identifier "test" does not exist +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 4 | 1 | 3 +(1 row) + +DROP TABLE test_2pc; +CREATE TABLE test_2pc_multi (a int); +INSERT INTO test_2pc_multi VALUES (1); +BEGIN; +INSERT INTO test_2pc_multi VALUES (1); +INSERT INTO test_2pc_multi VALUES (2); +ALTER TABLE test_2pc_multi ALTER COLUMN a TYPE bigint; +INSERT INTO test_2pc_multi VALUES (3); +INSERT INTO test_2pc_multi VALUES (4); +ALTER TABLE test_2pc_multi ALTER COLUMN a TYPE int; +INSERT INTO test_2pc_multi VALUES (5); +PREPARE TRANSACTION 'test'; +ERROR: prepared transactions are disabled +HINT: Set "max_prepared_transactions" to a nonzero value. +COMMIT PREPARED 'test'; +ERROR: prepared transaction with identifier "test" does not exist +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_multi'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 6 | 1 | 5 +(1 row) + +DROP TABLE test_2pc_multi; +CREATE TABLE test_2pc_rewrite_alone_abort (a int); +INSERT INTO test_2pc_rewrite_alone_abort VALUES (1); +BEGIN; +ALTER TABLE test_2pc_rewrite_alone_abort ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +ERROR: prepared transactions are disabled +HINT: Set "max_prepared_transactions" to a nonzero value. +ROLLBACK PREPARED 'test'; +ERROR: prepared transaction with identifier "test" does not exist +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_rewrite_alone_abort'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_2pc_rewrite_alone_abort; +CREATE TABLE test_2pc_abort (a int); +INSERT INTO test_2pc_abort VALUES (1); +BEGIN; +INSERT INTO test_2pc_abort VALUES (1); +INSERT INTO test_2pc_abort VALUES (2); +ALTER TABLE test_2pc_abort ALTER COLUMN a TYPE bigint; +INSERT INTO test_2pc_abort VALUES (3); +PREPARE TRANSACTION 'test'; +ERROR: prepared transactions are disabled +HINT: Set "max_prepared_transactions" to a nonzero value. +ROLLBACK PREPARED 'test'; +ERROR: prepared transaction with identifier "test" does not exist +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_abort'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 4 | 1 | 3 +(1 row) + +DROP TABLE test_2pc_abort; +CREATE TABLE test_2pc_savepoint (a int); +INSERT INTO test_2pc_savepoint VALUES (1); +BEGIN; +SAVEPOINT a; +INSERT INTO test_2pc_savepoint VALUES (1); +INSERT INTO test_2pc_savepoint VALUES (2); +ALTER TABLE test_2pc_savepoint ALTER COLUMN a TYPE bigint; +SAVEPOINT b; +INSERT INTO test_2pc_savepoint VALUES (3); +ALTER TABLE test_2pc_savepoint ALTER COLUMN a TYPE int; +SAVEPOINT c; +INSERT INTO test_2pc_savepoint VALUES (4); +INSERT INTO test_2pc_savepoint VALUES (5); +ROLLBACK TO SAVEPOINT b; +PREPARE TRANSACTION 'test'; +ERROR: prepared transactions are disabled +HINT: Set "max_prepared_transactions" to a nonzero value. +COMMIT PREPARED 'test'; +ERROR: prepared transaction with identifier "test" does not exist +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_savepoint'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 6 | 1 | 5 +(1 row) + +DROP TABLE test_2pc_savepoint; +-- Rewrite without 2PC +CREATE TABLE test_timestamp (a int) WITH (autovacuum_enabled = false); +VACUUM ANALYZE test_timestamp; +SELECT last_analyze AS last_vacuum_analyze FROM pg_stat_all_tables WHERE relname = 'test_timestamp' \gset +ALTER TABLE test_timestamp ALTER COLUMN a TYPE bigint; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT last_analyze = :'last_vacuum_analyze'::timestamptz FROM pg_stat_all_tables WHERE relname = 'test_timestamp'; + ?column? +---------- + t +(1 row) + +DROP TABLE test_timestamp; +CREATE TABLE test_alone (a int); +INSERT INTO test_alone VALUES (1); +BEGIN; +ALTER TABLE test_alone ALTER COLUMN a TYPE bigint; +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_alone'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_alone; +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +BEGIN; +INSERT INTO test VALUES (1); +INSERT INTO test VALUES (2); +INSERT INTO test VALUES (3); +ALTER TABLE test ALTER COLUMN a TYPE bigint; +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 4 | 4 | 0 +(1 row) + +DROP TABLE test; +CREATE TABLE test_multi (a int); +INSERT INTO test_multi VALUES (1); +BEGIN; +INSERT INTO test_multi VALUES (1); +INSERT INTO test_multi VALUES (2); +ALTER TABLE test_multi ALTER COLUMN a TYPE bigint; +INSERT INTO test_multi VALUES (3); +INSERT INTO test_multi VALUES (4); +ALTER TABLE test_multi ALTER COLUMN a TYPE int; +INSERT INTO test_multi VALUES (5); +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_multi'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 6 | 6 | 0 +(1 row) + +DROP TABLE test_multi; +CREATE TABLE test_rewrite_alone_abort (a int); +INSERT INTO test_rewrite_alone_abort VALUES (1); +BEGIN; +ALTER TABLE test_rewrite_alone_abort ALTER COLUMN a TYPE bigint; +ROLLBACK; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_rewrite_alone_abort'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_rewrite_alone_abort; +CREATE TABLE test_abort (a int); +INSERT INTO test_abort VALUES (1); +BEGIN; +INSERT INTO test_abort VALUES (1); +INSERT INTO test_abort VALUES (2); +ALTER TABLE test_abort ALTER COLUMN a TYPE bigint; +INSERT INTO test_abort VALUES (3); +ROLLBACK; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_abort'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 4 | 1 | 3 +(1 row) + +DROP TABLE test_abort; +CREATE TABLE test_savepoint (a int); +INSERT INTO test_savepoint VALUES (1); +BEGIN; +SAVEPOINT a; +INSERT INTO test_savepoint VALUES (1); +INSERT INTO test_savepoint VALUES (2); +ALTER TABLE test_savepoint ALTER COLUMN a TYPE bigint; +SAVEPOINT b; +INSERT INTO test_savepoint VALUES (3); +ALTER TABLE test_savepoint ALTER COLUMN a TYPE int; +SAVEPOINT c; +INSERT INTO test_savepoint VALUES (4); +INSERT INTO test_savepoint VALUES (5); +ROLLBACK TO SAVEPOINT b; +COMMIT; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_savepoint'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 6 | 3 | 3 +(1 row) + +DROP TABLE test_savepoint; +CREATE TABLE test_tbs (a int); +INSERT INTO test_tbs VALUES (1); +ALTER TABLE test_tbs SET TABLESPACE pg_default; +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_tbs'; + n_tup_ins | n_live_tup | n_dead_tup +-----------+------------+------------ + 1 | 1 | 0 +(1 row) + +DROP TABLE test_tbs; +-- End of Stats Test diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql index 8768e0f27fd..4130f9254a5 100644 --- a/src/test/regress/sql/stats.sql +++ b/src/test/regress/sql/stats.sql @@ -944,4 +944,190 @@ SELECT * FROM check_estimated_rows('SELECT * FROM table_fillfactor'); DROP TABLE table_fillfactor; +-- Test some rewrites +CREATE TABLE test_2pc_timestamp (a int) WITH (autovacuum_enabled = false); +VACUUM ANALYZE test_2pc_timestamp; +SELECT last_analyze AS last_vacuum_analyze FROM pg_stat_all_tables WHERE relname = 'test_2pc_timestamp' \gset +BEGIN; +ALTER TABLE test_2pc_timestamp ALTER COLUMN a TYPE int; +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); +SELECT last_analyze = :'last_vacuum_analyze'::timestamptz FROM pg_stat_all_tables WHERE relname = 'test_2pc_timestamp'; +DROP TABLE test_2pc_timestamp; + +CREATE TABLE test_2pc_rewrite_alone (a int); +INSERT INTO test_2pc_rewrite_alone VALUES (1); +BEGIN; +ALTER TABLE test_2pc_rewrite_alone ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_rewrite_alone'; +DROP TABLE test_2pc_rewrite_alone; + +CREATE TABLE test_2pc (a int); +INSERT INTO test_2pc VALUES (1); +BEGIN; +INSERT INTO test_2pc VALUES (1); +INSERT INTO test_2pc VALUES (2); +INSERT INTO test_2pc VALUES (3); +ALTER TABLE test_2pc ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc'; +DROP TABLE test_2pc; + +CREATE TABLE test_2pc_multi (a int); +INSERT INTO test_2pc_multi VALUES (1); +BEGIN; +INSERT INTO test_2pc_multi VALUES (1); +INSERT INTO test_2pc_multi VALUES (2); +ALTER TABLE test_2pc_multi ALTER COLUMN a TYPE bigint; +INSERT INTO test_2pc_multi VALUES (3); +INSERT INTO test_2pc_multi VALUES (4); +ALTER TABLE test_2pc_multi ALTER COLUMN a TYPE int; +INSERT INTO test_2pc_multi VALUES (5); +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_multi'; +DROP TABLE test_2pc_multi; + +CREATE TABLE test_2pc_rewrite_alone_abort (a int); +INSERT INTO test_2pc_rewrite_alone_abort VALUES (1); +BEGIN; +ALTER TABLE test_2pc_rewrite_alone_abort ALTER COLUMN a TYPE bigint; +PREPARE TRANSACTION 'test'; +ROLLBACK PREPARED 'test'; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_rewrite_alone_abort'; +DROP TABLE test_2pc_rewrite_alone_abort; + +CREATE TABLE test_2pc_abort (a int); +INSERT INTO test_2pc_abort VALUES (1); +BEGIN; +INSERT INTO test_2pc_abort VALUES (1); +INSERT INTO test_2pc_abort VALUES (2); +ALTER TABLE test_2pc_abort ALTER COLUMN a TYPE bigint; +INSERT INTO test_2pc_abort VALUES (3); +PREPARE TRANSACTION 'test'; +ROLLBACK PREPARED 'test'; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_abort'; +DROP TABLE test_2pc_abort; + +CREATE TABLE test_2pc_savepoint (a int); +INSERT INTO test_2pc_savepoint VALUES (1); +BEGIN; +SAVEPOINT a; +INSERT INTO test_2pc_savepoint VALUES (1); +INSERT INTO test_2pc_savepoint VALUES (2); +ALTER TABLE test_2pc_savepoint ALTER COLUMN a TYPE bigint; +SAVEPOINT b; +INSERT INTO test_2pc_savepoint VALUES (3); +ALTER TABLE test_2pc_savepoint ALTER COLUMN a TYPE int; +SAVEPOINT c; +INSERT INTO test_2pc_savepoint VALUES (4); +INSERT INTO test_2pc_savepoint VALUES (5); +ROLLBACK TO SAVEPOINT b; +PREPARE TRANSACTION 'test'; +COMMIT PREPARED 'test'; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_2pc_savepoint'; +DROP TABLE test_2pc_savepoint; + +-- Rewrite without 2PC +CREATE TABLE test_timestamp (a int) WITH (autovacuum_enabled = false); +VACUUM ANALYZE test_timestamp; +SELECT last_analyze AS last_vacuum_analyze FROM pg_stat_all_tables WHERE relname = 'test_timestamp' \gset +ALTER TABLE test_timestamp ALTER COLUMN a TYPE bigint; +SELECT pg_stat_force_next_flush(); +SELECT last_analyze = :'last_vacuum_analyze'::timestamptz FROM pg_stat_all_tables WHERE relname = 'test_timestamp'; +DROP TABLE test_timestamp; + +CREATE TABLE test_alone (a int); +INSERT INTO test_alone VALUES (1); +BEGIN; +ALTER TABLE test_alone ALTER COLUMN a TYPE bigint; +COMMIT; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_alone'; +DROP TABLE test_alone; + +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +BEGIN; +INSERT INTO test VALUES (1); +INSERT INTO test VALUES (2); +INSERT INTO test VALUES (3); +ALTER TABLE test ALTER COLUMN a TYPE bigint; +COMMIT; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test'; +DROP TABLE test; + +CREATE TABLE test_multi (a int); +INSERT INTO test_multi VALUES (1); +BEGIN; +INSERT INTO test_multi VALUES (1); +INSERT INTO test_multi VALUES (2); +ALTER TABLE test_multi ALTER COLUMN a TYPE bigint; +INSERT INTO test_multi VALUES (3); +INSERT INTO test_multi VALUES (4); +ALTER TABLE test_multi ALTER COLUMN a TYPE int; +INSERT INTO test_multi VALUES (5); +COMMIT; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_multi'; +DROP TABLE test_multi; + +CREATE TABLE test_rewrite_alone_abort (a int); +INSERT INTO test_rewrite_alone_abort VALUES (1); +BEGIN; +ALTER TABLE test_rewrite_alone_abort ALTER COLUMN a TYPE bigint; +ROLLBACK; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_rewrite_alone_abort'; +DROP TABLE test_rewrite_alone_abort; + +CREATE TABLE test_abort (a int); +INSERT INTO test_abort VALUES (1); +BEGIN; +INSERT INTO test_abort VALUES (1); +INSERT INTO test_abort VALUES (2); +ALTER TABLE test_abort ALTER COLUMN a TYPE bigint; +INSERT INTO test_abort VALUES (3); +ROLLBACK; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_abort'; +DROP TABLE test_abort; + +CREATE TABLE test_savepoint (a int); +INSERT INTO test_savepoint VALUES (1); +BEGIN; +SAVEPOINT a; +INSERT INTO test_savepoint VALUES (1); +INSERT INTO test_savepoint VALUES (2); +ALTER TABLE test_savepoint ALTER COLUMN a TYPE bigint; +SAVEPOINT b; +INSERT INTO test_savepoint VALUES (3); +ALTER TABLE test_savepoint ALTER COLUMN a TYPE int; +SAVEPOINT c; +INSERT INTO test_savepoint VALUES (4); +INSERT INTO test_savepoint VALUES (5); +ROLLBACK TO SAVEPOINT b; +COMMIT; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_savepoint'; +DROP TABLE test_savepoint; + +CREATE TABLE test_tbs (a int); +INSERT INTO test_tbs VALUES (1); +ALTER TABLE test_tbs SET TABLESPACE pg_default; +SELECT pg_stat_force_next_flush(); +SELECT n_tup_ins, n_live_tup, n_dead_tup FROM pg_stat_all_tables WHERE relname = 'test_tbs'; +DROP TABLE test_tbs; + -- End of Stats Test -- 2.34.1 --KMYjwlsACqdYx2P8 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-Key-PGSTAT_KIND_RELATION-by-relfile-locator.patch" ^ permalink raw reply [nested|flat] 50+ messages in thread
end of thread, other threads:[~2025-10-27 14:54 UTC | newest] Thread overview: 50+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2017-03-03 07:39 [PATCH 1/3] Rewrite sections in ddl.sgml related to partitioning amit <[email protected]> 2025-10-27 14:54 [PATCH v7 1/3] Add stats tests related to rewrite Bertrand Drouvot <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox