Subject: Re: [POC] Fast COPY FROM command for the table with foreign
 partitions
To: "Tang, Haiying" <tanghy.fnst@cn.fujitsu.com>
Cc: Alexey Kondratov <a.kondratov@postgrespro.ru>,
 Michael Paquier <michael@paquier.xyz>,
 Ashutosh Bapat <ashutosh.bapat@2ndquadrant.com>,
 Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
 Etsuro Fujita <etsuro.fujita@gmail.com>,
 PostgreSQL-Dev <pgsql-hackers@postgresql.org>,
 "Hou, Zhijie" <houzj.fnst@cn.fujitsu.com>,
 Amit Langote <amitlangote09@gmail.com>,
 "Tsunakawa, Takayuki" <tsunakawa.takay@fujitsu.com>
References: <ae25e79e-6122-720e-1349-fb832c6021df@postgrespro.ru>
 <8ddacbf0-d489-cd56-fecc-e09a7e02a7c1@postgrespro.ru>
 <TYAPR01MB2990DC396B338C98F27C8ED3FE1F0@TYAPR01MB2990.jpnprd01.prod.outlook.com>
 <CA+HiwqEZG__hJp2Zp1o5u-brPnF985nKnTqs9ox3PKHy5U=OMw@mail.gmail.com>
 <TYAPR01MB299070A2B01D63A31337F105FEF90@TYAPR01MB2990.jpnprd01.prod.outlook.com>
 <CA+HiwqFxJksYQtGhje-VozoC3GYxyOtv5QxNb3fJ5O1FHOB-+g@mail.gmail.com>
 <TYAPR01MB29900B0604FEB2D7AD9EE61BFEF40@TYAPR01MB2990.jpnprd01.prod.outlook.com>
 <CA+HiwqFJG-cJtdWZAqyEQEnA4e-rd=H2dzF+uNSbh5O-1O9tyw@mail.gmail.com>
 <9bff5ccc-8649-9699-1cdd-82b53164f9bb@postgrespro.ru>
 <7166ded145a947fbb11b67ebe68041f3@G08CNEXMBPEKD05.g08.fujitsu.local>
 <c46c1929-d44f-0ce3-a74b-4cfaddf9da82@postgrespro.ru>
 <2b465f4cdeeb4a78b3096d31e8af7c77@G08CNEXMBPEKD05.g08.fujitsu.local>
 <ca933cda-b8c3-9e9c-23a0-91af2e2a2624@postgrespro.ru>
 <bc75078fc9614263a0d5f006dd6f0c3c@G08CNEXMBPEKD05.g08.fujitsu.local>
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Organization: Postgres Professional
Message-ID: <880ddd37-bada-909e-b5b3-d9d20286e963@postgrespro.ru>
Date: Tue, 12 Jan 2021 08:45:19 +0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
 <bc75078fc9614263a0d5f006dd6f0c3c@G08CNEXMBPEKD05.g08.fujitsu.local>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 1/11/21 4:59 PM, Tang, Haiying wrote:
> Hi Andrey,
> 
> I had a general look at this extension feature, I think it's beneficial for some application scenarios of PostgreSQL. So I did 7 performance cases test on your patch(v13). The results are really good. As you can see below we can get 7-10 times improvement with this patch.
> 
> PSA test_copy_from.sql shows my test cases detail(I didn't attach my data file since it's too big).
> 
> Below are the test results:
> 'Test No' corresponds to the number(0 1...6) in attached test_copy_from.sql.
> %reg=(Patched-Unpatched)/Unpatched), Unit is millisecond.
> 
> |Test No| Test Case                                                                               |Patched(ms)  | Unpatched(ms) |%reg   |
> |-------|-----------------------------------------------------------------------------------------|-------------|---------------|-------|
> |0      |COPY FROM insertion into the partitioned table(parition is foreign table)                | 102483.223  |  1083300.907  |  -91% |
> |1      |COPY FROM insertion into the partitioned table(parition is foreign partition)            | 104779.893  |  1207320.287  |  -91% |
> |2      |COPY FROM insertion into the foreign table(without partition)                            | 100268.730  |  1077309.158  |  -91% |
> |3      |COPY FROM insertion into the partitioned table(part of foreign partitions)               | 104110.620  |  1134781.855  |  -91% |
> |4      |COPY FROM insertion into the partitioned table with constraint(part of foreign partition)| 136356.201  |  1238539.603  |  -89% |
> |5      |COPY FROM insertion into the foreign table with constraint(without partition)            | 136818.262  |  1189921.742  |  -89% |
> |6      |\copy insertion into the partitioned table with constraint.                              | 140368.072  |  1242689.924  |  -89% |
> 
> If there is any question on my tests, please feel free to ask.
> 
> Best Regard,
> Tang
Thank you for this work.
Sometimes before i suggested additional optimization [1] which can 
additionally speed up COPY by 2-4 times. Maybe you can perform the 
benchmark for this solution too?

[1] 
https://www.postgresql.org/message-id/da7ed3f5-b596-2549-3710-4cc2a602ec17%40postgrespro.ru

-- 
regards,
Andrey Lepikhov
Postgres Professional