Date: Fri, 02 Feb 2024 09:40:56 +0900 (JST)
Message-Id: <20240202.094056.2076796621253456531.kou@clear-code.com>
To: michael@paquier.xyz
Cc: sawada.mshk@gmail.com, zhjwpku@gmail.com, andrew@dunslane.net,
 nathandbossart@gmail.com, pgsql-hackers@postgresql.org
Subject: Re: Make COPY format extendable: Extract COPY TO format
 implementations
From: Sutou Kouhei <kou@clear-code.com>
In-Reply-To: <ZbwSRsCqVS638Xjz@paquier.xyz>
References: <ZbsU53b3eEV-mMT3@paquier.xyz>
	<20240202.001951.2021771265443873668.kou@clear-code.com>
	<ZbwSRsCqVS638Xjz@paquier.xyz>
Mime-Version: 1.0
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/20240202.094056.2076796621253456531.kou%40clear-code.com>
Precedence: bulk

Hi,

In <ZbwSRsCqVS638Xjz@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 Feb 2024 06:51:02 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> On Fri, Feb 02, 2024 at 12:19:51AM +0900, Sutou Kouhei wrote:
>> Here are some numbers on my local machine (Note that my
>> local machine isn't suitable for benchmark as I said
>> before. Each number is median of "\watch 15" results):
>>>
>> I'll measure again on my local machine later. I'll stop
>> other processes such as Web browser, editor and so on as
>> much as possible when I do.
> 
> Thanks for compiling some numbers.  This is showing a lot of variance.
> Expecially, these two lines in table 2 are showing surprising results
> for v7:
>   direction     format  n_columns     master         v7        v10
>        from        csv          1    917.973   1695.401    871.991
>        from     binary          1    841.104   1422.012    820.786

Here are more numbers:

1:
 direction     format  n_columns     master         v7        v10
        to       text          1   1053.844    978.998    956.575
        to        csv          1   1091.316   1020.584   1098.314
        to     binary          1   1034.685    969.224    980.458
        to       text         10   4216.264   3886.515   4111.417
        to        csv         10   4649.228   4530.882   4682.988
        to     binary         10   4219.228    4189.99   4211.942
      from       text          1    851.697    896.968    890.458
      from        csv          1    890.229    936.231     887.15
      from     binary          1    784.407     817.07    938.736
      from       text         10   2549.056   2233.899   2630.892
      from        csv         10   2809.441   2868.411   2895.196
      from     binary         10   2985.674   3027.522     3397.5

2:
 direction     format  n_columns     master         v7        v10
        to       text          1   1013.764   1011.968    940.855
        to        csv          1   1060.431   1065.468    1040.68
        to     binary          1   1013.652   1009.956    965.675
        to       text         10   4411.484   4031.571   3896.836
        to        csv         10   4739.625    4715.81   4631.002
        to     binary         10   4374.077   4357.942   4227.215
      from       text          1    955.078    922.346    866.222
      from        csv          1   1040.717    986.524    905.657
      from     binary          1    849.316    864.859    833.152
      from       text         10   2703.209   2361.651   2533.992
      from        csv         10    2990.35   3059.167   2930.632
      from     binary         10   3008.375   3368.714   3055.723

3:
 direction     format  n_columns     master         v7        v10
        to       text          1   1084.756   1003.822    994.409
        to        csv          1     1092.4   1062.536   1079.027
        to     binary          1   1046.774    994.168    993.633
        to       text         10    4363.51   3978.205   4124.359
        to        csv         10   4866.762   4616.001   4715.052
        to     binary         10   4382.412   4363.269   4213.456
      from       text          1    852.976    907.315    860.749
      from        csv          1    925.187    962.632    897.833
      from     binary          1    824.997    897.046    828.231
      from       text         10    2591.07   2358.541   2540.431
      from        csv         10   2907.033   3018.486   2915.997
      from     binary         10   3069.027    3209.21   3119.128

Other processes are stopped while I measure them. But I'm
not sure these numbers are more reliable than before...

> I am going to try to plug in some rusage() calls in the backend for
> the COPY paths.  I hope that gives more precision about the backend
> activity.  I'll post that with more numbers.

Thanks. It'll help us.


-- 
kou