public inbox for [email protected]
help / color / mirror / Atom feed9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
14+ messages / 3 participants
[nested] [flat]
* 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
@ 2016-05-25 14:33 Vladimir Borodin <[email protected]>
2016-05-27 16:57 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Антон Бушмелев <[email protected]>
0 siblings, 2 replies; 14+ messages in thread
From: Vladimir Borodin @ 2016-05-25 14:33 UTC (permalink / raw)
To: pgsql-performance; [email protected]
Hi all.
We have found that queries through PgBouncer 1.7.2 (with transaction pooling) to local PostgreSQL are almost two times slower in 9.5.3 than in 9.4.8 on RHEL 6 hosts (all packages are updated to last versions). Meanwhile the problem can’t be reproduced i.e. on Ubuntu 14.04 (also fully-updated).
Here is how the results look like for 9.4, 9.5 and 9.6. All are built from latest commits on yesterday in
* REL9_4_STABLE (a0cc89a28141595d888d8aba43163d58a1578bfb),
* REL9_5_STABLE (e504d915bbf352ecfc4ed335af934e799bf01053),
* master (6ee7fb8244560b7a3f224784b8ad2351107fa55d).
All of them are build on the host where testing is done (with stock gcc versions). Sysctls, pgbouncer config and everything we found are the same, postgres configs are default, PGDATA is in tmpfs. All numbers are reproducible, they are stable between runs.
Shortly:
OS PostgreSQL version TPS Avg. latency
RHEL 6 9.4 44898 1.425 ms
RHEL 6 9.5 26199 2.443 ms
RHEL 6 9.5 43027 1.487 ms
Ubuntu 14.04 9.4 67458 0.949 ms
Ubuntu 14.04 9.5 64065 0.999 ms
Ubuntu 14.04 9.6 64350 0.995 ms
You could see that the difference between major versions on Ubuntu is not significant, but on RHEL 9.5 is 70% slower than 9.4 and 9.6.
Below are more details.
RHEL 6:
postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg94'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 2693962
latency average: 1.425 ms
tps = 44897.461518 (including connections establishing)
tps = 44898.763258 (excluding connections establishing)
postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg95'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 1572014
latency average: 2.443 ms
tps = 26198.928627 (including connections establishing)
tps = 26199.803363 (excluding connections establishing)
postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg96'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 2581645
latency average: 1.487 ms
tps = 43025.676995 (including connections establishing)
tps = 43027.038275 (excluding connections establishing)
postgres@pgload05g ~ $
Ubuntu 14.04 (the same hardware):
postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg94'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 4047653
latency average: 0.949 ms
tps = 67458.361515 (including connections establishing)
tps = 67459.983480 (excluding connections establishing)
postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg95'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 3844084
latency average: 0.999 ms
tps = 64065.447458 (including connections establishing)
tps = 64066.943627 (excluding connections establishing)
postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg96'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 3861088
latency average: 0.995 ms
tps = 64348.573126 (including connections establishing)
tps = 64350.195750 (excluding connections establishing)
postgres@pgloadpublic02:~$
In both tests (RHEL and Ubuntu) the bottleneck is performance of singe CPU core which is 100% consumed by PgBouncer. If pgbench connects to postgres directly I get the following (expected) numbers:
postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5432'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 10010710
latency average: 0.384 ms
tps = 166835.937859 (including connections establishing)
tps = 166849.730224 (excluding connections establishing)
postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5433'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 13373890
latency average: 0.287 ms
tps = 222888.311289 (including connections establishing)
tps = 222951.470125 (excluding connections establishing)
postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5434'
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 64
number of threads: 64
duration: 60 s
number of transactions actually processed: 12989816
latency average: 0.296 ms
tps = 216487.458399 (including connections establishing)
tps = 216548.069976 (excluding connections establishing)
postgres@pgload05g ~ $
Compilation options look almost the same:
# RHEL 6
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2
# Ubuntu
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -O2
Attached are a simple script to deploy the testing environment (PgBouncer should be installed) and pgbouncer config. I could provide any other needed information like backtraces or perf reports or anything else.
--
May the force be with you…
https://simply.name
Attachments:
[application/octet-stream] pgbouncer.ini (7.3K, 3-pgbouncer.ini)
download
[application/octet-stream] deploy.sh (1.3K, 5-deploy.sh)
download
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
@ 2016-05-27 16:57 ` Vladimir Borodin <[email protected]>
2016-05-27 21:56 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-05-31 09:06 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
1 sibling, 2 replies; 14+ messages in thread
From: Vladimir Borodin @ 2016-05-27 16:57 UTC (permalink / raw)
To: PostgreSQL-development <[email protected]>; [email protected]
-performance
+hackers
> 25 мая 2016 г., в 17:33, Vladimir Borodin <[email protected]> написал(а):
>
> Hi all.
>
> We have found that queries through PgBouncer 1.7.2 (with transaction pooling) to local PostgreSQL are almost two times slower in 9.5.3 than in 9.4.8 on RHEL 6 hosts (all packages are updated to last versions). Meanwhile the problem can’t be reproduced i.e. on Ubuntu 14.04 (also fully-updated).
>
> Here is how the results look like for 9.4, 9.5 and 9.6. All are built from latest commits on yesterday in
> * REL9_4_STABLE (a0cc89a28141595d888d8aba43163d58a1578bfb),
> * REL9_5_STABLE (e504d915bbf352ecfc4ed335af934e799bf01053),
> * master (6ee7fb8244560b7a3f224784b8ad2351107fa55d).
>
> All of them are build on the host where testing is done (with stock gcc versions). Sysctls, pgbouncer config and everything we found are the same, postgres configs are default, PGDATA is in tmpfs. All numbers are reproducible, they are stable between runs.
>
> Shortly:
>
> OS PostgreSQL version TPS Avg. latency
> RHEL 6 9.4 44898 1.425 ms
> RHEL 6 9.5 26199 2.443 ms
> RHEL 6 9.5 43027 1.487 ms
> Ubuntu 14.04 9.4 67458 0.949 ms
> Ubuntu 14.04 9.5 64065 0.999 ms
> Ubuntu 14.04 9.6 64350 0.995 ms
The results above are not really fair, pgbouncer.ini was a bit different on Ubuntu host (application_name_add_host was disabled). Here are the right results with exactly the same configuration:
OS PostgreSQL version TPS Avg. latency
RHEL 6 9.4 44898 1.425 ms
RHEL 6 9.5 26199 2.443 ms
RHEL 6 9.5 43027 1.487 ms
Ubuntu 14.04 9.4 45971 1.392 ms
Ubuntu 14.04 9.5 40282 1.589 ms
Ubuntu 14.04 9.6 45410 1.409 ms
It can be seen that there is a regression for 9.5 in Ubuntu also, but not so significant. We first thought that the reason is 38628db8d8caff21eb6cf8d775c0b2d04cf07b9b (Add memory barriers for PgBackendStatus.st_changecount protocol), but in that case the regression should also be seen in 9.6 also.
There also was a bunch of changes in FE/BE communication (like 387da18874afa17156ee3af63766f17efb53c4b9 or 98a64d0bd713cb89e61bef6432befc4b7b5da59e) and that may answer the question of regression in 9.5 and normal results in 9.6. Probably the right way to find the answer is to do bisect. I’ll do it but if some more diagnostics information can help, feel free to ask about it.
>
> You could see that the difference between major versions on Ubuntu is not significant, but on RHEL 9.5 is 70% slower than 9.4 and 9.6.
>
> Below are more details.
>
> RHEL 6:
>
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg94'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 2693962
> latency average: 1.425 ms
> tps = 44897.461518 (including connections establishing)
> tps = 44898.763258 (excluding connections establishing)
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg95'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 1572014
> latency average: 2.443 ms
> tps = 26198.928627 (including connections establishing)
> tps = 26199.803363 (excluding connections establishing)
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg96'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 2581645
> latency average: 1.487 ms
> tps = 43025.676995 (including connections establishing)
> tps = 43027.038275 (excluding connections establishing)
> postgres@pgload05g ~ $
>
> Ubuntu 14.04 (the same hardware):
>
> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg94'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 2758348
> latency average: 1.392 ms
> tps = 45970.634737 (including connections establishing)
> tps = 45971.531098 (excluding connections establishing)
> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg95'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 2417009
> latency average: 1.589 ms
> tps = 40282.003641 (including connections establishing)
> tps = 40282.855938 (excluding connections establishing)
> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg96'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 2724666
> latency average: 1.409 ms
> tps = 45409.308603 (including connections establishing)
> tps = 45410.152406 (excluding connections establishing)
> postgres@pgloadpublic02:~$
>
> In both tests (RHEL and Ubuntu) the bottleneck is performance of singe CPU core which is 100% consumed by PgBouncer. If pgbench connects to postgres directly I get the following (expected) numbers:
>
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5432'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 10010710
> latency average: 0.384 ms
> tps = 166835.937859 (including connections establishing)
> tps = 166849.730224 (excluding connections establishing)
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5433'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 13373890
> latency average: 0.287 ms
> tps = 222888.311289 (including connections establishing)
> tps = 222951.470125 (excluding connections establishing)
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5434'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 12989816
> latency average: 0.296 ms
> tps = 216487.458399 (including connections establishing)
> tps = 216548.069976 (excluding connections establishing)
> postgres@pgload05g ~ $
>
> Compilation options look almost the same:
> # RHEL 6
> CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2
> # Ubuntu
> CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -O2
>
> Attached are a simple script to deploy the testing environment (PgBouncer should be installed) and pgbouncer config. I could provide any other needed information like backtraces or perf reports or anything else.
>
> <pgbouncer.ini>
> <deploy.sh>
>
> --
> May the force be with you…
> https://simply.name <https://simply.name/;
--
May the force be with you…
https://simply.name
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-05-27 16:57 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
@ 2016-05-27 21:56 ` Andres Freund <[email protected]>
2016-05-30 14:53 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-05-30 17:03 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
1 sibling, 2 replies; 14+ messages in thread
From: Andres Freund @ 2016-05-27 21:56 UTC (permalink / raw)
To: Vladimir Borodin <[email protected]>; +Cc: PostgreSQL-development <[email protected]>; [email protected]
Hi,
On 2016-05-27 19:57:34 +0300, Vladimir Borodin wrote:
> -performance
> > Here is how the results look like for 9.4, 9.5 and 9.6. All are built from latest commits on yesterday in
> > * REL9_4_STABLE (a0cc89a28141595d888d8aba43163d58a1578bfb),
> > * REL9_5_STABLE (e504d915bbf352ecfc4ed335af934e799bf01053),
> > * master (6ee7fb8244560b7a3f224784b8ad2351107fa55d).
> >
> > All of them are build on the host where testing is done (with stock gcc versions). Sysctls, pgbouncer config and everything we found are the same, postgres configs are default, PGDATA is in tmpfs. All numbers are reproducible, they are stable between runs.
> >
> > Shortly:
> >
> > OS PostgreSQL version TPS Avg. latency
> > RHEL 6 9.4 44898 1.425 ms
> > RHEL 6 9.5 26199 2.443 ms
> > RHEL 6 9.5 43027 1.487 ms
> > Ubuntu 14.04 9.4 67458 0.949 ms
> > Ubuntu 14.04 9.5 64065 0.999 ms
> > Ubuntu 14.04 9.6 64350 0.995 ms
>
> The results above are not really fair, pgbouncer.ini was a bit different on Ubuntu host (application_name_add_host was disabled). Here are the right results with exactly the same configuration:
>
> OS PostgreSQL version TPS Avg. latency
> RHEL 6 9.4 44898 1.425 ms
> RHEL 6 9.5 26199 2.443 ms
> RHEL 6 9.5 43027 1.487 ms
> Ubuntu 14.04 9.4 45971 1.392 ms
> Ubuntu 14.04 9.5 40282 1.589 ms
> Ubuntu 14.04 9.6 45410 1.409 ms
Hm. I'm a bit confused. You show one result for 9.5 with bad and one
with good performance. I suspect the second one is supposed to be a 9.6?
Am I understanding correctly that the performance near entirely
recovered with 9.6? If so, I suspect we might be dealing with a memory
alignment issue. Do the 9.5 results change if you increase
max_connections by one or two (without changing anything else)?
What's the actual hardware?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-05-27 16:57 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-05-27 21:56 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
@ 2016-05-30 14:53 ` Vladimir Borodin <[email protected]>
1 sibling, 0 replies; 14+ messages in thread
From: Vladimir Borodin @ 2016-05-30 14:53 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: PostgreSQL-development <[email protected]>; [email protected]
> 28 мая 2016 г., в 0:56, Andres Freund <[email protected]> написал(а):
>
> Hi,
>
>
> On 2016-05-27 19:57:34 +0300, Vladimir Borodin wrote:
>> -performance
>>> Here is how the results look like for 9.4, 9.5 and 9.6. All are built from latest commits on yesterday in
>>> * REL9_4_STABLE (a0cc89a28141595d888d8aba43163d58a1578bfb),
>>> * REL9_5_STABLE (e504d915bbf352ecfc4ed335af934e799bf01053),
>>> * master (6ee7fb8244560b7a3f224784b8ad2351107fa55d).
>>>
>>> All of them are build on the host where testing is done (with stock gcc versions). Sysctls, pgbouncer config and everything we found are the same, postgres configs are default, PGDATA is in tmpfs. All numbers are reproducible, they are stable between runs.
>>>
>>> Shortly:
>>>
>>> OS PostgreSQL version TPS Avg. latency
>>> RHEL 6 9.4 44898 1.425 ms
>>> RHEL 6 9.5 26199 2.443 ms
>>> RHEL 6 9.5 43027 1.487 ms
>>> Ubuntu 14.04 9.4 67458 0.949 ms
>>> Ubuntu 14.04 9.5 64065 0.999 ms
>>> Ubuntu 14.04 9.6 64350 0.995 ms
>>
>> The results above are not really fair, pgbouncer.ini was a bit different on Ubuntu host (application_name_add_host was disabled). Here are the right results with exactly the same configuration:
>>
>> OS PostgreSQL version TPS Avg. latency
>> RHEL 6 9.4 44898 1.425 ms
>> RHEL 6 9.5 26199 2.443 ms
>> RHEL 6 9.5 43027 1.487 ms
>> Ubuntu 14.04 9.4 45971 1.392 ms
>> Ubuntu 14.04 9.5 40282 1.589 ms
>> Ubuntu 14.04 9.6 45410 1.409 ms
>
> Hm. I'm a bit confused. You show one result for 9.5 with bad and one
> with good performance. I suspect the second one is supposed to be a 9.6?
No, they are both for 9.5. One of them is on RHEL 6 host, another one on Ubuntu 14.04.
>
> Am I understanding correctly that the performance near entirely
> recovered with 9.6?
Yes, 9.6 is much better than 9.5.
> If so, I suspect we might be dealing with a memory
> alignment issue. Do the 9.5 results change if you increase
> max_connections by one or two (without changing anything else)?
Results with max_connections=100:
OS Version TPS Avg. latency
RHEL 6 9.4 69810 0.917
RHEL 6 9.5 35303 1.812
RHEL 6 9.6 71827 0.891
Ubuntu 14.04 9.4 76829 0.833
Ubuntu 14.04 9.5 67574 0.947
Ubuntu 14.04 9.6 79200 0.808
Results with max_connections=101:
OS Version TPS Avg. latency
RHEL 6 9.4 70059 0.914
RHEL 6 9.5 35979 1.779
RHEL 6 9.6 71183 0.899
Ubuntu 14.04 9.4 78934 0.811
Ubuntu 14.04 9.5 67803 0.944
Ubuntu 14.04 9.6 79624 0.804
Results with max_connections=102:
OS Version TPS Avg. latency
RHEL 6 9.4 70710 0.905
RHEL 6 9.5 36615 1.748
RHEL 6 9.6 69742 0.918
Ubuntu 14.04 9.4 76356 0.838
Ubuntu 14.04 9.5 66814 0.958
Ubuntu 14.04 9.6 78528 0.815
Doesn’t seem that it is a memory alignment issue. Also please note that there is no performance degradation when connections from pgbench to postgres are established without pgbouncer:
OS Version TPS Avg. latency
RHEL 6 9.4 167427 0.382
RHEL 6 9.5 223674 0.286
RHEL 6 9.6 215580 0.297
Ubuntu 14.04 9.4 176659 0.362
Ubuntu 14.04 9.5 248277 0.258
Ubuntu 14.04 9.6 245871 0.260
>
> What's the actual hardware?
Host with RHEL has Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2 sockets, 16 physical cores, 32 cores with Hyper-Threading) and 256 GB of RAM while host with Ubuntu has Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz (2 sockets, 16 physical cores, 32 cores with Hyper-Threading) and 128 GB of RAM.
>
> Greetings,
>
> Andres Freund
--
May the force be with you…
https://simply.name
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-05-27 16:57 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-05-27 21:56 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
@ 2016-05-30 17:03 ` Vladimir Borodin <[email protected]>
1 sibling, 0 replies; 14+ messages in thread
From: Vladimir Borodin @ 2016-05-30 17:03 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: PostgreSQL-development <[email protected]>; [email protected]
> 28 мая 2016 г., в 0:56, Andres Freund <[email protected]> написал(а):
>
> On 2016-05-27 19:57:34 +0300, Vladimir Borodin wrote:
>>
>> OS PostgreSQL version TPS Avg. latency
>> RHEL 6 9.4 44898 1.425 ms
>> RHEL 6 9.5 26199 2.443 ms
>> RHEL 6 9.5 43027 1.487 ms
>
> Hm. I'm a bit confused. You show one result for 9.5 with bad and one
> with good performance. I suspect the second one is supposed to be a 9.6?
Sorry, I misunderstood. Yes, the last line above is for 9.6, that was a typo.
>
> Greetings,
>
> Andres Freund
--
May the force be with you…
https://simply.name
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-05-27 16:57 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
@ 2016-05-31 09:06 ` Vladimir Borodin <[email protected]>
1 sibling, 0 replies; 14+ messages in thread
From: Vladimir Borodin @ 2016-05-31 09:06 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: PostgreSQL-development <[email protected]>; [email protected]
> 27 мая 2016 г., в 19:57, Vladimir Borodin <[email protected]> написал(а):
>
> -performance
> +hackers
>
>> 25 мая 2016 г., в 17:33, Vladimir Borodin <[email protected] <mailto:[email protected]>> написал(а):
>>
>> Hi all.
>>
>> We have found that queries through PgBouncer 1.7.2 (with transaction pooling) to local PostgreSQL are almost two times slower in 9.5.3 than in 9.4.8 on RHEL 6 hosts (all packages are updated to last versions). Meanwhile the problem can’t be reproduced i.e. on Ubuntu 14.04 (also fully-updated).
>>
>> Here is how the results look like for 9.4, 9.5 and 9.6. All are built from latest commits on yesterday in
>> * REL9_4_STABLE (a0cc89a28141595d888d8aba43163d58a1578bfb),
>> * REL9_5_STABLE (e504d915bbf352ecfc4ed335af934e799bf01053),
>> * master (6ee7fb8244560b7a3f224784b8ad2351107fa55d).
>>
>> All of them are build on the host where testing is done (with stock gcc versions). Sysctls, pgbouncer config and everything we found are the same, postgres configs are default, PGDATA is in tmpfs. All numbers are reproducible, they are stable between runs.
>>
>> Shortly:
>>
>> OS PostgreSQL version TPS Avg. latency
>> RHEL 6 9.4 44898 1.425 ms
>> RHEL 6 9.5 26199 2.443 ms
>> RHEL 6 9.5 43027 1.487 ms
>> Ubuntu 14.04 9.4 67458 0.949 ms
>> Ubuntu 14.04 9.5 64065 0.999 ms
>> Ubuntu 14.04 9.6 64350 0.995 ms
>
> The results above are not really fair, pgbouncer.ini was a bit different on Ubuntu host (application_name_add_host was disabled). Here are the right results with exactly the same configuration:
>
> OS PostgreSQL version TPS Avg. latency
> RHEL 6 9.4 44898 1.425 ms
> RHEL 6 9.5 26199 2.443 ms
> RHEL 6 9.5 43027 1.487 ms
> Ubuntu 14.04 9.4 45971 1.392 ms
> Ubuntu 14.04 9.5 40282 1.589 ms
> Ubuntu 14.04 9.6 45410 1.409 ms
>
> It can be seen that there is a regression for 9.5 in Ubuntu also, but not so significant. We first thought that the reason is 38628db8d8caff21eb6cf8d775c0b2d04cf07b9b (Add memory barriers for PgBackendStatus.st <http://pgbackendstatus.st/>_changecount protocol), but in that case the regression should also be seen in 9.6 also.
>
> There also was a bunch of changes in FE/BE communication (like 387da18874afa17156ee3af63766f17efb53c4b9 or 98a64d0bd713cb89e61bef6432befc4b7b5da59e) and that may answer the question of regression in 9.5 and normal results in 9.6. Probably the right way to find the answer is to do bisect. I’ll do it but if some more diagnostics information can help, feel free to ask about it.
Yep, bisect confirms that the first bad commit in REL9_5_STABLE is 387da18874afa17156ee3af63766f17efb53c4b9. Full output is attached.
And bisect for master branch confirms that the situation became much better after 98a64d0bd713cb89e61bef6432befc4b7b5da59e. Output is also attached.
On Ubuntu performance degradation is ~15% and on RHEL it is ~100%. I don’t know what is the cause for different numbers on RHEL and Ubuntu but certainly there is a regression when pgbouncer is connected to postgres through localhost. When I try to connect pgbouncer to postgres through unix-socket performance is constantly bad on all postgres versions.
Both servers are for testing but I can easily provide you SSH access only to Ubuntu host if necessary. I can also gather more diagnostics if needed.
>
>>
>> You could see that the difference between major versions on Ubuntu is not significant, but on RHEL 9.5 is 70% slower than 9.4 and 9.6.
>>
>> Below are more details.
>>
>> RHEL 6:
>>
>> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg94'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 2693962
>> latency average: 1.425 ms
>> tps = 44897.461518 (including connections establishing)
>> tps = 44898.763258 (excluding connections establishing)
>> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg95'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 1572014
>> latency average: 2.443 ms
>> tps = 26198.928627 (including connections establishing)
>> tps = 26199.803363 (excluding connections establishing)
>> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg96'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 2581645
>> latency average: 1.487 ms
>> tps = 43025.676995 (including connections establishing)
>> tps = 43027.038275 (excluding connections establishing)
>> postgres@pgload05g ~ $
>>
>> Ubuntu 14.04 (the same hardware):
>>
>> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg94'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 2758348
>> latency average: 1.392 ms
>> tps = 45970.634737 (including connections establishing)
>> tps = 45971.531098 (excluding connections establishing)
>> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg95'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 2417009
>> latency average: 1.589 ms
>> tps = 40282.003641 (including connections establishing)
>> tps = 40282.855938 (excluding connections establishing)
>> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg96'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 2724666
>> latency average: 1.409 ms
>> tps = 45409.308603 (including connections establishing)
>> tps = 45410.152406 (excluding connections establishing)
>> postgres@pgloadpublic02:~$
>>
>> In both tests (RHEL and Ubuntu) the bottleneck is performance of singe CPU core which is 100% consumed by PgBouncer. If pgbench connects to postgres directly I get the following (expected) numbers:
>>
>> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5432'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 10010710
>> latency average: 0.384 ms
>> tps = 166835.937859 (including connections establishing)
>> tps = 166849.730224 (excluding connections establishing)
>> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5433'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 13373890
>> latency average: 0.287 ms
>> tps = 222888.311289 (including connections establishing)
>> tps = 222951.470125 (excluding connections establishing)
>> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5434'
>> transaction type: SELECT only
>> scaling factor: 100
>> query mode: simple
>> number of clients: 64
>> number of threads: 64
>> duration: 60 s
>> number of transactions actually processed: 12989816
>> latency average: 0.296 ms
>> tps = 216487.458399 (including connections establishing)
>> tps = 216548.069976 (excluding connections establishing)
>> postgres@pgload05g ~ $
>>
>> Compilation options look almost the same:
>> # RHEL 6
>> CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2
>> # Ubuntu
>> CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -O2
>>
>> Attached are a simple script to deploy the testing environment (PgBouncer should be installed) and pgbouncer config. I could provide any other needed information like backtraces or perf reports or anything else.
>>
>> <pgbouncer.ini>
>> <deploy.sh>
>>
>> --
>> May the force be with you…
>> https://simply.name <https://simply.name/;
>
>
> --
> May the force be with you…
> https://simply.name <https://simply.name/;
--
May the force be with you…
https://simply.name
Attachments:
[application/octet-stream] bisect95.out (5.9K, 3-bisect95.out)
download
[application/octet-stream] bisect96.out (6.0K, 5-bisect96.out)
download
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
@ 2016-06-02 11:18 ` Антон Бушмелев <[email protected]>
2016-06-09 22:28 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
1 sibling, 1 reply; 14+ messages in thread
From: Антон Бушмелев @ 2016-06-02 11:18 UTC (permalink / raw)
To: Vladimir Borodin <[email protected]>; +Cc: Антон Бушмелев <[email protected]>; pgsql-performance
UP. repeat tests on local vm.. reults are discouraging
OS PG TPS AVG latency
Centos 7 9.5.3 23.711023 168.421
Centos 7 9.5.3 26.609271 150.188
Centos 7 9.5.3 25.220044 158.416
Centos 7 9.5.3 25.598977 156.047
Centos 7 9.4.8 278.572191 14.077
Centos 7 9.4.8 247.237755 16.177
Centos 7 9.4.8 240.007524 16.276
Centos 7 9.4.8 237.862238 16.596
ps: latest update on centos 7 +xfs + lates database version from repo, no pgbouncer
> On 25 May 2016, at 17:33, Vladimir Borodin <[email protected]> wrote:
>
> Hi all.
>
> We have found that queries through PgBouncer 1.7.2 (with transaction pooling) to local PostgreSQL are almost two times slower in 9.5.3 than in 9.4.8 on RHEL 6 hosts (all packages are updated to last versions). Meanwhile the problem can’t be reproduced i.e. on Ubuntu 14.04 (also fully-updated).
>
> Here is how the results look like for 9.4, 9.5 and 9.6. All are built from latest commits on yesterday in
> * REL9_4_STABLE (a0cc89a28141595d888d8aba43163d58a1578bfb),
> * REL9_5_STABLE (e504d915bbf352ecfc4ed335af934e799bf01053),
> * master (6ee7fb8244560b7a3f224784b8ad2351107fa55d).
>
> All of them are build on the host where testing is done (with stock gcc versions). Sysctls, pgbouncer config and everything we found are the same, postgres configs are default, PGDATA is in tmpfs. All numbers are reproducible, they are stable between runs.
>
> Shortly:
>
> OS PostgreSQL version TPS Avg. latency
> RHEL 6 9.4 44898 1.425 ms
> RHEL 6 9.5 26199 2.443 ms
> RHEL 6 9.5 43027 1.487 ms
> Ubuntu 14.04 9.4 67458 0.949 ms
> Ubuntu 14.04 9.5 64065 0.999 ms
> Ubuntu 14.04 9.6 64350 0.995 ms
>
> You could see that the difference between major versions on Ubuntu is not significant, but on RHEL 9.5 is 70% slower than 9.4 and 9.6.
>
> Below are more details.
>
> RHEL 6:
>
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg94'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 2693962
> latency average: 1.425 ms
> tps = 44897.461518 (including connections establishing)
> tps = 44898.763258 (excluding connections establishing)
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg95'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 1572014
> latency average: 2.443 ms
> tps = 26198.928627 (including connections establishing)
> tps = 26199.803363 (excluding connections establishing)
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg96'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 2581645
> latency average: 1.487 ms
> tps = 43025.676995 (including connections establishing)
> tps = 43027.038275 (excluding connections establishing)
> postgres@pgload05g ~ $
>
> Ubuntu 14.04 (the same hardware):
>
> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg94'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 4047653
> latency average: 0.949 ms
> tps = 67458.361515 (including connections establishing)
> tps = 67459.983480 (excluding connections establishing)
> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg95'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 3844084
> latency average: 0.999 ms
> tps = 64065.447458 (including connections establishing)
> tps = 64066.943627 (excluding connections establishing)
> postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=6432 dbname=pg96'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 3861088
> latency average: 0.995 ms
> tps = 64348.573126 (including connections establishing)
> tps = 64350.195750 (excluding connections establishing)
> postgres@pgloadpublic02:~$
>
> In both tests (RHEL and Ubuntu) the bottleneck is performance of singe CPU core which is 100% consumed by PgBouncer. If pgbench connects to postgres directly I get the following (expected) numbers:
>
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5432'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 10010710
> latency average: 0.384 ms
> tps = 166835.937859 (including connections establishing)
> tps = 166849.730224 (excluding connections establishing)
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5433'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 13373890
> latency average: 0.287 ms
> tps = 222888.311289 (including connections establishing)
> tps = 222951.470125 (excluding connections establishing)
> postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n 'host=localhost port=5434'
> transaction type: SELECT only
> scaling factor: 100
> query mode: simple
> number of clients: 64
> number of threads: 64
> duration: 60 s
> number of transactions actually processed: 12989816
> latency average: 0.296 ms
> tps = 216487.458399 (including connections establishing)
> tps = 216548.069976 (excluding connections establishing)
> postgres@pgload05g ~ $
>
> Compilation options look almost the same:
> # RHEL 6
> CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2
> # Ubuntu
> CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -O2
>
> Attached are a simple script to deploy the testing environment (PgBouncer should be installed) and pgbouncer config. I could provide any other needed information like backtraces or perf reports or anything else.
>
> <pgbouncer.ini>
> <deploy.sh>
>
> --
> May the force be with you…
> https://simply.name <https://simply.name/;
Attachments:
[application/pkcs7-signature] smime.p7s (2.5K, 3-smime.p7s)
download
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Антон Бушмелев <[email protected]>
@ 2016-06-09 22:28 ` Andres Freund <[email protected]>
2016-06-12 21:42 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Andres Freund @ 2016-06-09 22:28 UTC (permalink / raw)
To: Антон Бушмелев <[email protected]>; +Cc: Vladimir Borodin <[email protected]>; pgsql-performance
Hi,
On 2016-06-02 14:18:26 +0300, Антон Бушмелев wrote:
> UP. repeat tests on local vm.. reults are discouraging
> OS PG TPS AVG latency
> Centos 7 9.5.3 23.711023 168.421
> Centos 7 9.5.3 26.609271 150.188
> Centos 7 9.5.3 25.220044 158.416
> Centos 7 9.5.3 25.598977 156.047
> Centos 7 9.4.8 278.572191 14.077
> Centos 7 9.4.8 247.237755 16.177
> Centos 7 9.4.8 240.007524 16.276
> Centos 7 9.4.8 237.862238 16.596
Could you provide profiles on 9.4 and 9.5? Which kernel did you have
enabled? Is /proc/sys/kernel/sched_autogroup_enabled 1 or 0?
Regards,
Andres
--
Sent via pgsql-performance mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Антон Бушмелев <[email protected]>
2016-06-09 22:28 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
@ 2016-06-12 21:42 ` Vladimir Borodin <[email protected]>
2016-06-12 21:51 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Vladimir Borodin @ 2016-06-12 21:42 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Антон Бушмелев <[email protected]>; pgsql-performance; PostgreSQL-development <[email protected]>
> 10 июня 2016 г., в 1:28, Andres Freund <[email protected]> написал(а):
>
> Hi,
>
> On 2016-06-02 14:18:26 +0300, Антон Бушмелев wrote:
>> UP. repeat tests on local vm.. reults are discouraging
>> OS PG TPS AVG latency
>> Centos 7 9.5.3 23.711023 168.421
>> Centos 7 9.5.3 26.609271 150.188
>> Centos 7 9.5.3 25.220044 158.416
>> Centos 7 9.5.3 25.598977 156.047
>> Centos 7 9.4.8 278.572191 14.077
>> Centos 7 9.4.8 247.237755 16.177
>> Centos 7 9.4.8 240.007524 16.276
>> Centos 7 9.4.8 237.862238 16.596
>
> Could you provide profiles on 9.4 and 9.5? Which kernel did you have
> enabled? Is /proc/sys/kernel/sched_autogroup_enabled 1 or 0?
I don’t know anything about Anton’s installation. I’m having troubles on RHEL 6 with stock kernel (2.6.32-642.el6.x86_64). I also tried a couple of non-official kernels (3.10, 3.19) but results didn’t change much.
/proc/sys/kernel/sched_autogroup_enabled doesn’t change the picture in general for 9.5 or 9.6 but improves for 9.4:
root@pgload05g ~ # cat /proc/sys/kernel/sched_autogroup_enabled
0
root@pgload05g ~ # /tmp/run.sh
RHEL 6 9.4 69163 0.925
RHEL 6 9.5 34495 1.855
RHEL 6 9.6 70631 0.906
root@pgload05g ~ # echo 1 >/proc/sys/kernel/sched_autogroup_enabled
root@pgload05g ~ # /tmp/run.sh
RHEL 6 9.4 82242 0.778
RHEL 6 9.5 34100 1.877
RHEL 6 9.6 70599 0.907
root@pgload05g ~ #
For taking perf profiles I’ve recompiled all versions with CFLAGS='-O2 -fno-omit-frame-pointer’ and issued the following command during pgbench runs:
perf record -g --call-graph=dwarf -a -o pg9?_all.data sleep 10
After run:
perf report -g -i pg9?_all.data >/tmp/pg9?_perf_report.txt
The results from pg9?_perf_report.txt are attached. Note that in all cases some events were lost, i.e.:
root@pgload05g ~ # perf report -g -i pg94_all.data >/tmp/pg94_perf_report.txt
Failed to open [vsyscall], continuing without symbols
Warning:
Processed 537137 events and lost 7846 chunks!
Check IO/CPU overload!
root@pgload05g ~ #
The reason for that is overloaded I/O subsystem.
>
> Regards,
>
> Andres
--
May the force be with you…
https://simply.name
Attachments:
[application/x-gzip] pg96_perf_report.txt.gz (1.4M, 3-pg96_perf_report.txt.gz)
download
[application/x-gzip] pg95_perf_report.txt.gz (1.2M, 5-pg95_perf_report.txt.gz)
download
[application/x-gzip] pg94_perf_report.txt.gz (2.1M, 7-pg94_perf_report.txt.gz)
download
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Антон Бушмелев <[email protected]>
2016-06-09 22:28 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-06-12 21:42 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
@ 2016-06-12 21:51 ` Andres Freund <[email protected]>
2016-06-13 18:58 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Andres Freund @ 2016-06-12 21:51 UTC (permalink / raw)
To: Vladimir Borodin <[email protected]>; +Cc: Антон Бушмелев <[email protected]>; pgsql-performance; PostgreSQL-development <[email protected]>
Hi Vladimir,
Thanks for these reports.
On 2016-06-13 00:42:19 +0300, Vladimir Borodin wrote:
> perf report -g -i pg9?_all.data >/tmp/pg9?_perf_report.txt
Any chance you could redo the reports with --no-children --call-graph=fractal
added? The mode that includes child overheads unfortunately makes the
output hard to interpet/compare.
> The results from pg9?_perf_report.txt are attached. Note that in all cases some events were lost, i.e.:
>
> root@pgload05g ~ # perf report -g -i pg94_all.data >/tmp/pg94_perf_report.txt
> Failed to open [vsyscall], continuing without symbols
> Warning:
> Processed 537137 events and lost 7846 chunks!
You can reduce the overhead by reducing the sampling frequency, e.g. by
specifying -F 300.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Антон Бушмелев <[email protected]>
2016-06-09 22:28 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-06-12 21:42 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-12 21:51 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
@ 2016-06-13 18:58 ` Vladimir Borodin <[email protected]>
2016-07-04 13:30 ` Re: [HACKERS] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-07-14 18:48 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
0 siblings, 2 replies; 14+ messages in thread
From: Vladimir Borodin @ 2016-06-13 18:58 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Антон Бушмелев <[email protected]>; pgsql-performance; PostgreSQL-development <[email protected]>
> 13 июня 2016 г., в 0:51, Andres Freund <[email protected]> написал(а):
>
> Hi Vladimir,
>
> Thanks for these reports.
>
> On 2016-06-13 00:42:19 +0300, Vladimir Borodin wrote:
>> perf report -g -i pg9?_all.data >/tmp/pg9?_perf_report.txt
>
> Any chance you could redo the reports with --no-children --call-graph=fractal
> added? The mode that includes child overheads unfortunately makes the
> output hard to interpet/compare.
Of course. Not sure if that is important but I upgraded perf for that (because --no-children option was introduced in ~3.16), so perf record and perf report were done with different perf versions.
Also I’ve done the same test on same host (RHEL 6) but with 4.6 kernel/perf and writing perf data to /dev/shm for not loosing events. Perf report output is also attached but important thing is that the regression is not so significant:
root@pgload05g ~ # uname -r
4.6.0-1.el6.elrepo.x86_64
root@pgload05g ~ # cat /proc/sys/kernel/sched_autogroup_enabled
1
root@pgload05g ~ # /tmp/run.sh
RHEL 6 9.4 71634 0.893
RHEL 6 9.5 54005 1.185
RHEL 6 9.6 65550 0.976
root@pgload05g ~ # echo 0 >/proc/sys/kernel/sched_autogroup_enabled
root@pgload05g ~ # /tmp/run.sh
RHEL 6 9.4 73041 0.876
RHEL 6 9.5 60105 1.065
RHEL 6 9.6 67984 0.941
root@pgload05g ~ #
>
>> The results from pg9?_perf_report.txt are attached. Note that in all cases some events were lost, i.e.:
>>
>> root@pgload05g ~ # perf report -g -i pg94_all.data >/tmp/pg94_perf_report.txt
>> Failed to open [vsyscall], continuing without symbols
>> Warning:
>> Processed 537137 events and lost 7846 chunks!
>
> You can reduce the overhead by reducing the sampling frequency, e.g. by
> specifying -F 300.
>
> Greetings,
>
> Andres Freund
>
>
> --
> Sent via pgsql-hackers mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
--
May the force be with you…
https://simply.name
Attachments:
[application/x-gzip] pg94_perf_report.txt.gz (366.8K, 3-pg94_perf_report.txt.gz)
download
[application/x-gzip] pg95_perf_report.txt.gz (219.2K, 5-pg95_perf_report.txt.gz)
download
[application/x-gzip] pg96_perf_report.txt.gz (256.8K, 7-pg96_perf_report.txt.gz)
download
[application/x-gzip] pg96_perf_report_4.6.txt.gz (276.6K, 9-pg96_perf_report_4.6.txt.gz)
download
[application/x-gzip] pg95_perf_report_4.6.txt.gz (256.1K, 11-pg95_perf_report_4.6.txt.gz)
download
[application/x-gzip] pg94_perf_report_4.6.txt.gz (299.9K, 13-pg94_perf_report_4.6.txt.gz)
download
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [HACKERS] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Антон Бушмелев <[email protected]>
2016-06-09 22:28 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-06-12 21:42 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-12 21:51 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-06-13 18:58 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
@ 2016-07-04 13:30 ` Vladimir Borodin <[email protected]>
2016-07-14 18:53 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
1 sibling, 1 reply; 14+ messages in thread
From: Vladimir Borodin @ 2016-07-04 13:30 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-performance; PostgreSQL-development <[email protected]>
> 13 июня 2016 г., в 21:58, Vladimir Borodin <[email protected]> написал(а):
>
>>
>> 13 июня 2016 г., в 0:51, Andres Freund <[email protected] <mailto:[email protected]>> написал(а):
>>
>> Hi Vladimir,
>>
>> Thanks for these reports.
>>
>> On 2016-06-13 00:42:19 +0300, Vladimir Borodin wrote:
>>> perf report -g -i pg9?_all.data >/tmp/pg9?_perf_report.txt
>>
>> Any chance you could redo the reports with --no-children --call-graph=fractal
>> added? The mode that includes child overheads unfortunately makes the
>> output hard to interpet/compare.
>
> Of course. Not sure if that is important but I upgraded perf for that (because --no-children option was introduced in ~3.16), so perf record and perf report were done with different perf versions.
>
> <pg94_perf_report.txt.gz>
> <pg95_perf_report.txt.gz>
> <pg96_perf_report.txt.gz>
>
> Also I’ve done the same test on same host (RHEL 6) but with 4.6 kernel/perf and writing perf data to /dev/shm for not loosing events. Perf report output is also attached but important thing is that the regression is not so significant:
>
> root@pgload05g ~ # uname -r
> 4.6.0-1.el6.elrepo.x86_64
> root@pgload05g ~ # cat /proc/sys/kernel/sched_autogroup_enabled
> 1
> root@pgload05g ~ # /tmp/run.sh
> RHEL 6 9.4 71634 0.893
> RHEL 6 9.5 54005 1.185
> RHEL 6 9.6 65550 0.976
> root@pgload05g ~ # echo 0 >/proc/sys/kernel/sched_autogroup_enabled
> root@pgload05g ~ # /tmp/run.sh
> RHEL 6 9.4 73041 0.876
> RHEL 6 9.5 60105 1.065
> RHEL 6 9.6 67984 0.941
> root@pgload05g ~ #
>
> <pg96_perf_report_4.6.txt.gz>
> <pg95_perf_report_4.6.txt.gz>
> <pg94_perf_report_4.6.txt.gz>
Andres, is there any chance that you would find time to look at those results? Are they actually useful?
>
>
>>
>>> The results from pg9?_perf_report.txt are attached. Note that in all cases some events were lost, i.e.:
>>>
>>> root@pgload05g ~ # perf report -g -i pg94_all.data >/tmp/pg94_perf_report.txt
>>> Failed to open [vsyscall], continuing without symbols
>>> Warning:
>>> Processed 537137 events and lost 7846 chunks!
>>
>> You can reduce the overhead by reducing the sampling frequency, e.g. by
>> specifying -F 300.
>>
>> Greetings,
>>
>> Andres Freund
>>
>>
>> --
>> Sent via pgsql-hackers mailing list ([email protected] <mailto:[email protected]>)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-hackers <http://www.postgresql.org/mailpref/pgsql-hackers;
>
>
> --
> May the force be with you…
> https://simply.name <https://simply.name/;
--
May the force be with you…
https://simply.name
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Антон Бушмелев <[email protected]>
2016-06-09 22:28 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-06-12 21:42 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-12 21:51 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-06-13 18:58 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-07-04 13:30 ` Re: [HACKERS] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
@ 2016-07-14 18:53 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 14+ messages in thread
From: Andres Freund @ 2016-07-14 18:53 UTC (permalink / raw)
To: Vladimir Borodin <[email protected]>; +Cc: pgsql-performance; PostgreSQL-development <[email protected]>
On 2016-07-04 16:30:51 +0300, Vladimir Borodin wrote:
>
> > 13 июня 2016 г., в 21:58, Vladimir Borodin <[email protected]> написал(а):
> >
> >>
> >> 13 июня 2016 г., в 0:51, Andres Freund <[email protected] <mailto:[email protected]>> написал(а):
> >>
> >> Hi Vladimir,
> >>
> >> Thanks for these reports.
> >>
> >> On 2016-06-13 00:42:19 +0300, Vladimir Borodin wrote:
> >>> perf report -g -i pg9?_all.data >/tmp/pg9?_perf_report.txt
> >>
> >> Any chance you could redo the reports with --no-children --call-graph=fractal
> >> added? The mode that includes child overheads unfortunately makes the
> >> output hard to interpet/compare.
> >
> > Of course. Not sure if that is important but I upgraded perf for that (because --no-children option was introduced in ~3.16), so perf record and perf report were done with different perf versions.
> >
> > <pg94_perf_report.txt.gz>
> > <pg95_perf_report.txt.gz>
> > <pg96_perf_report.txt.gz>
> >
> > Also I’ve done the same test on same host (RHEL 6) but with 4.6
> > kernel/perf and writing perf data to /dev/shm for not loosing
> > events. Perf report output is also attached but important thing is
> > that the regression is not so significant:
FWIW, you can instead use -F 300 or something to reduce the sampling
frequency.
> > root@pgload05g ~ # uname -r
> > 4.6.0-1.el6.elrepo.x86_64
> > root@pgload05g ~ # cat /proc/sys/kernel/sched_autogroup_enabled
> > 1
> > root@pgload05g ~ # /tmp/run.sh
> > RHEL 6 9.4 71634 0.893
> > RHEL 6 9.5 54005 1.185
> > RHEL 6 9.6 65550 0.976
> > root@pgload05g ~ # echo 0 >/proc/sys/kernel/sched_autogroup_enabled
> > root@pgload05g ~ # /tmp/run.sh
> > RHEL 6 9.4 73041 0.876
> > RHEL 6 9.5 60105 1.065
> > RHEL 6 9.6 67984 0.941
> > root@pgload05g ~ #
> >
> > <pg96_perf_report_4.6.txt.gz>
> > <pg95_perf_report_4.6.txt.gz>
> > <pg94_perf_report_4.6.txt.gz>
>
> Andres, is there any chance that you would find time to look at those results? Are they actually useful?
I don't really see anything suspicious in the profile. This looks more
like a kernel scheduler issue than a postgres bottleneck one. It seems
that somehow using nonblocking IO (started in 9.5) causes scheduling
issues when pgbouncer is also local.
Could you do perf stat -ddd -a sleep 10 or something during both runs? I
suspect that the context switch ratios will be quite different.
Andres
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Антон Бушмелев <[email protected]>
2016-06-09 22:28 ` Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-06-12 21:42 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-06-12 21:51 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Andres Freund <[email protected]>
2016-06-13 18:58 ` Re: [PERFORM] 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
@ 2016-07-14 18:48 ` Andres Freund <[email protected]>
1 sibling, 0 replies; 14+ messages in thread
From: Andres Freund @ 2016-07-14 18:48 UTC (permalink / raw)
To: Vladimir Borodin <[email protected]>; +Cc: Антон Бушмелев <[email protected]>; pgsql-performance; PostgreSQL-development <[email protected]>
On 2016-06-13 21:58:30 +0300, Vladimir Borodin wrote:
>
> > 13 июня 2016 г., в 0:51, Andres Freund <[email protected]> написал(а):
> >
> > Hi Vladimir,
> >
> > Thanks for these reports.
> >
> > On 2016-06-13 00:42:19 +0300, Vladimir Borodin wrote:
> >> perf report -g -i pg9?_all.data >/tmp/pg9?_perf_report.txt
> >
> > Any chance you could redo the reports with --no-children --call-graph=fractal
> > added? The mode that includes child overheads unfortunately makes the
> > output hard to interpet/compare.
>
> Of course. Not sure if that is important but I upgraded perf for that (because --no-children option was introduced in ~3.16), so perf record and perf report were done with different perf versions.
>
>
>
> Also I’ve done the same test on same host (RHEL 6) but with 4.6 kernel/perf and writing perf data to /dev/shm for not loosing events. Perf report output is also attached but important thing is that the regression is not so significant:
>
> root@pgload05g ~ # uname -r
> 4.6.0-1.el6.elrepo.x86_64
> root@pgload05g ~ # cat /proc/sys/kernel/sched_autogroup_enabled
> 1
> root@pgload05g ~ # /tmp/run.sh
> RHEL 6 9.4 71634 0.893
> RHEL 6 9.5 54005 1.185
> RHEL 6 9.6 65550 0.976
> root@pgload05g ~ # echo 0 >/proc/sys/kernel/sched_autogroup_enabled
> root@pgload05g ~ # /tmp/run.sh
> RHEL 6 9.4 73041 0.876
> RHEL 6 9.5 60105 1.065
> RHEL 6 9.6 67984 0.941
> root@pgload05g ~ #
Hm. Have you measured how large the slowdown is if you connect via tcp
to pgbouncer, but have pgbouncer connect to postgres via unix sockets?
Andres
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 14+ messages in thread
end of thread, other threads:[~2016-07-14 18:53 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-05-25 14:33 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 Vladimir Borodin <[email protected]>
2016-05-27 16:57 ` Vladimir Borodin <[email protected]>
2016-05-27 21:56 ` Andres Freund <[email protected]>
2016-05-30 14:53 ` Vladimir Borodin <[email protected]>
2016-05-30 17:03 ` Vladimir Borodin <[email protected]>
2016-05-31 09:06 ` Vladimir Borodin <[email protected]>
2016-06-02 11:18 ` Антон Бушмелев <[email protected]>
2016-06-09 22:28 ` Andres Freund <[email protected]>
2016-06-12 21:42 ` Vladimir Borodin <[email protected]>
2016-06-12 21:51 ` Andres Freund <[email protected]>
2016-06-13 18:58 ` Vladimir Borodin <[email protected]>
2016-07-04 13:30 ` Vladimir Borodin <[email protected]>
2016-07-14 18:53 ` Andres Freund <[email protected]>
2016-07-14 18:48 ` Andres Freund <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox