pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
5+ messages / 4 participants
[nested] [flat]
* BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
@ 2021-05-13 00:31 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 5+ messages in thread
From: PG Bug reporting form @ 2021-05-13 00:31 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: rony.kurniawan@oracle.com
The following bug has been logged on the website:
Bug reference: 17005
Logged by: Rony Kurniawan
Email address: rony.kurniawan@oracle.com
PostgreSQL version: 11.7
Operating system: Oracle Linux Server release 7.9
Description:
Hi,
I measured the throughput of reading the logical replication slot and found
that in smaller row size (512 bytes) the throughput is 50% lower compared to
1024 bytes.
tcpdump shows that ethernet packets sent by the replication server contain
only one message per packet (see tcpdump output below).
May be this is the intended design to achieve low latency but this is not
favorable in application that requires high throughput.
Is it possible for PostgreSQL to enable Nagle's algorithm on the streaming
socket for replication?
Or aggegate the messages manually before sending them in one send()?
Thank you,
Rony
test case:
client and server are on different machines or run the server in a docker.
create table public.test (id integer generated always as identity, name
varchar(512));
alter table public.test replica identity full;
select * from pg_create_logical_replication_slot('testslot',
'test_decoding');
insert into public.test (name) values (rpad('a', 512, 'a'));
...
insert into public.test (name) values (rpad('a', 512, 'a'));
I used pgbench to insert million of records to the test table to measure the
throughput, but one insert is enough to show how the server send the
message.
client terminal 1:
$ sudo tcpdump -D
1.enp0s3
2.virbr0
3.docker0
$ sudo tcpdump -i 3 -w psql.pcap "tcp port 5432"
client terminal 2:
$ pg_recvlogical --start --slot=testslot -d postgres -h 172.17.0.2 -U
postgres -f -
client terminal 1:
$ sudo tcpdump -i 3 -w psql.pcap "tcp port 5432"
ctrl-c
37 packets captured
37 packets received by filter
0 packets dropped by kernel
$ tcpdump --number -nn -A -r psql.pcap
...
22 16:38:37.217677 IP 172.17.0.1.56140 > 172.17.0.2.5432:
...START_REPLICATION SLOT "testslot" LOGICAL 0/0.
...
28 16:38:37.218209 IP 172.17.0.2.5432 > 172.17.0.1.56140: ...BEGIN
1888650
...
30 16:38:37.218332 IP 172.17.0.2.5432 > 172.17.0.1.56140: ...table
public.test: INSERT: id[integer]: 1 name[character
varying]:'aaa...512...aaa'
31 16:38:37.218345 IP 172.17.0.2.5432 > 172.17.0.1.56140: ...COMMIT
1888650
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
@ 2021-05-16 22:44 Andres Freund <andres@anarazel.de>
parent: PG Bug reporting form <noreply@postgresql.org>
0 siblings, 2 replies; 5+ messages in thread
From: Andres Freund @ 2021-05-16 22:44 UTC (permalink / raw)
To: rony.kurniawan@oracle.com; pgsql-bugs@lists.postgresql.org
Hi,
On 2021-05-13 00:31:53 +0000, PG Bug reporting form wrote:
> I measured the throughput of reading the logical replication slot and found
> that in smaller row size (512 bytes) the throughput is 50% lower compared to
> 1024 bytes.
Huh, that is interesting.
> tcpdump shows that ethernet packets sent by the replication server contain
> only one message per packet (see tcpdump output below).
> May be this is the intended design to achieve low latency but this is not
> favorable in application that requires high throughput.
What kind of network is this? I would have expected that if the network
can't keep up the small sends would end up getting aggregated into
larger packets anyway? Are you hitting a PPS limit due to the small
packages, but not yet the throughput limit?
> Is it possible for PostgreSQL to enable Nagle's algorithm on the streaming
> socket for replication?
> Or aggegate the messages manually before sending them in one send()?
I think we can probably do better in cases a transaction is more than a
single change - but I don't think either enabling nagle's or aggregation
are really an option in case of single row transactions. The latency
impact for some scenarios seems too high.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
@ 2021-05-17 13:21 Yura Sokolov <y.sokolov@postgrespro.ru>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 5+ messages in thread
From: Yura Sokolov @ 2021-05-17 13:21 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: rony.kurniawan@oracle.com; pgsql-bugs@lists.postgresql.org
Andres Freund писал 2021-05-17 01:44:
> Hi,
>
> On 2021-05-13 00:31:53 +0000, PG Bug reporting form wrote:
>> I measured the throughput of reading the logical replication slot and
>> found
>> that in smaller row size (512 bytes) the throughput is 50% lower
>> compared to
>> 1024 bytes.
>
> Huh, that is interesting.
>
>
>> tcpdump shows that ethernet packets sent by the replication server
>> contain
>> only one message per packet (see tcpdump output below).
>> May be this is the intended design to achieve low latency but this is
>> not
>> favorable in application that requires high throughput.
>
> What kind of network is this? I would have expected that if the network
> can't keep up the small sends would end up getting aggregated into
> larger packets anyway? Are you hitting a PPS limit due to the small
> packages, but not yet the throughput limit?
I believe the reason is more in sys-call and kernel cpu time overhead
than in
network throughput. Especially in this "after meltdown+spectre" time.
>
>> Is it possible for PostgreSQL to enable Nagle's algorithm on the
>> streaming
>> socket for replication?
>> Or aggegate the messages manually before sending them in one send()?
>
> I think we can probably do better in cases a transaction is more than a
> single change - but I don't think either enabling nagle's or
> aggregation
> are really an option in case of single row transactions. The latency
> impact for some scenarios seems too high.
We have commit_siblings and commit_delay options. Probably similar could
be introduced for replication slot?
regards
Yura
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
@ 2021-05-17 16:27 Andres Freund <andres@anarazel.de>
parent: Yura Sokolov <y.sokolov@postgrespro.ru>
0 siblings, 0 replies; 5+ messages in thread
From: Andres Freund @ 2021-05-17 16:27 UTC (permalink / raw)
To: Yura Sokolov <y.sokolov@postgrespro.ru>; +Cc: rony.kurniawan@oracle.com; pgsql-bugs@lists.postgresql.org
Hi,
On 2021-05-17 16:21:36 +0300, Yura Sokolov wrote:
> Andres Freund писал 2021-05-17 01:44:
> > What kind of network is this? I would have expected that if the network
> > can't keep up the small sends would end up getting aggregated into
> > larger packets anyway? Are you hitting a PPS limit due to the small
> > packages, but not yet the throughput limit?
>
> I believe the reason is more in sys-call and kernel cpu time overhead than
> in network throughput. Especially in this "after meltdown+spectre" time.
Well, enabling Nagle's wouldn't change anything if the issue is just
syscall and not network overhead. Yet the ask was to enable Nagle's...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
@ 2021-05-22 05:39 Amit Kapila <amit.kapila16@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 5+ messages in thread
From: Amit Kapila @ 2021-05-22 05:39 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: rony.kurniawan@oracle.com, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, May 17, 2021 at 4:14 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2021-05-13 00:31:53 +0000, PG Bug reporting form wrote:
> > I measured the throughput of reading the logical replication slot and found
> > that in smaller row size (512 bytes) the throughput is 50% lower compared to
> > 1024 bytes.
>
> Huh, that is interesting.
>
>
> > tcpdump shows that ethernet packets sent by the replication server contain
> > only one message per packet (see tcpdump output below).
> > May be this is the intended design to achieve low latency but this is not
> > favorable in application that requires high throughput.
>
> What kind of network is this? I would have expected that if the network
> can't keep up the small sends would end up getting aggregated into
> larger packets anyway? Are you hitting a PPS limit due to the small
> packages, but not yet the throughput limit?
>
>
> > Is it possible for PostgreSQL to enable Nagle's algorithm on the streaming
> > socket for replication?
> > Or aggegate the messages manually before sending them in one send()?
>
> I think we can probably do better in cases a transaction is more than a
> single change - but I don't think either enabling nagle's or aggregation
> are really an option in case of single row transactions. The latency
> impact for some scenarios seems too high.
>
Can we think of combining Begin single_change Commit and send them as
one message for a single-row change xacts?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2021-05-22 05:39 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-05-13 00:31 BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send PG Bug reporting form <noreply@postgresql.org>
2021-05-16 22:44 ` Andres Freund <andres@anarazel.de>
2021-05-17 13:21 ` Yura Sokolov <y.sokolov@postgrespro.ru>
2021-05-17 16:27 ` Andres Freund <andres@anarazel.de>
2021-05-22 05:39 ` Amit Kapila <amit.kapila16@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox