pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Re: [External] : Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
3+ messages / 2 participants
[nested] [flat]

* Re: [External] : Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
@ 2021-05-17 18:19  Rony Kurniawan <rony.kurniawan@oracle.com>
  0 siblings, 1 reply; 3+ messages in thread

From: Rony Kurniawan @ 2021-05-17 18:19 UTC (permalink / raw)
  To: Andres Freund <andres@anarazel.de>; Yura Sokolov <y.sokolov@postgrespro.ru>; +Cc: pgsql-bugs@lists.postgresql.org

Hi,

On 5/17/2021 9:27 AM, Andres Freund wrote:
> Hi,
>
> On 2021-05-17 16:21:36 +0300, Yura Sokolov wrote:
>> Andres Freund писал 2021-05-17 01:44:
>>> What kind of network is this? I would have expected that if the network
>>> can't keep up the small sends would end up getting aggregated into
>>> larger packets anyway? Are you hitting a PPS limit due to the small
>>> packages, but not yet the throughput limit?
>> I believe the reason is more in sys-call and kernel cpu time overhead than
>> in network throughput. Especially in this "after meltdown+spectre" time.
> Well, enabling Nagle's wouldn't change anything if the issue is just
> syscall and not network overhead. Yet the ask was to enable Nagle's...
>
> Greetings,
>
> Andres Freun

The networks that I tested were gigabits and docker (local). With 
TCP_NODELAY enabled, the only time small sends would be aggregated is by 
auto corking in tcp/ip when there is network congestion. But as you can 
see from the tcpdump output the messages are in individual packet 
therefore there is no aggregation and no network congestion.

There is network overhead in both sender and receiver like tcp/ip 
header, number of skb, ethernet tx/rx descriptors, and interrupts. Also 
syscall overhead in pg_recvlogical where for one insert in the example 
requires 3 recv() calls to read BEGIN, INSERT, COMMIT messages instead 
of one recv() to read all three messages when Nagle's is enabled. This 
syscall overhead is the same in transaction case with multiple changes 
where each change is one recv().

I agree that in some cases low latency in replication is required, but 
there are also cases where high throughput is desired especially when 
the standby server is behind due to outage where latency doesn't exist.

I experimented by simply disabling TCP_NODELAY in 
walsender.c:StartLogicalReplication() and the throughput went up by 60%. 
This is just a prove of concept that some kind of message aggregations 
would result to higher throughput.

Thank you,

Rony






^ permalink  raw  reply  [nested|flat] 3+ messages in thread

* Re: [External] : Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
@ 2021-05-17 18:54  Andres Freund <andres@anarazel.de>
  parent: Rony Kurniawan <rony.kurniawan@oracle.com>
  0 siblings, 1 reply; 3+ messages in thread

From: Andres Freund @ 2021-05-17 18:54 UTC (permalink / raw)
  To: Rony Kurniawan <rony.kurniawan@oracle.com>; +Cc: Yura Sokolov <y.sokolov@postgrespro.ru>; pgsql-bugs@lists.postgresql.org

Hi,

On 2021-05-17 11:19:31 -0700, Rony Kurniawan wrote:
> The networks that I tested were gigabits and docker (local). With
> TCP_NODELAY enabled, the only time small sends would be aggregated is by
> auto corking in tcp/ip when there is network congestion. But as you can see
> from the tcpdump output the messages are in individual packet therefore
> there is no aggregation and no network congestion.

I don't understand why "individual packages" implies that there can be
no network congestion? Or are you just saying that in the specific
period traced you didn't observe that?

I just verified this with iperf - I see large packets with
iperf -l 500 --nodelay -c $other_host
but not
iperf -b 10M -l 500 --nodelay -c $other_host

I had to remember how to disable tcp segmentation offloading to see
proper package sizes in the first case, without there were a lot of
65226 byte sized packets in the first case...


> There is network overhead in both sender and receiver like tcp/ip header,
> number of skb, ethernet tx/rx descriptors, and interrupts.

Right.


> Also syscall overhead in pg_recvlogical where for one insert in the
> example requires 3 recv() calls to read BEGIN, INSERT, COMMIT messages
> instead of one recv() to read all three messages when Nagle's is
> enabled. This syscall overhead is the same in transaction case with
> multiple changes where each change is one recv().

I think the obvious and unproblematic improvement is to only send data
to the socket if WalSndWriteData's last_write parameter is set, or if
there's a certain amount of data in the socket. That'll only get rid of
some of the overhead, since we'd still send things like transactions
separately.

Another improvement might be that WalSndWriteData() possibly shouldn't
block even if pq_is_send_pending() and the pending amount isn't huge,
iff !last_write. That way we'd end up doing syscalls sending more data
at once.

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 3+ messages in thread

* Re: [External] : Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send
@ 2021-05-17 22:45  Rony Kurniawan <rony.kurniawan@oracle.com>
  parent: Andres Freund <andres@anarazel.de>
  0 siblings, 0 replies; 3+ messages in thread

From: Rony Kurniawan @ 2021-05-17 22:45 UTC (permalink / raw)
  To: Andres Freund <andres@anarazel.de>; +Cc: Yura Sokolov <y.sokolov@postgrespro.ru>; pgsql-bugs@lists.postgresql.org


On 5/17/2021 11:54 AM, Andres Freund wrote:
> Hi,
>
> On 2021-05-17 11:19:31 -0700, Rony Kurniawan wrote:
>> The networks that I tested were gigabits and docker (local). With
>> TCP_NODELAY enabled, the only time small sends would be aggregated is by
>> auto corking in tcp/ip when there is network congestion. But as you can see
>> from the tcpdump output the messages are in individual packet therefore
>> there is no aggregation and no network congestion.
> I don't understand why "individual packages" implies that there can be
> no network congestion? Or are you just saying that in the specific
> period traced you didn't observe that?

Since TCP_NODELAY=0 in PosgreSQL then it is up to the kernel to 
aggregate those sends. In case of auto corking, it happens when the NIC 
has outstanding packet in the tx queue due to network congestion or the 
NIC can not catch up with the amount of send() by the application.

On a gigabit ethernet, the amount of data produced by the logical 
replication server is not enough to trigger auto corking or other 
aggregation hence the individual packet per message. Although, 
aggregation could still happened sometimes.

In my bigger test case using pgbench to insert 20 records/transaction 
for 1 minute, I see some bigger packets but they are mostly 629 bytes.

> I just verified this with iperf - I see large packets with
> iperf -l 500 --nodelay -c $other_host
> but not
> iperf -b 10M -l 500 --nodelay -c $other_host
>
> I had to remember how to disable tcp segmentation offloading to see
> proper package sizes in the first case, without there were a lot of
> 65226 byte sized packets in the first case...
>
>> There is network overhead in both sender and receiver like tcp/ip header,
>> number of skb, ethernet tx/rx descriptors, and interrupts.
> Right.
>
>
>> Also syscall overhead in pg_recvlogical where for one insert in the
>> example requires 3 recv() calls to read BEGIN, INSERT, COMMIT messages
>> instead of one recv() to read all three messages when Nagle's is
>> enabled. This syscall overhead is the same in transaction case with
>> multiple changes where each change is one recv().
> I think the obvious and unproblematic improvement is to only send data
> to the socket if WalSndWriteData's last_write parameter is set, or if
> there's a certain amount of data in the socket. That'll only get rid of
> some of the overhead, since we'd still send things like transactions
> separately.
>
> Another improvement might be that WalSndWriteData() possibly shouldn't
> block even if pq_is_send_pending() and the pending amount isn't huge,
> iff !last_write. That way we'd end up doing syscalls sending more data
> at once.

Thank you for looking into this,

Rony






^ permalink  raw  reply  [nested|flat] 3+ messages in thread


end of thread, other threads:[~2021-05-17 22:45 UTC | newest]

Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-05-17 18:19 Re: [External] : Re: BUG #17005: Enhancement request: Improve walsender throughput by aggregating multiple messages in one send Rony Kurniawan <rony.kurniawan@oracle.com>
2021-05-17 18:54 ` Andres Freund <andres@anarazel.de>
2021-05-17 22:45   ` Rony Kurniawan <rony.kurniawan@oracle.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox