Subject: Re: [External] : Re: BUG #17005: Enhancement request: Improve
 walsender throughput by aggregating multiple messages in one send
To: Andres Freund <andres@anarazel.de>,
        Yura Sokolov <y.sokolov@postgrespro.ru>
Cc: pgsql-bugs@lists.postgresql.org
References: <17005-3e1030784d5440c4@postgresql.org>
 <20210516224413.xitbilinlyrlt42m@alap3.anarazel.de>
 <aae9709fddecd6f6b59405c62a861dae@postgrespro.ru>
 <20210517162753.lmhmnfd72bn3nfms@alap3.anarazel.de>
From: Rony Kurniawan <rony.kurniawan@oracle.com>
Message-ID: <3f60a4c4-71df-bce0-d743-ec06ff1b08fe@oracle.com>
Date: Mon, 17 May 2021 11:19:31 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.10.1
In-Reply-To: <20210517162753.lmhmnfd72bn3nfms@alap3.anarazel.de>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from [IPv6:2601:643:c100:76b0:f552:f87f:10a3:ad33]
 (2601:643:c100:76b0:f552:f87f:10a3:ad33) by
 BY5PR17CA0009.namprd17.prod.outlook.com (2603:10b6:a03:1b8::22) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.32 via Frontend
 Transport; Mon, 17 May 2021 18:19:33 +0000
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: a00e4cd1-4079-44c0-6117-08d919605244
X-MS-TrafficTypeDiagnostic: BY5PR10MB3778:
X-Microsoft-Antispam-PRVS: 
	<BY5PR10MB37787178A0A66AC2EC5B8F51F52D9@BY5PR10MB3778.namprd10.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:9508;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 
	1jQlkyTvl0EFmM9QQQXzMOPM6QGgwqZ5BJgrY+L8aM29Lxkf3boO197hRBphU9Zry4QeDLjF/rZMerI0DdG6SDaUrjOrtGAjPXlPd68rJhgk8q7DLatuTWALTLANvXcQ5r6s/kpVUStj94mk7LaRAlizGpXI4uxuhScZsySwrFYoCrNSKRpgiTECG197/jpJrpAluyCwEIoOX4wFxl4mfLit3+Jr1EGCO+lMp/SxBPHPXdQ1jwFQzmj8BVVUxOzBYx9uVsPx93velX/Nd0AWKngl0ZHEm7ny9Wk5G80xVHZ87Gr4FH+drfR14+ZJVNcEkJz40eoY84SeMlcWu3tbmVYK2WdVH1HLOiGXV16pjdRKlQOChsescVPM5dIme9H+ffyQqzZh0zOrbLvji1gUjqgCMRv/FSLWnlk/EkrGjWkLaKls3rfodYhx0RXQc/7+AoWuv2n5tXM8BFpcyO54r4IHPtQybruv9hqfd+BldIZomb5dlj0jc5vCG0oNPZ3O+Dl4/5r43CgZ1kJRlslNl6B4wNgFZmyNXAtpIHUrJvCrPr08ts89R9Zo+Q6EVYgMz2YY4RSlQVLi6zImGPL3q2+8zY6nbZ+kCTkF1/0NAgFDjgRWwAkjTYsubfGXQC/IUSN6f+GRy/ZZt6/yr9vhdviHRRGpBtxc5lN5MhQ6Z0D+ffv1VPfKif/NKZPwoyN+
X-Forefront-Antispam-Report: 
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR10MB4446.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(136003)(396003)(376002)(346002)(366004)(6486002)(8936002)(83380400001)(53546011)(8676002)(110136005)(31696002)(44832011)(2616005)(66556008)(38100700002)(36756003)(5660300002)(2906002)(31686004)(4326008)(15650500001)(186003)(16526019)(66946007)(478600001)(86362001)(66476007)(316002)(43740500002)(45980500001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData: 
	=?utf-8?B?SkN4STkyV3pvY1ZXQUhuQW1zb3JyNlZBaEgraExvSGo0TWNMVFdFSlNxVmR2?=
 =?utf-8?B?cm40OTdiKytDSjZnN2t5UnZuODhsUGxvOFVhMGw4dm82dENRN2pDaEdFR2hH?=
 =?utf-8?B?VTVSQUpsSmxpVkwyUWk0dzJ5dldPMWxRalFvN00rNUEvL2p1MGJyamQ2OEMy?=
 =?utf-8?B?a055S1pPWE90VzZZZ2FoUGJUZ0RjSUR2c1dKVDJObExPMHBXYmNXNjhnYlVa?=
 =?utf-8?B?L2JIUFlNY1hGckM5RHZ3OGRyVkhQV2EvdnNvSkEwL3JqNnRHMnVGTmdOdVc0?=
 =?utf-8?B?YmhPR2ZuVFNVLzNiai9IblUzVXljdE1FdkFNbmpMQnFQTEtxVWl1QWxVTkds?=
 =?utf-8?B?K1BLNTloTkg5WlFYbjhLeTZlUEMwalVQZWFiSVlGcGM3TFl5azVEd2k2dEo1?=
 =?utf-8?B?VmpnaTVFd0ZETEVIbW40MitmZzk1WXIwbVd4RmU1cFgyOEZyMjY5bmpQLzhN?=
 =?utf-8?B?ZE9uUXdyUXp3azY2RTU1azBDMkJwaGFuSHptb05wTVJ1NzIyNHBVNUFTZENz?=
 =?utf-8?B?cUlmS1VxU2ZQSUNocEp6dzZFd3QvSlZGbGszeldlYXdTbEsrWHF5eVg1MXBt?=
 =?utf-8?B?YWlDMnBZdEVPeThIVU5XUXhWZWdEQ1d5S1YwaFBIQWlIRkR4Ym82SHpobXY2?=
 =?utf-8?B?T2hKU1FyZlFTcmgwbDRyL3hWQTRkSzVpVWtyd01pdmlIMXM5K3lKTVROcG8x?=
 =?utf-8?B?QnVxMmxvSDZVRzFManV0bzN5VG9SU2hYVnk3Y2hSMmxwSDdnSzJoOWsyakpR?=
 =?utf-8?B?SXVqa0xEWUsyYjhWNmFIQTlBVDI5NnRWM3hPZG5yVCtvbVloTXBaaWdlU1R2?=
 =?utf-8?B?d0JmbnpxaFVaQjZiRFJSd0k5R1pIeVB1b1RpZlBEN0NMam5RbmFXa3l6aDNy?=
 =?utf-8?B?M0xISjBIL0FVWGNCVVZxRHp3S0dWVGlYNS9haTdOaUZ5SVo0TWx5d2N1MlJr?=
 =?utf-8?B?b0k2K1lGSGxUUWtJejVUNU5NcDdpMEpTU0E3bUJaTGRGUEVwNXZUUWp0QTE4?=
 =?utf-8?B?SkV4cEw2bWY1TUJZYlNTWFlZZzZVa0JWMk1oemgvNTY2SkhNQTZETk1KcEJo?=
 =?utf-8?B?cnMxSzhIZW84bjk5bGdlUWVFeWNKNS8rSVI3eFdoV3ZZamhTaXBWMkU3bjBy?=
 =?utf-8?B?UDBuRXpOL1pyVE8xUW8xUWE5azhacHNTaFJjaTU1K21IK20xYVNaZWxZdGRz?=
 =?utf-8?B?U2xhcFJwR3RDblpvODV4bDBuQkZNVjZNQXg0NHVyMG5oQnZzTDJYeUc5NUgw?=
 =?utf-8?B?VXB4V0JOSlcvV0drdHpwVWhmQkdSZk9wa2VFK3pFNUgzRThkVjRUclZtREJX?=
 =?utf-8?B?SFkwS3huNVVRNE1MbnBCWkwxKzlhenNnSmhJTko2TlRnZTQyR1dHc2NPWmpr?=
 =?utf-8?B?V3pRS0hQWGVQZHN3d0k3VHZIZmZiV2J3c2lUZ0JUUzlhdlAxOGd3dHQzbVpj?=
 =?utf-8?B?aGFDTFpndmY4M3N2b2dMWUhQbU1DNFp0RU4yV0doNXBBK3hsMXZMRFpQTVgz?=
 =?utf-8?B?MWgyanQvZ2FpSnJFOHhuekQrUEJwZ3BtNUNTYWR3UUQ2amEwQW5lUXBaWFRV?=
 =?utf-8?B?TDE5QWlXU2MzSWZlN1hWRTlCTmpoZ093N3NyT3J2cWhMMU1IdFNHd3oxYlpD?=
 =?utf-8?B?akpMTmltSy9iWFBHV2JpbUJ5UndUZU95Tkt1SURMbVE5WFBkK2Rrc2JWamlr?=
 =?utf-8?B?L0tyVTZVQkxnM3JrSEdIWDZteEl2cFp0SmoxT0RwdmNaVlZVUjdDY0VVTzNZ?=
 =?utf-8?B?UTNPdnpsVDdLbmQ3UlNmOEc4VVdjWEl4bjdQVFB1bXRkNWZBdkZzQ1RNZTVi?=
 =?utf-8?B?dlVPN2hHL1V2akZaSFpoYTVWdFhocUVHeHVOWVJHaUpYeGZqNllFYjJQMGww?=
 =?utf-8?Q?Kmq2OrwhJmOvh?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 
 a00e4cd1-4079-44c0-6117-08d919605244
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4446.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2021 18:19:33.9066
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 
 T1t59vdAJ2F1rrq7QKWJ0/ne0pslkeJ22Fjc5dDtOcfDV26VckIoK06rL2/PZYanHeLy5dJ5t0g81mtvT3x0NzLq5AAWQkTzRu/FGgYVG38=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3778
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9987
 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0
 malwarescore=0 spamscore=0
 phishscore=0 bulkscore=0 suspectscore=0 mlxlogscore=591 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000
 definitions=main-2105170127
X-Proofpoint-ORIG-GUID: 7bM9qcHE1wwqfSS8zmyQJOyEpfpxQi_c
X-Proofpoint-GUID: 7bM9qcHE1wwqfSS8zmyQJOyEpfpxQi_c
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9987
 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0
 lowpriorityscore=0 malwarescore=0
 spamscore=0 priorityscore=1501 suspectscore=0 mlxlogscore=844 mlxscore=0
 impostorscore=0 adultscore=0 clxscore=1011 phishscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000
 definitions=main-2105170127
List-Id: <pgsql-bugs.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-bugs@lists.postgresql.org>
List-Owner: <mailto:pgsql-bugs-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-bugs>
Archived-At: 
 <https://www.postgresql.org/message-id/3f60a4c4-71df-bce0-d743-ec06ff1b08fe%40oracle.com>
Precedence: bulk

Hi,

On 5/17/2021 9:27 AM, Andres Freund wrote:
> Hi,
>
> On 2021-05-17 16:21:36 +0300, Yura Sokolov wrote:
>> Andres Freund писал 2021-05-17 01:44:
>>> What kind of network is this? I would have expected that if the network
>>> can't keep up the small sends would end up getting aggregated into
>>> larger packets anyway? Are you hitting a PPS limit due to the small
>>> packages, but not yet the throughput limit?
>> I believe the reason is more in sys-call and kernel cpu time overhead than
>> in network throughput. Especially in this "after meltdown+spectre" time.
> Well, enabling Nagle's wouldn't change anything if the issue is just
> syscall and not network overhead. Yet the ask was to enable Nagle's...
>
> Greetings,
>
> Andres Freun

The networks that I tested were gigabits and docker (local). With 
TCP_NODELAY enabled, the only time small sends would be aggregated is by 
auto corking in tcp/ip when there is network congestion. But as you can 
see from the tcpdump output the messages are in individual packet 
therefore there is no aggregation and no network congestion.

There is network overhead in both sender and receiver like tcp/ip 
header, number of skb, ethernet tx/rx descriptors, and interrupts. Also 
syscall overhead in pg_recvlogical where for one insert in the example 
requires 3 recv() calls to read BEGIN, INSERT, COMMIT messages instead 
of one recv() to read all three messages when Nagle's is enabled. This 
syscall overhead is the same in transaction case with multiple changes 
where each change is one recv().

I agree that in some cases low latency in replication is required, but 
there are also cases where high throughput is desired especially when 
the standby server is behind due to outage where latency doesn't exist.

I experimented by simply disabling TCP_NODELAY in 
walsender.c:StartLogicalReplication() and the throughput went up by 60%. 
This is just a prove of concept that some kind of message aggregations 
would result to higher throughput.

Thank you,

Rony