pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Re: invalid alloc size error possible in shm_mq
6+ messages / 4 participants
[nested] [flat]

* Re: invalid alloc size error possible in shm_mq
@ 2020-08-25 10:00  Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Peter Eisentraut @ 2020-08-25 10:00 UTC (permalink / raw)
  To: Markus Wanner <markus.wanner@2ndquadrant.com>; pgsql-bugs@postgresql.org, Robert Haas <robertmhaas@gmail.com>

On 2020-07-07 12:30, Markus Wanner wrote:
> in shm_mq_receive of shm_mq.c, a huge payload may trigger an unjustified
> "invalid memory alloc request size" error due to the way the buffer size
> is increased.
> 
> I understand that this fine piece of infrastructure may not have been
> designed for huge payloads.  I still argue it should better not fail
> that way for payloads between half a gigabyte and MaxAllocSize.  The
> attached patch adds an appropriate assertion (documenting the upper
> limit) and avoids the error by limiting the allocation size to
> MaxAllocSize (to correct for such huge payloads below the limit).

I wonder if the assertion is appropriate or whether it should be a full 
error check.  Is anything on the sending side ensuring that the maximum 
size is kept?  All the size variables are Size/size_t so could be much 
larger than MaxAllocSize.

Robert, thoughts?

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: invalid alloc size error possible in shm_mq
@ 2020-09-09 12:37  Markus Wanner <markus.wanner@2ndquadrant.com>
  parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Markus Wanner @ 2020-09-09 12:37 UTC (permalink / raw)
  To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Markus Wanner <markus.wanner@2ndquadrant.com>; pgsql-bugs@postgresql.org, Robert Haas <robertmhaas@gmail.com>

On 8/25/20 12:00 PM, Peter Eisentraut wrote:
> I wonder if the assertion is appropriate or whether it should be a full 
> error check.

Good point.  Originally, it used to be an error.  With the patch (but 
w/o assertions enabled) it could result in a buffer overrun.  Not good.

I changed the patch to add an error (instead of just an assert) when 
asked to read a message larger than MaxAllocSize.  So this patch 
essentially corrects handling of messages in size between MaxAllocSize/2 
and MaxAllocSize.

> Is anything on the sending side ensuring that the maximum 
> size is kept?  All the size variables are Size/size_t so could be much 
> larger than MaxAllocSize.

In this v2 of the patch, I added a check that errors out on the sender 
side as well (for trying to send a message larger than MaxAllocSize).

I'd still recommend to back-patch this.

-- 
Markus Wanner
Senior PostgreSQL Developer
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/

Attachments:

shm_mq_inv_allocation_fix_v2.diff (text/x-patch, 1.8K) download diff --git a/src/backend/storage/ipc/shm_mq.c b/src/backend/storage/ipc/shm_mq.c index 6f40fb165d..262faecf13 100644 --- a/src/backend/storage/ipc/shm_mq.c +++ b/src/backend/storage/ipc/shm_mq.c @@ -24,6 +24,7 @@ #include "storage/procsignal.h" #include "storage/shm_mq.h" #include "storage/spin.h" +#include "utils/memutils.h" /* * This structure represents the actual queue, stored in shared memory. @@ -360,6 +361,13 @@ shm_mq_sendv(shm_mq_handle *mqh, shm_mq_iovec *iov, int iovcnt, bool nowait) for (i = 0; i < iovcnt; ++i) nbytes += iov[i].len; + /* Prevent writing messages overwhelming the receiver. */ + if (nbytes > MaxAllocSize) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + errmsg("cannot send a message of size %zu via shared memory queue", + nbytes))); + /* Try to write, or finish writing, the length word into the buffer. */ while (!mqh->mqh_length_word_complete) { @@ -675,6 +683,17 @@ shm_mq_receive(shm_mq_handle *mqh, Size *nbytesp, void **datap, bool nowait) } nbytes = mqh->mqh_expected_bytes; + /* + * Should be disallowed on the sending side already, but better check + * and error out on the receiver side as well, rather than trying to + * read a prohibitively large message. + */ + if (nbytes > MaxAllocSize) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + errmsg("invalid message size %zu in shared memory queue", + nbytes))); + if (mqh->mqh_partial_bytes == 0) { /* @@ -703,8 +722,13 @@ shm_mq_receive(shm_mq_handle *mqh, Size *nbytesp, void **datap, bool nowait) { Size newbuflen = Max(mqh->mqh_buflen, MQH_INITIAL_BUFSIZE); + /* + * Double the buffer size until the payload fits, but limit to + * MaxAllocSize. + */ while (newbuflen < nbytes) newbuflen *= 2; + newbuflen = Min(newbuflen, MaxAllocSize); if (mqh->mqh_buffer != NULL) {
^ permalink raw reply [nested|flat] 6+ messages in thread

* Re: invalid alloc size error possible in shm_mq
@ 2020-10-19 19:31  Robert Haas <robertmhaas@gmail.com>
  parent: Markus Wanner <markus.wanner@2ndquadrant.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Robert Haas @ 2020-10-19 19:31 UTC (permalink / raw)
  To: Markus Wanner <markus.wanner@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-bugs

On Wed, Sep 9, 2020 at 8:37 AM Markus Wanner
<markus.wanner@2ndquadrant.com> wrote:
> I'd still recommend to back-patch this.

+1. Sorry for not seeing this thread sooner.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: invalid alloc size error possible in shm_mq
@ 2020-10-20 01:52  Michael Paquier <michael@paquier.xyz>
  parent: Robert Haas <robertmhaas@gmail.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Michael Paquier @ 2020-10-20 01:52 UTC (permalink / raw)
  To: Robert Haas <robertmhaas@gmail.com>; +Cc: Markus Wanner <markus.wanner@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-bugs

On Mon, Oct 19, 2020 at 03:31:50PM -0400, Robert Haas wrote:
> +1. Sorry for not seeing this thread sooner.

Please note that this has been applied as of 26ec6b5.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: invalid alloc size error possible in shm_mq
@ 2020-10-20 16:41  Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 6+ messages in thread

From: Peter Eisentraut @ 2020-10-20 16:41 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; +Cc: Markus Wanner <markus.wanner@2ndquadrant.com>; pgsql-bugs

On 2020-10-20 03:52, Michael Paquier wrote:
> On Mon, Oct 19, 2020 at 03:31:50PM -0400, Robert Haas wrote:
>> +1. Sorry for not seeing this thread sooner.
> 
> Please note that this has been applied as of 26ec6b5.

and backpatched now

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: invalid alloc size error possible in shm_mq
@ 2020-10-20 17:29  Markus Wanner <markus.wanner@2ndquadrant.com>
  parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
  0 siblings, 0 replies; 6+ messages in thread

From: Markus Wanner @ 2020-10-20 17:29 UTC (permalink / raw)
  To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; +Cc: pgsql-bugs

On 10/20/20 6:41 PM, Peter Eisentraut wrote:
> On 2020-10-20 03:52, Michael Paquier wrote:
>> Please note that this has been applied as of 26ec6b5.
> 
> and backpatched now

Awesome, thanks to everybody who reviewed.

-- 
Markus Wanner
Senior PostgreSQL Developer
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/





^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2020-10-20 17:29 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-08-25 10:00 Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-09-09 12:37 ` Markus Wanner <markus.wanner@2ndquadrant.com>
2020-10-19 19:31   ` Robert Haas <robertmhaas@gmail.com>
2020-10-20 01:52     ` Michael Paquier <michael@paquier.xyz>
2020-10-20 16:41       ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-10-20 17:29         ` Markus Wanner <markus.wanner@2ndquadrant.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox