pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: invalid alloc size error possible in shm_mq
6+ messages / 4 participants
[nested] [flat]
* Re: invalid alloc size error possible in shm_mq
@ 2020-08-25 10:00 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-09-09 12:37 ` Re: invalid alloc size error possible in shm_mq Markus Wanner <markus.wanner@2ndquadrant.com>
0 siblings, 1 reply; 6+ messages in thread
From: Peter Eisentraut @ 2020-08-25 10:00 UTC (permalink / raw)
To: Markus Wanner <markus.wanner@2ndquadrant.com>; pgsql-bugs@postgresql.org, Robert Haas <robertmhaas@gmail.com>
On 2020-07-07 12:30, Markus Wanner wrote:
> in shm_mq_receive of shm_mq.c, a huge payload may trigger an unjustified
> "invalid memory alloc request size" error due to the way the buffer size
> is increased.
>
> I understand that this fine piece of infrastructure may not have been
> designed for huge payloads. I still argue it should better not fail
> that way for payloads between half a gigabyte and MaxAllocSize. The
> attached patch adds an appropriate assertion (documenting the upper
> limit) and avoids the error by limiting the allocation size to
> MaxAllocSize (to correct for such huge payloads below the limit).
I wonder if the assertion is appropriate or whether it should be a full
error check. Is anything on the sending side ensuring that the maximum
size is kept? All the size variables are Size/size_t so could be much
larger than MaxAllocSize.
Robert, thoughts?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invalid alloc size error possible in shm_mq
2020-08-25 10:00 Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2020-09-09 12:37 ` Markus Wanner <markus.wanner@2ndquadrant.com>
2020-10-19 19:31 ` Re: invalid alloc size error possible in shm_mq Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Markus Wanner @ 2020-09-09 12:37 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Markus Wanner <markus.wanner@2ndquadrant.com>; pgsql-bugs@postgresql.org, Robert Haas <robertmhaas@gmail.com>
On 8/25/20 12:00 PM, Peter Eisentraut wrote:
> I wonder if the assertion is appropriate or whether it should be a full
> error check.
Good point. Originally, it used to be an error. With the patch (but
w/o assertions enabled) it could result in a buffer overrun. Not good.
I changed the patch to add an error (instead of just an assert) when
asked to read a message larger than MaxAllocSize. So this patch
essentially corrects handling of messages in size between MaxAllocSize/2
and MaxAllocSize.
> Is anything on the sending side ensuring that the maximum
> size is kept? All the size variables are Size/size_t so could be much
> larger than MaxAllocSize.
In this v2 of the patch, I added a check that errors out on the sender
side as well (for trying to send a message larger than MaxAllocSize).
I'd still recommend to back-patch this.
--
Markus Wanner
Senior PostgreSQL Developer
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
Attachments:
shm_mq_inv_allocation_fix_v2.diff (text/x-patch, 1.8K) download
diff --git a/src/backend/storage/ipc/shm_mq.c b/src/backend/storage/ipc/shm_mq.c
index 6f40fb165d..262faecf13 100644
--- a/src/backend/storage/ipc/shm_mq.c
+++ b/src/backend/storage/ipc/shm_mq.c
@@ -24,6 +24,7 @@
#include "storage/procsignal.h"
#include "storage/shm_mq.h"
#include "storage/spin.h"
+#include "utils/memutils.h"
/*
* This structure represents the actual queue, stored in shared memory.
@@ -360,6 +361,13 @@ shm_mq_sendv(shm_mq_handle *mqh, shm_mq_iovec *iov, int iovcnt, bool nowait)
for (i = 0; i < iovcnt; ++i)
nbytes += iov[i].len;
+ /* Prevent writing messages overwhelming the receiver. */
+ if (nbytes > MaxAllocSize)
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("cannot send a message of size %zu via shared memory queue",
+ nbytes)));
+
/* Try to write, or finish writing, the length word into the buffer. */
while (!mqh->mqh_length_word_complete)
{
@@ -675,6 +683,17 @@ shm_mq_receive(shm_mq_handle *mqh, Size *nbytesp, void **datap, bool nowait)
}
nbytes = mqh->mqh_expected_bytes;
+ /*
+ * Should be disallowed on the sending side already, but better check
+ * and error out on the receiver side as well, rather than trying to
+ * read a prohibitively large message.
+ */
+ if (nbytes > MaxAllocSize)
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("invalid message size %zu in shared memory queue",
+ nbytes)));
+
if (mqh->mqh_partial_bytes == 0)
{
/*
@@ -703,8 +722,13 @@ shm_mq_receive(shm_mq_handle *mqh, Size *nbytesp, void **datap, bool nowait)
{
Size newbuflen = Max(mqh->mqh_buflen, MQH_INITIAL_BUFSIZE);
+ /*
+ * Double the buffer size until the payload fits, but limit to
+ * MaxAllocSize.
+ */
while (newbuflen < nbytes)
newbuflen *= 2;
+ newbuflen = Min(newbuflen, MaxAllocSize);
if (mqh->mqh_buffer != NULL)
{
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invalid alloc size error possible in shm_mq
2020-08-25 10:00 Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-09-09 12:37 ` Re: invalid alloc size error possible in shm_mq Markus Wanner <markus.wanner@2ndquadrant.com>
@ 2020-10-19 19:31 ` Robert Haas <robertmhaas@gmail.com>
2020-10-20 01:52 ` Re: invalid alloc size error possible in shm_mq Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 6+ messages in thread
From: Robert Haas @ 2020-10-19 19:31 UTC (permalink / raw)
To: Markus Wanner <markus.wanner@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-bugs
On Wed, Sep 9, 2020 at 8:37 AM Markus Wanner
<markus.wanner@2ndquadrant.com> wrote:
> I'd still recommend to back-patch this.
+1. Sorry for not seeing this thread sooner.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invalid alloc size error possible in shm_mq
2020-08-25 10:00 Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-09-09 12:37 ` Re: invalid alloc size error possible in shm_mq Markus Wanner <markus.wanner@2ndquadrant.com>
2020-10-19 19:31 ` Re: invalid alloc size error possible in shm_mq Robert Haas <robertmhaas@gmail.com>
@ 2020-10-20 01:52 ` Michael Paquier <michael@paquier.xyz>
2020-10-20 16:41 ` Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 6+ messages in thread
From: Michael Paquier @ 2020-10-20 01:52 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Markus Wanner <markus.wanner@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-bugs
On Mon, Oct 19, 2020 at 03:31:50PM -0400, Robert Haas wrote:
> +1. Sorry for not seeing this thread sooner.
Please note that this has been applied as of 26ec6b5.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invalid alloc size error possible in shm_mq
2020-08-25 10:00 Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-09-09 12:37 ` Re: invalid alloc size error possible in shm_mq Markus Wanner <markus.wanner@2ndquadrant.com>
2020-10-19 19:31 ` Re: invalid alloc size error possible in shm_mq Robert Haas <robertmhaas@gmail.com>
2020-10-20 01:52 ` Re: invalid alloc size error possible in shm_mq Michael Paquier <michael@paquier.xyz>
@ 2020-10-20 16:41 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-10-20 17:29 ` Re: invalid alloc size error possible in shm_mq Markus Wanner <markus.wanner@2ndquadrant.com>
0 siblings, 1 reply; 6+ messages in thread
From: Peter Eisentraut @ 2020-10-20 16:41 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; +Cc: Markus Wanner <markus.wanner@2ndquadrant.com>; pgsql-bugs
On 2020-10-20 03:52, Michael Paquier wrote:
> On Mon, Oct 19, 2020 at 03:31:50PM -0400, Robert Haas wrote:
>> +1. Sorry for not seeing this thread sooner.
>
> Please note that this has been applied as of 26ec6b5.
and backpatched now
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: invalid alloc size error possible in shm_mq
2020-08-25 10:00 Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-09-09 12:37 ` Re: invalid alloc size error possible in shm_mq Markus Wanner <markus.wanner@2ndquadrant.com>
2020-10-19 19:31 ` Re: invalid alloc size error possible in shm_mq Robert Haas <robertmhaas@gmail.com>
2020-10-20 01:52 ` Re: invalid alloc size error possible in shm_mq Michael Paquier <michael@paquier.xyz>
2020-10-20 16:41 ` Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2020-10-20 17:29 ` Markus Wanner <markus.wanner@2ndquadrant.com>
0 siblings, 0 replies; 6+ messages in thread
From: Markus Wanner @ 2020-10-20 17:29 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; +Cc: pgsql-bugs
On 10/20/20 6:41 PM, Peter Eisentraut wrote:
> On 2020-10-20 03:52, Michael Paquier wrote:
>> Please note that this has been applied as of 26ec6b5.
>
> and backpatched now
Awesome, thanks to everybody who reviewed.
--
Markus Wanner
Senior PostgreSQL Developer
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2020-10-20 17:29 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-08-25 10:00 Re: invalid alloc size error possible in shm_mq Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-09-09 12:37 ` Markus Wanner <markus.wanner@2ndquadrant.com>
2020-10-19 19:31 ` Robert Haas <robertmhaas@gmail.com>
2020-10-20 01:52 ` Michael Paquier <michael@paquier.xyz>
2020-10-20 16:41 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2020-10-20 17:29 ` Markus Wanner <markus.wanner@2ndquadrant.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox