pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Bug with memory leak on cert validation in libpq
5+ messages / 4 participants
[nested] [flat]

Commits from this thread:
  2020-04-21 27dbe1a184 e1c08722ae 35d08658d1 1bb0293d84 c9c15790d5 e5e295d500  Fix memory leak in libpq when using sslmode=verify-full
* Bug with memory leak on cert validation in libpq
@ 2020-04-20 17:40  Roman Peshkurov <roman.peshkurov@gmail.com>
  0 siblings, 1 reply; 5+ messages in thread

From: Roman Peshkurov @ 2020-04-20 17:40 UTC (permalink / raw)
  To: pgsql-bugs

Hi,

The problem is incorrect usage of openssl stack "destructor". We noticed it
on a graphs for the few months. It looked lika an increasing line of res
memory usage.

Sorry, I've already made pull request (wasn't familiar with your flow). The
all information is described there:
https://github.com/postgres/postgres/pull/52

Best regards, Roman.

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Bug with memory leak on cert validation in libpq
@ 2020-04-21 04:42  Michael Paquier <michael@paquier.xyz>
  parent: Roman Peshkurov <roman.peshkurov@gmail.com>
  0 siblings, 2 replies; 5+ messages in thread

From: Michael Paquier @ 2020-04-21 04:42 UTC (permalink / raw)
  To: Roman Peshkurov <roman.peshkurov@gmail.com>; +Cc: pgsql-bugs

On Mon, Apr 20, 2020 at 08:40:06PM +0300, Roman Peshkurov wrote:
> The problem is incorrect usage of openssl stack "destructor". We noticed it
> on a graphs for the few months. It looked lika an increasing line of res
> memory usage.
> 
> Sorry, I've already made pull request (wasn't familiar with your flow). The
> all information is described there:
> https://github.com/postgres/postgres/pull/52

Thanks for taking the time to send an email here.  I suspect that few
people around here look at the cloned repo on github (I don't FWIW).

If this reference goes away, then then community archives would lose a
part of its history.  So, first, here is the patch for reference:
inline patch 1: src/interfaces/libpq/fe-secure-openssl.c (+1/-1 lines) download inline-1.patch--- a/src/interfaces/libpq/fe-secure-openssl.c +++ b/src/interfaces/libpq/fe-secure-openssl.c @@ -552,7 +552,7 @@ pgtls_verify_peer_name_matches_certificate_guts(PGconn *conn, if (rc != 0) break; } - sk_GENERAL_NAME_free(peer_san); + sk_GENERAL_NAME_pop_free(peer_san, GENERAL_NAME_free); }
Looking at the samples of the OpenSSL wiki, it looks that your guess is right: https://wiki.openssl.org/index.php/Hostname_validation After extracting the SAN data from the certificate using X509_get_ext_d2i(), we get to use sk_GENERAL_NAME_pop_free() for a correct in-depth free(). Now, we have in core a set of TAP tests in src/test/ssl with certificates that include some SAN configuration for sslmode=verify-full, able to trigger the code path you are mentioning for pgtls_verify_peer_name_matches_certificate_guts() within fe-secure-openssl.c. In order to see the leak, (I have been lazy with the method and did not use valgrind with a custom C program or just psql, but the result is the same), I have put a custom hack in the code path involved here by adding a tight loop repeating X509_get_ext_d2i() + sk_GENERAL_NAME_free(), then ran the TAP tests of src/test/ssl/. After that, the leak becomes really clear after 5k loops or such in psql, with sk_GENERAL_NAME_pop_free() preventing the issue any rise of memory usage. sk_GENERAL_NAME_pop_free() exists down to OpenSSL 0.9.6, which is what I recall we officially support down to Postgres 9.5, so we are safe. And finally, this needs to be backpatched all the way down to 9.5, so I'll take care of it if there are no objections. Thanks for the report, Roman! -- Michael Attachments: signature.asc (application/pgp-signature, 832B) download ^ permalink raw reply [nested|flat] 5+ messages in thread

* Re: Bug with memory leak on cert validation in libpq
@ 2020-04-21 05:00  Hamid Akhtar <hamid.akhtar@gmail.com>
  parent: Michael Paquier <michael@paquier.xyz>
  1 sibling, 0 replies; 5+ messages in thread

From: Hamid Akhtar @ 2020-04-21 05:00 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: Roman Peshkurov <roman.peshkurov@gmail.com>; pgsql-bugs

+1

Specifically the openssl documentation says:

void sk_free(STACK *);
    This function free()'s a stack structure.  The elements in the
stack will not be freed so one should 'pop' and free all elements from the
stack before calling this function or call sk_pop_free() instead.

void sk_pop_free(STACK *st; void (*func)());
    This function calls 'func' for each element on the stack, passing
the element as the argument.  sk_free() is then called to free the 'stack'
structure.



On Tue, Apr 21, 2020 at 9:42 AM Michael Paquier <michael@paquier.xyz> wrote:

> On Mon, Apr 20, 2020 at 08:40:06PM +0300, Roman Peshkurov wrote:
> > The problem is incorrect usage of openssl stack "destructor". We noticed
> it
> > on a graphs for the few months. It looked lika an increasing line of res
> > memory usage.
> >
> > Sorry, I've already made pull request (wasn't familiar with your flow).
> The
> > all information is described there:
> > https://github.com/postgres/postgres/pull/52
>
> Thanks for taking the time to send an email here.  I suspect that few
> people around here look at the cloned repo on github (I don't FWIW).
>
> If this reference goes away, then then community archives would lose a
> part of its history.  So, first, here is the patch for reference:
> --- a/src/interfaces/libpq/fe-secure-openssl.c
> +++ b/src/interfaces/libpq/fe-secure-openssl.c
> @@ -552,7 +552,7 @@ pgtls_verify_peer_name_matches_certificate_guts(PGconn
> *conn,
>             if (rc != 0)
>                 break;
>             }
> -           sk_GENERAL_NAME_free(peer_san);
> +           sk_GENERAL_NAME_pop_free(peer_san, GENERAL_NAME_free);
>         }
>
> Looking at the samples of the OpenSSL wiki, it looks that your guess
> is right:
> https://wiki.openssl.org/index.php/Hostname_validation
> After extracting the SAN data from the certificate using
> X509_get_ext_d2i(), we get to use sk_GENERAL_NAME_pop_free() for a
> correct in-depth free().
>
> Now, we have in core a set of TAP tests in src/test/ssl with
> certificates that include some SAN configuration for
> sslmode=verify-full, able to trigger the code path you are mentioning
> for pgtls_verify_peer_name_matches_certificate_guts() within
> fe-secure-openssl.c.  In order to see the leak, (I have been lazy with
> the method and did not use valgrind with a custom C program or just
> psql, but the result is the same), I have put a custom hack in the
> code path involved here by adding a tight loop repeating
> X509_get_ext_d2i() + sk_GENERAL_NAME_free(), then ran the TAP tests of
> src/test/ssl/.  After that, the leak becomes really clear after 5k
> loops or such in psql, with sk_GENERAL_NAME_pop_free() preventing the
> issue any rise of memory usage.
>
> sk_GENERAL_NAME_pop_free() exists down to OpenSSL 0.9.6, which is what
> I recall we officially support down to Postgres 9.5, so we are safe.
>
> And finally, this needs to be backpatched all the way down to 9.5, so
> I'll take care of it if there are no objections.
>
> Thanks for the report, Roman!
> --
> Michael
>


-- 
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca
ADDR: 10318 WHALLEY BLVD, Surrey, BC
CELL:+923335449950  EMAIL: mailto:hamid.akhtar@highgo.ca
SKYPE: engineeredvirus

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Bug with memory leak on cert validation in libpq
@ 2020-04-21 14:46  David Steele <david@pgmasters.net>
  parent: Michael Paquier <michael@paquier.xyz>
  1 sibling, 1 reply; 5+ messages in thread

From: David Steele @ 2020-04-21 14:46 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; Roman Peshkurov <roman.peshkurov@gmail.com>; +Cc: pgsql-bugs

On 4/21/20 12:42 AM, Michael Paquier wrote:
> On Mon, Apr 20, 2020 at 08:40:06PM +0300, Roman Peshkurov wrote:
>> The problem is incorrect usage of openssl stack "destructor". We noticed it
>> on a graphs for the few months. It looked lika an increasing line of res
>> memory usage.
> 
> And finally, this needs to be backpatched all the way down to 9.5, so
> I'll take care of it if there are no objections.

+1. We have similar code in pgBackRest and I can see the leak there as 
well when I adjust our Valgrind supressions a bit.

I also reviewed the OpenSSL code and agree this is the right fix.

Regards,
-- 
-David
david@pgmasters.net





^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Bug with memory leak on cert validation in libpq
@ 2020-04-21 22:31  Michael Paquier <michael@paquier.xyz>
  parent: David Steele <david@pgmasters.net>
  0 siblings, 0 replies; 5+ messages in thread

From: Michael Paquier @ 2020-04-21 22:31 UTC (permalink / raw)
  To: David Steele <david@pgmasters.net>; +Cc: Roman Peshkurov <roman.peshkurov@gmail.com>; pgsql-bugs

On Tue, Apr 21, 2020 at 10:46:34AM -0400, David Steele wrote:
> I also reviewed the OpenSSL code and agree this is the right fix.

Thanks for confirming.  The fix has been applied down to 9.5.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 5+ messages in thread


end of thread, other threads:[~2020-04-21 22:31 UTC | newest]

Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-04-20 17:40 Bug with memory leak on cert validation in libpq Roman Peshkurov <roman.peshkurov@gmail.com>
2020-04-21 04:42 ` Michael Paquier <michael@paquier.xyz>
2020-04-21 05:00   ` Hamid Akhtar <hamid.akhtar@gmail.com>
2020-04-21 14:46   ` David Steele <david@pgmasters.net>
2020-04-21 22:31     ` Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox