public inbox for [email protected]  

help / color / mirror / Atom feed
From: Tomas Vondra <[email protected]>
To: Amit Langote <[email protected]>
Cc: Alvaro Herrera <[email protected]>
Cc: Erik Rijkers <[email protected]>
Cc: jian he <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Andrew Dunstan <[email protected]>
Cc: PostgreSQL-development <[email protected]>
Subject: Re: remaining sql/json patches
Date: Thu, 7 Mar 2024 12:02:27 +0100
Message-ID: <[email protected]> (raw)
In-Reply-To: <CA+HiwqHXzxg8-raLpzCa7BYmYVUUFPOEZo3EVgCkcnXks9=vUQ@mail.gmail.com>
References: <[email protected]>
	<[email protected]>
	<CA+HiwqGtdVCg-KboUUU=svwukwXGT4EOFOk3eCZLz6h-d6YLNw@mail.gmail.com>
	<[email protected]>
	<CA+HiwqGYkYjXe2g0=c==BGSpVQfMJZBhPE-vesX3xCz4enBYPg@mail.gmail.com>
	<CA+HiwqHXzxg8-raLpzCa7BYmYVUUFPOEZo3EVgCkcnXks9=vUQ@mail.gmail.com>



On 3/7/24 08:26, Amit Langote wrote:
> On Wed, Mar 6, 2024 at 1:07 PM Amit Langote <[email protected]> wrote:
>> Hi Tomas,
>>
>> On Wed, Mar 6, 2024 at 6:30 AM Tomas Vondra
>> <[email protected]> wrote:
>>> I'd say 2.5GB in ExecutorState seems a bit excessive ... Seems there's
>>> some memory management issue? My guess is we're not releasing memory
>>> allocated while parsing the JSON or building JSON output.
>>>
>>> I'm not attaching the data, but I can provide that if needed - it's
>>> about 600MB compressed. The structure is not particularly complex, it's
>>> movie info from [1] combined into a JSON document (one per movie).
>>
>> Thanks for the report.
>>
>> Yeah, I'd like to see the data to try to drill down into what's piling
>> up in ExecutorState.  I want to be sure of if the 1st, query functions
>> patch, is not implicated in this, because I'd like to get that one out
>> of the way sooner than later.
> 
> I tracked this memory-hogging down to a bug in the query functions
> patch (0001) after all.  The problem was with a query-lifetime cache
> variable that was never set to point to the allocated memory.  So a
> struct was allocated and then not freed for every row where it should
> have only been allocated once.
> 

Thanks! I can confirm the query works with the new patches.

Exporting the 7GB table takes ~250 seconds (the result is ~10.6GB). That
seems maybe a bit much, but I'm not sure it's the fault of this patch.
Attached is a flamegraph for the export, and clearly most of the time is
spent in jsonpath. I wonder if there's a way to improve this, but I
don't think it's up to this patch.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Attachments:

  [image/svg+xml] flamegraph.svg (153.0K, ../[email protected]/2-flamegraph.svg)
  download | view image

view thread (2+ messages)  latest in thread

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: remaining sql/json patches
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox