Subject: Re: temporary file log lines
To: Laurenz Albe <laurenz.albe@cybertec.at>
Cc: PostgreSQL Performance <pgsql-performance@postgresql.org>
References: <d658f01b-7a1d-da5e-e996-3403e6b7cf49@sqlexec.com>
 <4827fef49e0d0410eedca61e235a16efc1d835de.camel@cybertec.at>
From: MichaelDBA <MichaelDBA@sqlexec.com>
Message-ID: <ce8f3920-764e-a495-1bab-8b0bf22d671e@sqlexec.com>
Date: Mon, 12 Jul 2021 08:13:16 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 PostboxApp/7.0.48
MIME-Version: 1.0
In-Reply-To: <4827fef49e0d0410eedca61e235a16efc1d835de.camel@cybertec.at>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Archived-At: 
 <https://www.postgresql.org/message-id/ce8f3920-764e-a495-1bab-8b0bf22d671e%40sqlexec.com>
Precedence: bulk

hmmm, I think spilling over to disk for temporary tables is handled by 
an entirely different branch in the PG source code.  In fact, some other 
folks have chimed in and said log_temp_files doesn't relate to temp 
files at all use by temporary tables, just queries as you mentioned 
below elsewhere.  This seems to be a dark area of PG that is not 
convered well.

Regards,
Michael Vitale


Laurenz Albe wrote on 7/12/2021 8:01 AM:
> On Thu, 2021-07-08 at 17:22 -0400, MichaelDBA wrote:
>> I got a question about PG log lines with temporary file info like this:
>>
>> case 1: log line with no contextual info
>> 2021-07-07 20:28:15 UTC:10.100.11.95(50274):myapp@mydb:[35200]:LOG:
>> temporary file: path "base/pgsql_tmp/pgsql_tmp35200.0", size 389390336
>>
>> case 2: log line with contextual info
>> 2021-07-07 20:56:18 UTC:172.16.193.118(56080):myapp@mydb:[22418]:LOG:
>> temporary file: path "base/pgsql_tmp/pgsql_tmp22418.0", size 1048576000
>> 2021-07-07 20:56:18
>> UTC:172.16.193.118(56080):myapp@mydb:[22418]:CONTEXT:  PL/pgSQL function
>> memory.f_memory_usage(boolean) line 13 at RETURN QUERY
>>
>> There are at least 2 cases where stuff can spill over to disk:
>> * queries that don't fit in work_mem, and
>> * temporary tables that don't fit in temp_buffers
>>
>> Question, if log_temp_files is turned on (=0), then how can you tell
>> from where the temporary log line comes from?
>> I see a pattern where work_mem spill overs have a CONTEXT line that
>> immediately follows the LOG LINE with keyword, temporary. See case 2 above.
>>
>> For other LOG lines with keyword, temporary, there is no such pattern.
>> Could those be the ones caused by temp_buffer spill overs to disk?  case
>> 1 above.
>>
>> I really want to tune temp_buffers, but I would like to be able to
>> detect when temporary tables are spilling over to disk, so that I can
>> increase temp_buffers.
>>
>> Any help would be appreciated.
> I am not sure if you can istinguish those two cases from the log.
>
> What I would do is identify the problematic query and run it with
> EXPLAIN (ANALYZE, BUFFERS).  Then you should see which part of the query
> creates the temporary files.
>
> If it is a statement in a function called from your top level query,
> auto_explain with the correct parameters can get you that output for
> those statements too.
>
> Yours,
> Laurenz Albe