public inbox for [email protected]
help / color / mirror / Atom feedFrom: lakshmi <[email protected]>
To: pgsql-hackers <[email protected]>
Cc: Filip Janus <[email protected]>
Cc: [email protected]
Subject: Re: Proposal: Adding compression of temporary files
Date: Wed, 21 Jan 2026 12:02:58 +0530
Message-ID: <CAEvyyTj5G3BAFYt=VHJPLFF_meJGmSEk+J4BqM7gGUen8sXchw@mail.gmail.com> (raw)
In-Reply-To: <CAEvyyTjiUhEUX5jxJkHHncnKSuCmf=MitZLtDt+Z6WsNNo0vCQ@mail.gmail.com>
References: <CAFjYY++22tmGDSac_YPA_NLJ4-cEJo7kS3SxXFHf7ZFTnwO-QA@mail.gmail.com>
<CAFjYY+J7ZcJAoE+K4AoyuiS_N6w8vVCQ6HQ72mPB=i4C+QbAeQ@mail.gmail.com>
<CAFjYY+KhiuzocMdgEr8wQtCkSaec=Mu_YiLE6N3JfYtQ-Xv=ug@mail.gmail.com>
<CAFjYY++YjtunZcFt7MFBWwseP_=GS11=V36_xig07enHA0eRjA@mail.gmail.com>
<CAPpHfdvD-oqh=qvtkD3qWqZrfxt0+PtdwyRYC+faQd-rr+V0mg@mail.gmail.com>
<[email protected]>
<CAFjYY+Lhw2tvsiC8_Dbx+K9dMaK=myT5Cck2=8FgLUW5f9Hp-A@mail.gmail.com>
<udbqvyuxlxk4a64uomo5xmgloemk7jrbeeluhvxbdqxcdv2pt5@who34o73ttmo>
<CAFjYY+LcBCPe9cboef1GteAPKEiqyTTww7NGxa==roc6QXSOxA@mail.gmail.com>
<CAFjYY+LJ5_j82PrSxm2RjM0OjwfN9smJ1VKKMOka-25Jg-0ofg@mail.gmail.com>
<7gbzkoxrpqvj2sgxof4pqirz7pf6wb5vizak3mxrz2zpvrf3pa@lvnwdsy3em56>
<CAFjYY+LH=MbMpGiYfMtPsANDBAqiDC66hD3tBs_JeVcBYVcmHA@mail.gmail.com>
<CAFjYY+Jfsy1r4ZP4+RyHZeWE7JyT4LSdVFvA9ck04LFkKd1ebw@mail.gmail.com>
<CAFjYY+K=s8gL=3SN8puwjSm5ZvUXumsdeB-Z2rwLP5OQNtEF+Q@mail.gmail.com>
<CAEvyyThUp+Cyv8Se6dxpJvP7w=u1670Xd4k6zeXGVSac7UOm_A@mail.gmail.com>
<CAFjYY+KjoNLYGFeM4z=DOgG1uDwXD1Z6zyurgz1d9Y-96X=9GA@mail.gmail.com>
<CAFjYY+JDSpOQwYAfTQQ43=BA=d32XfcAdaPVJgHheV9fQBbLWg@mail.gmail.com>
<CAN4CZFPmAwOvR8r1gtv=r+4akArZJyjna_Nxgkn_eTcsRs4XPA@mail.gmail.com>
<CAEvyyTjiUhEUX5jxJkHHncnKSuCmf=MitZLtDt+Z6WsNNo0vCQ@mail.gmail.com>
HI all,
While testing the temp file compression patch,noticed that the new
temp_file_compression GUC isn't documented yet.I put together a small docs
patch to add a short description and clarify that the effect of compression
depends on the workload(for example ,hash join spills may not show visible
size reduction due to fixed_size chunks).
patch is attached.Happy to adjust the wording if needed.
thanks,
lakshmi
On Tue, Jan 20, 2026 at 4:21 PM lakshmi <[email protected]> wrote:
> Hi Filip,
>
> I tested both patches on current master using git am -3 .They apply
> cleanly,build fine,and the temp_file _compression GUC works as expected.
> Query results are unchanged.
>
> For hash join spill test,temp files were created as expected,but the
> logged size were same for no,lz4,and pglz,which seems consistent with
> fixed-size fileset chunking.It might be helpful to briefly note this in the
> documentation to avoid confusion.
>
> Thanks for working on this .
> best regards,
> lakshmi
>
> On Tue, Jan 20, 2026 at 4:10 AM Zsolt Parragi <[email protected]>
> wrote:
>
>> Hello!
>>
>> I tried to review the code. It compiled, the test suite passed.
>>
>> I noticed two typos:
>>
>> buffile.c:77 - "Disaled"
>> buffile.c:133 - "mathods"
>>
>> And a few other small findings:
>>
>> buffile.h:35 and buffile.c:63 - same constants defined first as an
>> Enum and then as #defines - code builds properly without the defines.
>>
>> buffile.c:121 - compress_tempfile is defined, set to false at :167,
>> but never used otherwise
>>
>> guc_tables.c:470 - the comment says that pglz isn't supported yet, but
>> we have a value for it, and I see support for it in the code
>>
>> buffile.c:659: (and at other places) if USE_LZ4 is undefined, the
>> codepath doesn't do anything. I think these ifdefs should follow how
>> other compression code works, such as wal compression where there's an
>> #else path with elog(ERROR, ...)
>> Similarly, maybe there should be an explicit TEMP_NONE_COMPRESSION
>> branch that does nothing, and the default branch should be an error?
>>
>> buffile.c:265: If seek isn't supported/limited, shouldn't there be at
>> least an assertion about it in BufFileSeek? And tell isn't mentioned,
>> but it seems to me that tell also doesn't work properly.
>>
>
Attachments:
[text/x-patch] doc-temp-file-compression-doc.patch (1.2K, 3-doc-temp-file-compression-doc.patch)
download | inline diff:
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 0fad34da6eb..57a8af2a2fc 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1955,6 +1955,26 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-temp-file-compression">
+ <term><varname>temp_file_compression</varname> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ Enables transparent compression of temporary files used by query execution.
+ Supported values are <literal>no</literal>, <literal>lz4</literal>, and
+ <literal>pglz</literal>.
+ </para>
+
+ <para>
+ The effectiveness of temporary file compression depends on the workload.
+ For example, temporary files created by hash join spills use fixed-size
+ chunks, so on-disk file sizes may not visibly shrink even when compression
+ is enabled. Statistics such as <literal>temp_bytes</literal> report logical
+ bytes written before compression.
+ </para>
+ </listitem>
+ </varlistentry>
+
+
<varlistentry id="guc-hash-mem-multiplier" xreflabel="hash_mem_multiplier">
<term><varname>hash_mem_multiplier</varname> (<type>floating point</type>)
<indexterm>
view thread (13+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: Proposal: Adding compression of temporary files
In-Reply-To: <CAEvyyTj5G3BAFYt=VHJPLFF_meJGmSEk+J4BqM7gGUen8sXchw@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox