From: Sergey Koposov <Sergey.Koposov@ed.ac.uk>
To: lepihov@gmail.com <lepihov@gmail.com>
To: tgl@sss.pgh.pa.us <tgl@sss.pgh.pa.us>
Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #18909: Query creates millions of temporary files and stalls
Date: Sat, 3 May 2025 21:50:04 +0000
Message-ID: <0ce957be37434cfca4820d1e89a70cb2b456d3db.camel@ed.ac.uk> (raw)
In-Reply-To: <ba927eb7126844c1b964e75cadb774488794dfde.camel@ed.ac.uk>
References: <18909-e5e1b702c9441b8a@postgresql.org>
<2051617.1746223978@sss.pgh.pa.us>
<beb051161520fc48248c2f90582eb15c6a71d239.camel@ed.ac.uk>
<5e98fba645682c40bf5f88ef858963ade9681956.camel@ed.ac.uk>
<2177960.1746289626@sss.pgh.pa.us>
<80bab58be182e80e73b5d5f71664ede9ce58f957.camel@ed.ac.uk>
<8fc51cd7-115a-425a-828e-c99defb5afef@gmail.com>
<ba927eb7126844c1b964e75cadb774488794dfde.camel@ed.ac.uk>
And this is explain analyze run that finishes in psql, with the default settings (i.e. with parallelism) :
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------
Gather (cost=14443102.62..111487693.90 rows=65470868 width=100) (actual time=212912.377..334892.676 rows=65514296 loops=1)
Workers Planned: 4
Workers Launched: 4
-> Nested Loop Left Join (cost=14442102.62..104939607.10 rows=16367717 width=100) (actual time=209806.019..317684.146 rows=13102859 loops=5)
-> Parallel Hash Left Join (cost=14442102.04..22124798.60 rows=16367717 width=60) (actual time=209805.943..273008.489 rows=13102859 loops=5)
Hash Cond: (d.objid = gaps1.original_ext_source_id)
-> Parallel Seq Scan on disk_sample1 d (cost=0.00..1218371.17 rows=16367717 width=60) (actual time=37.353..25185.340 rows=13095751 loops=5)
-> Parallel Hash (cost=10307380.24..10307380.24 rows=237862624 width=16) (actual time=169633.067..169633.068 rows=190290095 loops=5)
Buckets: 67108864 Batches: 32 Memory Usage: 1919904kB
-> Parallel Seq Scan on panstarrs1bestneighbour gaps1 (cost=0.00..10307380.24 rows=237862624 width=16) (actual time=132.295..117548.803
rows=190290095 loops=5)
-> Index Scan using gaia_sourcex_source_id_idx on gaia_source g (cost=0.58..5.05 rows=1 width=48) (actual time=0.003..0.003 rows=0 loops=65514296)
Index Cond: (source_id = gaps1.source_id)
Planning Time: 1.266 ms
JIT:
Functions: 75
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 11.796 ms, Inlining 293.374 ms, Optimization 81.354 ms, Emission 173.338 ms, Total 559.861 ms
Execution Time: 337814.695 ms
(18 rows)
And I did verify again that when I query through python (i.e. using the cursor) with
max_parallel_workers_per_gather=0 it finishes fine.
Also I clearly see that when I query through the cursor in python with the default settings (and when I see the issue), it clearly uses
a different plan, as opposed to just running the query in psql.
Because when running in psql I see these kinds of files in tmp
pgsql_tmp75270.0.fileset
as opposed to :
pgsql_tmp73459.0 ...
I don't think I know how to see the plan of the declare cursor query.
So summarizing:
the query produces millions of files in
1) query through cursor with default settings (max_parallel_workers_per_gather=4)
2) query through psql with no parallelism(max_parallel_workers_per_gather=0)
it works
3) query through cursor with no parallelism (max_parallel_workers_per_gather=0)
4) query through psql with default settings (max_parallel_workers_per_gather=4)
I hope it makes sense to someone & helps.
Sergey
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: Sergey.Koposov@ed.ac.uk, lepihov@gmail.com, tgl@sss.pgh.pa.us, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18909: Query creates millions of temporary files and stalls
In-Reply-To: <0ce957be37434cfca4820d1e89a70cb2b456d3db.camel@ed.ac.uk>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox