From: Justin Pryzby <pryzby@telsasoft.com>
To: Allan Barrielle <allan.barrielle@gmail.com>
Cc: pgsql-performance@lists.postgresql.org, Pierrick Martin <pierrick@mart1.dev>
Subject: Re: ETL - sql orchestrator is stuck when there is not sleep() between queries
Date: Thu, 8 Jul 2021 07:33:20 -0500
Message-ID: <20210708123320.GB22043@telsasoft.com> (raw)
In-Reply-To: <CANrBv8rnhJkFySe8qqW=bXiBPKKDjhXpdh=airBPgex3bTp-0A@mail.gmail.com>
References: <CANrBv8rnhJkFySe8qqW=bXiBPKKDjhXpdh=airBPgex3bTp-0A@mail.gmail.com>
On Thu, Jul 08, 2021 at 01:00:28AM +0200, Allan Barrielle wrote:
> All is fine, and can work great.
> But sometimes, some queries that used to take about 20 secs to complete can
> suddenly end in 5mins.
> Important all queries have the same shape -> CREATE TABLE SELECT AS *(a bit
> of transform) FROM TABLE). No update, nothing, it’s dead simple.
>
> Nothing works, it’s very random, some query won’t simply work ( even after
> hours ).
When it doesn't work, you could check SELECT * FROM pg_stat_activity, and
SELECT pg_blocking_pids(pid), * FROM pg_locks, to see what's going on.
> Important all queries have the same shape -> CREATE TABLE SELECT AS *(a bit
> of transform) FROM TABLE). No update, nothing, it’s dead simple.
> We are just trying to copy a table from schema1, to schema2, to schema3 and
> finally schema3. That’s it.
Is it true that the SELECTs have no joins in them ?
Did this ever work better or differently under different versions of postgres ?
> Why does the query never end even after hours ? Why there is no log about
> where the query is stuck.
Please send your nondefault config.
https://wiki.postgresql.org/wiki/Server_Configuration
Also enable logging (I just added this to the wiki).
https://wiki.postgresql.org/wiki/Slow_Query_Questions#Enable_Logging
It'd be very useful to get "explain analyze" for a working query and for a
stuck query. It sound like the stuck query never finishes, so maybe the second
part is impossible (?)
But it'd be good to get at least "explain" output. You'd have to edit your sql
script to run an "explain" before each query, and run it, logging the ouput,
until you capture the plan for a stuck query. Save the output and send here,
along with the query plan for a working query.
--
Justin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: pryzby@telsasoft.com, allan.barrielle@gmail.com, pierrick@mart1.dev
Subject: Re: ETL - sql orchestrator is stuck when there is not sleep() between queries
In-Reply-To: <20210708123320.GB22043@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox