Content-Type: text/plain;
	charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\))
Subject: Re: How to perform a long running dry run transaction without
 blocking
From: Robert Leach <rleach@princeton.edu>
In-Reply-To: <ef01c973-fecf-4c1f-a7ed-da7326dd0ca7@aklaver.com>
Date: Fri, 7 Feb 2025 17:02:03 -0500
Cc: pgsql-general@postgresql.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <15A999B4-9D53-4F35-84B1-7B8696256EE9@princeton.edu>
References: <BD62E056-3F3B-4CC0-A8CA-E5B7B9CB35CA@princeton.edu>
 <88d60ace-45e6-4d41-afc4-113df7219c4d@aklaver.com>
 <4000D0EE-B250-4E9E-831F-00C034D6D0B5@princeton.edu>
 <6d833658-f461-4ad4-a3e1-86d3c515bc18@aklaver.com>
 <0FE9C709-A108-4ED5-8132-B802B8D9908F@princeton.edu>
 <b89a7132-7fa5-4229-a03c-20f5d1e11cf4@aklaver.com>
 <00DF52D1-5ED0-4599-8480-8C671434EE4E@princeton.edu>
 <a24665e3-d712-4965-a904-26371e32ffe1@aklaver.com>
 <90368E5C-6F0C-45AE-B73A-5A467EC97043@princeton.edu>
 <CBFCB3DA-64AF-4A93-839B-B0D162FA5903@princeton.edu>
 <5b7b7573-3b75-4061-a6fe-1387d1e0e51c@aklaver.com>
 <67D44BCB-71FA-4EF6-8C41-53F852683CFA@princeton.edu>
 <ef01c973-fecf-4c1f-a7ed-da7326dd0ca7@aklaver.com>
To: Adrian Klaver <adrian.klaver@aklaver.com>
Archived-At: <https://www.postgresql.org/message-id/15A999B4-9D53-4F35-84B1-7B8696256EE9%40princeton.edu>
Precedence: bulk

>> Anyway, thanks so much for your help.  This discussion has been very =
useful, and I think I will proceed at first, exactly how you suggested, =
by queuing every validation job (using celery).  Then I will explore =
whether or not I can apply the "on timeout" strategy in a small patch.
>> Incidentally, during our Wednesday meeting this week, we actually =
opened our public instance to the world for the first time, in =
preparation for the upcoming publication.  This discussion is about the =
data submission interface, but that interface is actually disabled on =
the public-facing instance.  The other part of the codebase that I was =
primarily responsible for was the advanced search.  Everything else was =
primarily by other team members.  If you would like to check it out, let =
me know what you think: http://tracebase.princeton.edu =
<http://tracebase.princeton.edu>
>=20
> I would have to hit the books again to understand all of what is going =
on here.

It's a mass spec tracing database.  Animals are infused with radio =
labeled compounds and mass spec is used to see what the animal's =
biochemistry turns those compounds into.  (My undergrad was biochem, so =
I've been resurrecting my biochem knowledge, as needed for this project. =
 I've been mostly doing RNA and DNA sequence analysis since undergrad, =
and most of that was prokaryotic.

> One quibble with the Download tab, there is no indication of the size =
of the datasets. I generally like to know what I am getting into before =
I start a download. Also, is there explicit throttling going on? I am =
seeing 10.2kb/sec, whereas from here =
https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page I =
downloaded a 47.65M file at 41.9MB/s

Thank you!  Not knowing the download size is exactly a complaint I had.  =
That download actually uses my advanced search interface (in browse =
mode).  There is the same issue with the download buttons on the =
advanced search.  With the streaming, we're not dealing with temp files, =
which is nice, at least for the advanced search, but we can't know the =
download size that way.  So I had wanted a progress bar to at least show =
progress (current record per total).  I could even estimate the size (an =
option I explored for a few days).  Eventually, I proposed a celery =
solution for that and I was overruled.

As for the download in the nav bar, we have an issue to change that to a =
listing of actual files broken down by study (3 files per study).  =
There's not much actual utility from a user perspective for downloading =
everything anyway.  We've just been focussed on other things.  In fact, =
we have a request from a user for that specific feature, done in a way =
that's compatible with curl/scp.  We just have to figure out how to not =
have to CAS authenticate each command, something I don't have experience =
with.=