Feedback-ID: i76984098:Fastmail
Message-ID: <ef01c973-fecf-4c1f-a7ed-da7326dd0ca7@aklaver.com>
Date: Fri, 7 Feb 2025 13:36:54 -0800
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: How to perform a long running dry run transaction without
 blocking
To: Robert Leach <rleach@princeton.edu>
Cc: pgsql-general@postgresql.org
References: <BD62E056-3F3B-4CC0-A8CA-E5B7B9CB35CA@princeton.edu>
 <88d60ace-45e6-4d41-afc4-113df7219c4d@aklaver.com>
 <4000D0EE-B250-4E9E-831F-00C034D6D0B5@princeton.edu>
 <6d833658-f461-4ad4-a3e1-86d3c515bc18@aklaver.com>
 <0FE9C709-A108-4ED5-8132-B802B8D9908F@princeton.edu>
 <b89a7132-7fa5-4229-a03c-20f5d1e11cf4@aklaver.com>
 <00DF52D1-5ED0-4599-8480-8C671434EE4E@princeton.edu>
 <a24665e3-d712-4965-a904-26371e32ffe1@aklaver.com>
 <90368E5C-6F0C-45AE-B73A-5A467EC97043@princeton.edu>
 <CBFCB3DA-64AF-4A93-839B-B0D162FA5903@princeton.edu>
 <5b7b7573-3b75-4061-a6fe-1387d1e0e51c@aklaver.com>
 <67D44BCB-71FA-4EF6-8C41-53F852683CFA@princeton.edu>
Content-Language: en-US
From: Adrian Klaver <adrian.klaver@aklaver.com>
In-Reply-To: <67D44BCB-71FA-4EF6-8C41-53F852683CFA@princeton.edu>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: <https://www.postgresql.org/message-id/ef01c973-fecf-4c1f-a7ed-da7326dd0ca7%40aklaver.com>
Precedence: bulk

On 2/7/25 10:20, Robert Leach wrote:


> 
> Anyway, thanks so much for your help.  This discussion has been very 
> useful, and I think I will proceed at first, exactly how you suggested, 
> by queuing every validation job (using celery).  Then I will explore 
> whether or not I can apply the "on timeout" strategy in a small patch.
> 
> Incidentally, during our Wednesday meeting this week, we actually opened 
> our public instance to the world for the first time, in preparation for 
> the upcoming publication.  This discussion is about the data submission 
> interface, but that interface is actually disabled on the public-facing 
> instance.  The other part of the codebase that I was primarily 
> responsible for was the advanced search.  Everything else was primarily 
> by other team members.  If you would like to check it out, let me know 
> what you think: http://tracebase.princeton.edu 
> <http://tracebase.princeton.edu>

I would have to hit the books again to understand all of what is going 
on here. One quibble with the Download tab, there is no indication of 
the size of the datasets. I generally like to know what I am getting 
into before I start a download. Also, is there explicit throttling going 
on? I am seeing 10.2kb/sec, whereas from here 
https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page I 
downloaded a 47.65M file at 41.9MB/s

> 
> Cheers,
> Rob
> 
> 
> Robert William Leach
> Research Software Engineer
> 133 Carl C. Icahn Lab
> Lewis-Sigler Institute for Integrative Genomics
> Princeton University
> Princeton, NJ 08544
> 

-- 
Adrian Klaver
adrian.klaver@aklaver.com