pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Rob Sargent <robjsargent@gmail.com>
To: Joe Carlson <jwcarlson@lbl.gov>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: TEXT column > 1Gb
Date: Wed, 12 Apr 2023 15:29:34 -0600
Message-ID: <4e976074-5708-01a8-4cbf-f317e4f424b0@gmail.com> (raw)
In-Reply-To: <131DA24F-23D6-4EA1-816F-ED0E6E5A219D@lbl.gov>
References: <85984fc4-6e08-06ad-78d7-d6f73a5c90a0@gmail.com>
	<0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov>
	<E18E02B1-39DE-4464-9C8C-F0C06BBF771B@enterprisedb.com>
	<A29ABFF1-8014-46B2-9425-34F59812581A@lbl.gov>
	<4777754a-ad1f-eae3-f6c9-a4f783e803e3@gmail.com>
	<04ab2b33-be6e-5073-1004-31a7117531af@gmail.com>
	<131DA24F-23D6-4EA1-816F-ED0E6E5A219D@lbl.gov>

On 4/12/23 15:03, Joe Carlson wrote:
>
>
>> On Apr 12, 2023, at 12:21 PM, Rob Sargent <robjsargent@gmail.com> wrote:
>>
>> On 4/12/23 13:02, Ron wrote:
>>> /Must/ the genome all be in one big file, or can you store them one 
>>> line per table row?
>
> The assumption in the schema I’m using is 1 chromosome per record. 
> Chromosomes are typically strings of continuous sequence (A, C, G, or 
> T) separated by gaps (N) of approximately known, or completely unknown 
> size. In the past this has not been a problem since sequenced 
> chromosomes were maybe 100 megabases. But sequencing is better now 
> with the technology improvements and tackling more complex genomes. So 
> gigabase chromosomes are common.
>
> A typical use case might be from someone interested in seeing if they 
> can identify the regulatory elements (the on or off switches) of a 
> gene. The protein coding part of a gene can be predicted pretty 
> reliably, but the upstream untranslated region and regulatory elements 
> are tougher. So they might come to our web site and want to extract 
> the 5 kb bit of sequence before the start of the gene and look for 
> some of the common motifs that signify a protein binding site. Being 
> able to quickly pull out a substring of the genome to drive a web app 
> is something we want to do quickly.
>>

Well if you're actually using the sequence, both text and bytea are 
inherently substring friendly.  Your problem goes back to transferring 
large strings and that's where http/tomcat is you friend.  Sounds like 
you're web friendly already.  You have to stream from the 
client/supplier, of course.

view thread (15+ messages)

Message-ID: <4e976074-5708-01a8-4cbf-f317e4f424b0@gmail.com>
Permalink:  ../4e976074-5708-01a8-4cbf-f317e4f424b0@gmail.com/
Also on:    postgresql.org/message-id/4e976074-5708-01a8-4cbf-f317e4f424b0@gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: robjsargent@gmail.com, jwcarlson@lbl.gov
  Subject: Re: TEXT column > 1Gb
  In-Reply-To: <4e976074-5708-01a8-4cbf-f317e4f424b0@gmail.com>
  References: <85984fc4-6e08-06ad-78d7-d6f73a5c90a0@gmail.com> <0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov> <E18E02B1-39DE-4464-9C8C-F0C06BBF771B@enterprisedb.com> <A29ABFF1-8014-46B2-9425-34F59812581A@lbl.gov> <4777754a-ad1f-eae3-f6c9-a4f783e803e3@gmail.com> <04ab2b33-be6e-5073-1004-31a7117531af@gmail.com> <131DA24F-23D6-4EA1-816F-ED0E6E5A219D@lbl.gov> <4e976074-5708-01a8-4cbf-f317e4f424b0@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox