Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pmg21-00047M-6m for pgsql-general@arkaria.postgresql.org; Wed, 12 Apr 2023 19:21:25 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pmg1y-0001il-VZ for pgsql-general@arkaria.postgresql.org; Wed, 12 Apr 2023 19:21:22 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pmg1y-0001ic-Kh for pgsql-general@lists.postgresql.org; Wed, 12 Apr 2023 19:21:22 +0000 Received: from mail-pj1-x1031.google.com ([2607:f8b0:4864:20::1031]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1pmg1s-0003wf-3c for pgsql-general@lists.postgresql.org; Wed, 12 Apr 2023 19:21:21 +0000 Received: by mail-pj1-x1031.google.com with SMTP id b2-20020a17090a6e0200b002470b249e59so1602570pjk.4 for ; Wed, 12 Apr 2023 12:21:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681327274; x=1683919274; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=1L5bRd2A8WzyERxxFSZc0mB0+9jDDp1msQeIRj1DzOM=; b=F0pibD9nQFLVsxceOWPbsinkHtNWbUaRXJ+FiBF+y1MioOaXrf/cTwM0QVSaPp58nW nPCLx8nnuOPpr8yWwLaMC2ECa9pfz4h7onrPBrwIZYX+Sc/yhN/VUAxyDMex1WfnFU0R YORzgfA3eCUHSbsp/fJXtsH0a0MFyf2ccIU3hEQC43mNWsVorXagcFaWuo5kBDt2kfH4 8bYvJJxzTZFvP52InBJnEBRi+6GsrOOVbhm9PCPx8TfxdQkKt38SI5oj4aGUeKR7RaNJ LVgjlJerPomM/OKYnpEpD6RALBoG6SgDTwLv1VkntNpSS13RUxrKHvMF9XKT5RFNplQ7 ejpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681327274; x=1683919274; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=1L5bRd2A8WzyERxxFSZc0mB0+9jDDp1msQeIRj1DzOM=; b=kBoTdX5wGHu9M1TztwCOq1jIQFaiLx5BFzf2B3keDV57SsT/IC2YfusWqAnSAUDV2n L+dI/U5dLkefAJDc3a/J7nErsj/WKe/16XIf3ek1rSSDTFXHq/LwhadtOeOt1mYUuF4F 2Kw6Y43nNHYm7WcJnrlx7tcGbS+M8t0IqbgcUsleoEZUVGv9Nx6+XDtCZNn3oPAfd7CO DITASMzYO5b8/yjepn23vfGmAajBb8XMJoHv6QCCyAxMnlsT4SoP9+l9W3ZieG+VNOjA GndNmKgQuyRTxBGfMni9i3taX10dUCmP7//BBmcQYPPCjHFj1JqP59CV7toVwF3jnjdz Frbg== X-Gm-Message-State: AAQBX9ft/uf1tQE+XyOMa96vtk0uBsc35X/VCw9Kuz02AxO8tL4+9qAb JVdtKZLM7AbznuFCFt6FlJVmOLQgZCg= X-Google-Smtp-Source: AKy350YQIqlFanUctlHbn5gnKnSXmIS8CCAKl2eYxqZXcHW8tFm4i8sgdf77NvKPeCzxqZ8LKu+45Q== X-Received: by 2002:a17:903:248:b0:1a4:f7c9:33e7 with SMTP id j8-20020a170903024800b001a4f7c933e7mr42554plh.1.1681327274185; Wed, 12 Apr 2023 12:21:14 -0700 (PDT) Received: from ?IPV6:2601:681:5a00:f790:7096:ab4:cda6:9ce7? ([2601:681:5a00:f790:7096:ab4:cda6:9ce7]) by smtp.gmail.com with ESMTPSA id bh3-20020a170902a98300b001a2b4c79f34sm11890176plb.252.2023.04.12.12.21.13 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 12 Apr 2023 12:21:13 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------IeaJiNwbENnMJjQmyu7WbmHk" Message-ID: <04ab2b33-be6e-5073-1004-31a7117531af@gmail.com> Date: Wed, 12 Apr 2023 13:21:12 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: TEXT column > 1Gb Content-Language: en-CA To: pgsql-general@lists.postgresql.org References: <85984fc4-6e08-06ad-78d7-d6f73a5c90a0@gmail.com> <0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov> <4777754a-ad1f-eae3-f6c9-a4f783e803e3@gmail.com> From: Rob Sargent In-Reply-To: <4777754a-ad1f-eae3-f6c9-a4f783e803e3@gmail.com> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------IeaJiNwbENnMJjQmyu7WbmHk Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 4/12/23 13:02, Ron wrote: > /Must/ the genome all be in one big file, or can you store them one > line per table row? Not sure what OP is doing with plant genomes (other than some genomics) but the tools all use files and pipeline of sub-tools.  In and out of tuples would be expensive.  Very,very little "editing" done in the usual "update table set val where id" sense. Lines in a vcf file can have thousands of colums fo nasty, cryptic garbage data that only really makes sense to tools, reader.  Highly denormalized of course.  (Btw, I hate sequencing :) ) --------------IeaJiNwbENnMJjQmyu7WbmHk Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 4/12/23 13:02, Ron wrote:
Must the genome all be in one big file, or can you store them one line per table row?

Not sure what OP is doing with plant genomes (other than some genomics) but the tools all use files and pipeline of sub-tools.  In and out of tuples would be expensive.  Very,very little "editing" done in the usual "update table set val where id" sense.

Lines in a vcf file can have thousands of colums fo nasty, cryptic garbage data that only really makes sense to tools, reader.  Highly denormalized of course.  (Btw, I hate sequencing :) )



--------------IeaJiNwbENnMJjQmyu7WbmHk--