Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q7crG-0006g1-JS for pgsql-general@arkaria.postgresql.org; Fri, 09 Jun 2023 14:12:54 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1q7crF-0001Xu-4Q for pgsql-general@arkaria.postgresql.org; Fri, 09 Jun 2023 14:12:53 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q7crE-0001T0-PK for pgsql-general@lists.postgresql.org; Fri, 09 Jun 2023 14:12:52 +0000 Received: from mout-u-107.mailbox.org ([2001:67c:2050:101:465::107]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q7cr7-0015jd-Bs for pgsql-general@lists.postgresql.org; Fri, 09 Jun 2023 14:12:51 +0000 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:b231:465::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-u-107.mailbox.org (Postfix) with ESMTPS id 4Qd31z2BWkz9sRW; Fri, 9 Jun 2023 16:12:39 +0200 (CEST) Date: Fri, 9 Jun 2023 16:12:38 +0200 (CEST) From: Erik Wienhold To: Wim Bertels , "mail@joeconway.com" , "pgsql-general@lists.postgresql.org" , "pat.trainor@gmail.com" Message-ID: <1244897668.199925.1686319958286@office.mailbox.org> In-Reply-To: <4bb77358282da0509415de05079f118ef73f7033.camel@ucll.be> References: <626833f2-50e2-342c-e8ea-cb16b0d01785@joeconway.com> <4bb77358282da0509415de05079f118ef73f7033.camel@ucll.be> Subject: Re: How To: A large [2D] matrix, 100,000+ rows/columns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Priority: 3 Importance: Normal X-Rspamd-Queue-Id: 4Qd31z2BWkz9sRW List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk > On 09/06/2023 16:00 CEST Wim Bertels wrote: > > Joe Conway schreef op vr 09-06-2023 om 09:16 [-0400]: > > On 6/8/23 22:17, Pat Trainor wrote: > > > I need to have a very large matrix to maintain & query, and if not > > > (1,600 column limit), then how could such data be broken down to > > > work? > > > > =C2=A0 100,000 rows * > > =C2=A0 100,000 columns * > > =C2=A0 8 bytes (assuming float8) > > =3D about 80 GB per matrix if I got the math correct. > > based on my personal experience i would not use postgres in the case > where you need many columns, u can work around this with json for > example, but it will likely end up being less easy to work with > > as Joe replied: R or Python are probably a better fit, > or another database that can easily handle a lot of columns, > postgres is a great database, but not when you need a lot of columns > > (as you noted+: > there might be another backend storage for postgres that can handle > this better (or in the future?), but i don't think there is one; > also there is the header for which standard 8K is provisioned anyway, > so that is the first bottleneck (you can change this value, if you > compile postgres yourself) > https://www.postgresql.org/docs/current/limits.html ) Rasdaman may also be an option. Saw it a few weeks ago on this very list. https://rasdaman.org https://www.postgresql.org/message-id/CAFj8pRDjE0mdL6_b86ZDawHtNeRPQLciWos3= m3PGJueJ5COSjQ%40mail.gmail.com -- Erik