From: Joe Conway <mail@joeconway.com>
To: Pat Trainor <pat.trainor@gmail.com>
To: pgsql-general@lists.postgresql.org
Subject: Re: How To: A large [2D] matrix, 100,000+ rows/columns
Date: Fri, 9 Jun 2023 09:16:02 -0400
Message-ID: <626833f2-50e2-342c-e8ea-cb16b0d01785@joeconway.com> (raw)
In-Reply-To: <CAK_m+w=zf1mLP4+N+rNDaG5xU1_MjqTUDYbf+D-RyiF4ek5mQw@mail.gmail.com>
References: <CAK_m+w=zf1mLP4+N+rNDaG5xU1_MjqTUDYbf+D-RyiF4ek5mQw@mail.gmail.com>
On 6/8/23 22:17, Pat Trainor wrote:
> Imagine something akin to stocks, where you have a row for every stock,
> and a column for every stock. Except where the same stock is the row &
> col, a number is at each X-Y (row/column), and that is the big picture.
> I need to have a very large matrix to maintain & query, and if not
> (1,600 column limit), then how could such data be broken down to work?
100,000 rows *
100,000 columns *
8 bytes (assuming float8)
= about 80 GB per matrix if I got the math correct.
Is this really a dense matrix or is it sparse? What kind of operations?
Does it really need to be stored as such or could it be stored as
vectors that are converted to a matrix on the fly when needed?
Seems like using python or R makes more sense. Perhaps it might make
sense to store the data in Postgres and use plpython or plr. But it is
hard to say with more details.
--
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: mail@joeconway.com, pat.trainor@gmail.com
Subject: Re: How To: A large [2D] matrix, 100,000+ rows/columns
In-Reply-To: <626833f2-50e2-342c-e8ea-cb16b0d01785@joeconway.com>
References: <CAK_m+w=zf1mLP4+N+rNDaG5xU1_MjqTUDYbf+D-RyiF4ek5mQw@mail.gmail.com> <626833f2-50e2-342c-e8ea-cb16b0d01785@joeconway.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox