public inbox for [email protected]  

help / color / mirror / Atom feed

From: Sohum Banerjea <[email protected]>
To: [email protected]
Cc: Tim McEwan <[email protected]>
Cc: Waseem Girach <[email protected]>
Cc: [email protected]
Subject: Suitability of postgres for high cardinality high volume usecase?
Date: Wed, 17 Jun 2026 13:21:23 +1000
Message-ID: <CAOdjd2R4HAaRx_oyJhRbRf1RZ=TEv9fxOdLqQ6dBAW6Z1UDd3g@mail.gmail.com> (raw)

Hello,

I am trying to determine the suitability of Postgres for a significant
climate risk modelling project.

We are batch processing a large (500 million) collection of
geographical points. For each point, we store ~6 dimensions of various
risks (total cardinality of several millions of floats per
geographical point).

We need to perform various ad-hoc aggregations on geographical subsets
of the values associated with these points. These aggregations could
require median/percentiles, so they won't be as simple as mean/sum,
and we expect we may have to write custom aggregations for some cases.

Because we may want to run computations that would use PostGIS
features (certainly polygon containment; potentially others), and
because our existing applications already use Postgres, we have some
degree of preference to do this in Postgres.

I'd like to know if anyone here has successfully built a system to run
this sort of computation at this scale in Postgres. If so, what sort
of schema design did you use? Columnar stores referencing spatially
indexed row stores that contain the spatial references, sharded by
geographical region? What sort of throughput did you achieve?

I'm also interested in any general observations folks may have about
this project. Perhaps we should use Clickhouse (for the main data)
together with Postgres (for the GIS computations)? Perhaps our float
dataset should live outside any kind of oltp/olap database at all?
Something else?

And finally, if you have developed a system like this, are you
available to assist us with building this system on a consulting
basis?

Thanks in advance,
—Sohum

view thread (3+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Suitability of postgres for high cardinality high volume usecase?
  In-Reply-To: <CAOdjd2R4HAaRx_oyJhRbRf1RZ=TEv9fxOdLqQ6dBAW6Z1UDd3g@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox