public inbox for [email protected]  

help / color / mirror / Atom feed

Re: Quesion about querying distributed databases
2+ messages / 2 participants
[nested] [flat]

* Re: Quesion about querying distributed databases
@ 2025-03-05 07:39 Laurenz Albe <[email protected]>
  2025-03-05 09:27 ` Re: Quesion about querying distributed databases me nefcanto <[email protected]>
  0 siblings, 1 reply; 2+ messages in thread

From: Laurenz Albe @ 2025-03-05 07:39 UTC (permalink / raw)
  To: me nefcanto <[email protected]>; Adrian Klaver <[email protected]>; +Cc: [email protected]

On Wed, 2025-03-05 at 10:12 +0330, me nefcanto wrote:
> Adrian Klaver, thank you for the link. I asked the AI to create a query for me using FDW.
> 
> The problem here is that it collects all of the product_id values from the ItemCategories table [...]
> 
> That's not scalable. Is there a workaround for this?

Without having scrutinized the case in detail: if your data are organized in an
entity-attribute-value design and distributed across three databases, you cannot
expect to end up with efficient queries.

Perhaps you can extract the data and load them into a reasonably organized
single database.  Such an ETL process might make the task much easier.

Yours,
Laurenz Albe






^ permalink  raw  reply  [nested|flat] 2+ messages in thread

* Re: Quesion about querying distributed databases
  2025-03-05 07:39 Re: Quesion about querying distributed databases Laurenz Albe <[email protected]>
@ 2025-03-05 09:27 ` me nefcanto <[email protected]>
  0 siblings, 0 replies; 2+ messages in thread

From: me nefcanto @ 2025-03-05 09:27 UTC (permalink / raw)
  To: Laurenz Albe <[email protected]>; +Cc: Adrian Klaver <[email protected]>; [email protected]

Laurenz Albe, thanks for your answer.

Right now this data is in MariaDB, on separate databases (schema) but on
one server. The solution in this situation is to have a cross-database
query. (this is the status quo of our application)

Now our team has decided to migrate to Postgres. However, we realized that
Postgres does not support cross-database queries. And if we want to do so,
we should use FDW. So, we thought we might as well put databases on
separate servers for scalability if we have to write more code. That's the
reason behind this question.

But we're stuck at performance. In SQL Server and MariaDB, cross-database
queries allow for neat separation of data while delivering good performance
in the orchestration layer. You have separate databases, which allows for
fine-grained management (different backup schedules, index recalculation,
deployment, etc.) but at the same time you can write a query in your app,
or in an orchestrator database (let's call it All) that is fast enough for
millions of records.

However, we're stuck in this in Postgres. What solutions exist for this
problem?

Regards
Saeed


On Wed, Mar 5, 2025 at 11:09 AM Laurenz Albe <[email protected]>
wrote:

> On Wed, 2025-03-05 at 10:12 +0330, me nefcanto wrote:
> > Adrian Klaver, thank you for the link. I asked the AI to create a query
> for me using FDW.
> >
> > The problem here is that it collects all of the product_id values from
> the ItemCategories table [...]
> >
> > That's not scalable. Is there a workaround for this?
>
> Without having scrutinized the case in detail: if your data are organized
> in an
> entity-attribute-value design and distributed across three databases, you
> cannot
> expect to end up with efficient queries.
>
> Perhaps you can extract the data and load them into a reasonably organized
> single database.  Such an ETL process might make the task much easier.
>
> Yours,
> Laurenz Albe
>


^ permalink  raw  reply  [nested|flat] 2+ messages in thread

end of thread, other threads:[~2025-03-05 09:27 UTC | newest]

Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-03-05 07:39 Re: Quesion about querying distributed databases Laurenz Albe <[email protected]>
2025-03-05 09:27 ` me nefcanto <[email protected]>

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox