pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedPros/cons of big databases vs smaller databases and RDS
3+ messages / 3 participants
[nested] [flat]
* Pros/cons of big databases vs smaller databases and RDS
@ 2021-10-05 22:06 Wells Oliver <wells.oliver@gmail.com>
0 siblings, 2 replies; 3+ messages in thread
From: Wells Oliver @ 2021-10-05 22:06 UTC (permalink / raw)
To: pgsql-admin
Hi-- to keep it short, I feel like I've generally heard the larger your DB
is, the less efficient it might run, likely due to disk I/O. Maybe I'm
terribly mistaken in this perception.
We have two DBs, one primarily accessed by humans and systems, which is
~1TB in size, and aggregates most of what we store in raw, longer format on
a second DB that is about ~6TB in size.
As we consider plans to migrate to RDS, we've talked a lot about
combining the two as more and more the case is querying the larger DB and
wanting data only available in the smaller DB.
Of course, we can solve this by copying things back and forth, but we're
also thinking: why not just one big DB?
Anyone have any experiences with a similar project, and especially any
technical configurations that might be beneficial in using RDS?
Appreciate it.
--
Wells Oliver
wells.oliver@gmail.com <wellsoliver@gmail.com>
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Pros/cons of big databases vs smaller databases and RDS
@ 2021-10-06 03:05 Ron <ronljohnsonjr@gmail.com>
parent: Wells Oliver <wells.oliver@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: Ron @ 2021-10-06 03:05 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 10/5/21 5:06 PM, Wells Oliver wrote:
> Hi-- to keep it short, I feel like I've generally heard the larger your DB
> is, the less efficient it might run, likely due to disk I/O. Maybe I'm
> terribly mistaken in this perception.
Less *efficient*???
Of course /ceteris paribus/ it takes longer to do "whole database"
activities on a larger database because... there's more data, but that
should be obvious to all.
So you scale up the IO (SAN, RAID 50, faster disks, etc) or partition the
data *or both*.
>
> We have two DBs, one primarily accessed by humans and systems, which is
> ~1TB in size, and aggregates most of what we store in raw, longer format
> on a second DB that is about ~6TB in size.
>
> As we consider plans to migrate to RDS, we've talked a lot about
> combining the two as more and more the case is querying the larger DB and
> wanting data only available in the smaller DB.
>
> Of course, we can solve this by copying things back and forth, but we're
> also thinking: why not just one big DB?
>
> Anyone have any experiences with a similar project, and especially any
> technical configurations that might be beneficial in using RDS?
Nothing like your scenario, but we recently migrated an 8TB Oracle database
hosted on a VM + (big fancy) SAN to a 6TB RDS Postgresql database. Reads
are faster, and writes seem a bit slower, but enough software changed that
comparisons are dubious.
One thing I /can/ say is, "watch your costs!!" Amazon does an excellent job
of obfuscating that, hiding it from everyone but the accountants. It's
dreadfully easy for technical people to unknowingly spend tens or hundreds
of thousands of dollars in a month when testing things.
--
Angular momentum makes the world go 'round.
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Pros/cons of big databases vs smaller databases and RDS
@ 2021-10-06 03:28 Pradeep Kumar <pradeep.kumar.pc@gmail.com>
parent: Wells Oliver <wells.oliver@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: Pradeep Kumar @ 2021-10-06 03:28 UTC (permalink / raw)
To: Wells Oliver <wells.oliver@gmail.com>; +Cc: pgsql-admin
You can opt for Aws *Provisioned IOPS storage is designed to meet the
needs of I/O-intensive workloads, *
On Wed, Oct 6, 2021, 3:37 AM Wells Oliver <wells.oliver@gmail.com> wrote:
> Hi-- to keep it short, I feel like I've generally heard the larger your DB
> is, the less efficient it might run, likely due to disk I/O. Maybe I'm
> terribly mistaken in this perception.
>
> We have two DBs, one primarily accessed by humans and systems, which is
> ~1TB in size, and aggregates most of what we store in raw, longer format on
> a second DB that is about ~6TB in size.
>
> As we consider plans to migrate to RDS, we've talked a lot about
> combining the two as more and more the case is querying the larger DB and
> wanting data only available in the smaller DB.
>
> Of course, we can solve this by copying things back and forth, but we're
> also thinking: why not just one big DB?
>
> Anyone have any experiences with a similar project, and especially any
> technical configurations that might be beneficial in using RDS?
>
> Appreciate it.
>
> --
> Wells Oliver
> wells.oliver@gmail.com <wellsoliver@gmail.com>
>
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2021-10-06 03:28 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-05 22:06 Pros/cons of big databases vs smaller databases and RDS Wells Oliver <wells.oliver@gmail.com>
2021-10-06 03:05 ` Ron <ronljohnsonjr@gmail.com>
2021-10-06 03:28 ` Pradeep Kumar <pradeep.kumar.pc@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox