Subject: Re: Pros/cons of big databases vs smaller databases and RDS
To: pgsql-admin@lists.postgresql.org
References: 
 <CAOC+FBUWjYqef5KOCcSh4OUi7d2EcStMTgwt6jJwGeChGRvopA@mail.gmail.com>
From: Ron <ronljohnsonjr@gmail.com>
Message-ID: <ff999d1b-bfe9-111c-8c49-69caa1417d32@gmail.com>
Date: Tue, 5 Oct 2021 22:05:19 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: 
 <CAOC+FBUWjYqef5KOCcSh4OUi7d2EcStMTgwt6jJwGeChGRvopA@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------1C54B9D3D4AFDEDAF0227C74"
Content-Language: en-US
Archived-At: 
 <https://www.postgresql.org/message-id/ff999d1b-bfe9-111c-8c49-69caa1417d32%40gmail.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------1C54B9D3D4AFDEDAF0227C74
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

On 10/5/21 5:06 PM, Wells Oliver wrote:
> Hi-- to keep it short, I feel like I've generally heard the larger your DB 
> is, the less efficient it might run, likely due to disk I/O. Maybe I'm 
> terribly mistaken in this perception.

Less *efficient*???

Of course /ceteris paribus/ it takes longer to do "whole database" 
activities on a larger database because... there's more data, but that 
should be obvious to all.

So you scale up the IO (SAN, RAID 50, faster disks, etc)  or partition the 
data *or both*.

>
> We have two DBs, one primarily accessed by humans and systems, which is 
> ~1TB in size, and aggregates most of what we store in raw, longer format 
> on a second DB that is about ~6TB in size.
>
> As we consider plans to migrate to RDS, we've talked a lot about 
> combining the two as more and more the case is querying the larger DB and 
> wanting data only available in the smaller DB.
>
> Of course, we can solve this by copying things back and forth, but we're 
> also thinking: why not just one big DB?
>
> Anyone have any experiences with a similar project, and especially any 
> technical configurations that might be beneficial in using RDS?

Nothing like your scenario, but we recently migrated an 8TB Oracle database 
hosted on a VM + (big fancy) SAN to a 6TB RDS Postgresql database.  Reads 
are faster, and writes seem a bit slower, but enough software changed that 
comparisons are dubious.

One thing I /can/ say is, "watch your costs!!"  Amazon does an excellent job 
of obfuscating that, hiding it from everyone but the accountants.  It's 
dreadfully easy for technical people to unknowingly spend tens or hundreds 
of thousands of dollars in a month when testing things.

-- 
Angular momentum makes the world go 'round.

--------------1C54B9D3D4AFDEDAF0227C74
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    On 10/5/21 5:06 PM, Wells Oliver wrote:<br>
    <blockquote type="cite"
cite="mid:CAOC+FBUWjYqef5KOCcSh4OUi7d2EcStMTgwt6jJwGeChGRvopA@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">Hi-- to keep it short, I feel like I've generally
        heard the larger your DB is, the less efficient it might run,
        likely due to disk I/O. Maybe I'm terribly mistaken in this
        perception.</div>
    </blockquote>
    <br>
    Less *efficient*???<br>
    <br>
    Of course <i>ceteris paribus</i> it takes longer to do "whole
    database" activities on a larger database because... there's more
    data, but that should be obvious to all.<br>
    <br>
    So you scale up the IO (SAN, RAID 50, faster disks, etc)  or
    partition the data <b>or both</b>.<br>
    <br>
    <blockquote type="cite"
cite="mid:CAOC+FBUWjYqef5KOCcSh4OUi7d2EcStMTgwt6jJwGeChGRvopA@mail.gmail.com">
      <div dir="ltr">
        <div><br>
        </div>
        <div>We have two DBs, one primarily accessed by humans and
          systems, which is ~1TB in size, and aggregates most of what we
          store in raw, longer format on a second DB that is about ~6TB
          in size.</div>
        <div><br>
        </div>
        <div>As we consider plans to migrate to RDS, we've talked a lot
          about combining the two as more and more the case is querying
          the larger DB and wanting data only available in the smaller
          DB.</div>
        <div><br>
        </div>
        <div>Of course, we can solve this by copying things back and
          forth, but we're also thinking: why not just one big DB?</div>
        <div><br>
        </div>
        <div>Anyone have any experiences with a similar project, and
          especially any technical configurations that might be
          beneficial in using RDS?</div>
      </div>
    </blockquote>
    <br>
    Nothing like your scenario, but we recently migrated an 8TB Oracle
    database hosted on a VM + (big fancy) SAN to a 6TB RDS Postgresql
    database.  Reads are faster, and writes seem a bit slower, but
    enough software changed that comparisons are dubious.<br>
    <br>
    One thing I <i>can</i> say is, "watch your costs!!"  Amazon does an
    excellent job of obfuscating that, hiding it from everyone but the
    accountants.  It's dreadfully easy for technical people to
    unknowingly spend tens or hundreds of thousands of dollars in a
    month when testing things.<br>
    <br>
    <div class="moz-signature">-- <br>
      Angular momentum makes the world go 'round.</div>
  </body>
</html>

--------------1C54B9D3D4AFDEDAF0227C74--