public inbox for [email protected]
help / color / mirror / Atom feedFrom: Pilling, Michael DR <[email protected]>
To: Laurenz Albe <[email protected]>
To: Onni Hakala <[email protected]>
To: [email protected] <[email protected]>
Subject: RE: Recommendations on how to combine SSD and HDD drives in bare metal PostgreSQL server [SEC=UNOFFICIAL]
Date: Thu, 17 Oct 2024 03:26:51 +0000
Message-ID: <ME3P282MB393552D2683C9FA24594EC15CA472@ME3P282MB3935.AUSP282.PROD.OUTLOOK.COM> (raw)
UNOFFICIAL
I would say the best way to handle this would be to do it in the hardware drivers or file system components of the OS so that the SSD drives were used as a cache for the HDDs. This would generally result in all the active indices of the tables residing on the SSDs. It all depends whether your OS and its files systems are to the job. Certainly, Reisier4 file system aimed for all of its indices to be held in memory with only writes going to the HDDs - by using the SSDs for swap space this would have the desired effect under Reiser4.
Dr Michael Pilling (him / they)
Defence Analyst / Software Specialist
Human and Decision Sciences Division | Analytical Wargaming | Capability Analysis and Design
Defence Science and Technology Group
_____________________________________________________________________________
Department of Defence | Edinburgh | 23L Bay 35
PO Box 1500 | EDINBURGH SA 5111
M: 0414 405 711 | www.defence.gov.au
E: [email protected]
Defence acknowledges the Traditional Custodians of the Country throughout Australia. We recognise their continuing connection to land, waters and community. We pay our respects to them, their culture and to their Elders past and present.
-----Original Message-----
From: Laurenz Albe <[email protected]>
Sent: Thursday, 17 October 2024 2:53 AM
To: Onni Hakala <[email protected]>; [email protected]
Subject: Re: Recommendations on how to combine SSD and HDD drives in bare metal PostgreSQL server
EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe.
On Wed, 2024-10-16 at 17:06 +0300, Onni Hakala wrote:
> I have a large dataset of > 100TB which would be very expensive to store solely into SSD drives.
>
> I have access to a server which has 2x 3.84TB NVME SSD disks and large array of HDD drives 8 x 22TB.
>
> Most of the data that I have in my dataset is very rarely accessed and is stored only for archival purposes.
>
> What would be the de-facto way to use both SSD and HDD together in a
> way use where commonly used data would be fast to access and old data would eventually only be stored in compressed format in the HDDs?
>
> I was initially looking into building zpool using zfs with raidz3 and
> zstd compression for my HDDs but I’m unsure how to add the SSDs into
> this equation and I thought that this is probably a common scenario and wanted to ask opinions from here.
The traditionalway to handle that would be to create an extra tablespace on a file system built on the slow disks. You'd move tables that are no longer hot to that slow tablespace.
If you cannot move the whole table, use partitioning and move old partitions to the slow tablespace.
Yours,
Laurenz Albe
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: RE: Recommendations on how to combine SSD and HDD drives in bare metal PostgreSQL server [SEC=UNOFFICIAL]
In-Reply-To: <ME3P282MB393552D2683C9FA24594EC15CA472@ME3P282MB3935.AUSP282.PROD.OUTLOOK.COM>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox