pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedZFS vs. UFS
12+ messages / 6 participants
[nested] [flat]
* ZFS vs. UFS
@ 2012-07-24 12:51 Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 13:03 ` Re: ZFS vs. UFS Georgi Naplatanov <gosho@oles.biz>
2012-07-24 13:18 ` Re: ZFS vs. UFS Ivan Voras <ivoras@freebsd.org>
2012-07-24 14:23 ` Re: ZFS vs. UFS Torsten Zuehlsdorff <foo@meisterderspiele.de>
2012-07-27 05:24 ` Re: ZFS vs. UFS Greg Smith <greg@2ndQuadrant.com>
0 siblings, 4 replies; 12+ messages in thread
From: Laszlo Nagy @ 2012-07-24 12:51 UTC (permalink / raw)
To: pgsql-performance
Hello,
Under FreeBSD 9, what filesystem should I use for PostgreSQL? (Dell
PowerEdge 2900, 24G mem, 10x2T SATA2 disk, Intel RAID controller.)
* ZFS is journaled, and it is more independent of the hardware. So if
the computer goes wrong, I can move the zfs array to a different server.
* UFS is not journaled. Also I have to rely on the RAID card to build
the RAID array. If there is a hw problem with it, then I won't be
able to recover the data easily.
I wonder if UFS has better performance or not. Or can you suggest
another fs? Just of the PGDATA directory.
Thanks,
Laszlo
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
@ 2012-07-24 13:03 ` Georgi Naplatanov <gosho@oles.biz>
3 siblings, 0 replies; 12+ messages in thread
From: Georgi Naplatanov @ 2012-07-24 13:03 UTC (permalink / raw)
To: Laszlo Nagy <gandalf@shopzeus.com>; +Cc: pgsql-performance
Hi.
As far as I know UFS is faster than ZFS on FreeBSD 9.0.
Some users reported stability problem with ZFS on AMD64 and maybe UFS is
better choice.
Best regards
Georgi
On 07/24/2012 03:51 PM, Laszlo Nagy wrote:
>
> Hello,
>
> Under FreeBSD 9, what filesystem should I use for PostgreSQL? (Dell
> PowerEdge 2900, 24G mem, 10x2T SATA2 disk, Intel RAID controller.)
>
> * ZFS is journaled, and it is more independent of the hardware. So
> if the computer goes wrong, I can move the zfs array to a
> different server.
> * UFS is not journaled. Also I have to rely on the RAID card to
> build the RAID array. If there is a hw problem with it, then I
> won't be able to recover the data easily.
>
> I wonder if UFS has better performance or not. Or can you suggest
> another fs? Just of the PGDATA directory.
>
> Thanks,
>
> Laszlo
>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
@ 2012-07-24 13:18 ` Ivan Voras <ivoras@freebsd.org>
2012-07-24 14:34 ` Re: ZFS vs. UFS Craig James <cjames@emolecules.com>
3 siblings, 1 reply; 12+ messages in thread
From: Ivan Voras @ 2012-07-24 13:18 UTC (permalink / raw)
To: pgsql-performance
On 24/07/2012 14:51, Laszlo Nagy wrote:
>
> Hello,
>
> Under FreeBSD 9, what filesystem should I use for PostgreSQL? (Dell
> PowerEdge 2900, 24G mem, 10x2T SATA2 disk, Intel RAID controller.)
>
> * ZFS is journaled, and it is more independent of the hardware. So if
> the computer goes wrong, I can move the zfs array to a different server.
> * UFS is not journaled. Also I have to rely on the RAID card to build
> the RAID array. If there is a hw problem with it, then I won't be
> able to recover the data easily.
>
> I wonder if UFS has better performance or not. Or can you suggest
> another fs? Just of the PGDATA directory.
Hi,
I think you might actually get a bit more performance out of ZFS,
depending on your load, server configuration and (more so) the tuning of
ZFS... however UFS is IMO more stable so I use it more often. A hardware
RAID card would be good to have, but you can use soft-RAID the usual way
and not be locked-in by the controller.
You can activate softupdates-journalling on UFS if you really want it,
but I find that regular softupdates is perfectly fine for PostgreSQL,
which has its own journalling.
Attachments:
signature.asc (application/pgp-signature, 259B) download
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 13:18 ` Re: ZFS vs. UFS Ivan Voras <ivoras@freebsd.org>
@ 2012-07-24 14:34 ` Craig James <cjames@emolecules.com>
2012-07-24 18:27 ` Re: ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
0 siblings, 1 reply; 12+ messages in thread
From: Craig James @ 2012-07-24 14:34 UTC (permalink / raw)
To: Ivan Voras <ivoras@freebsd.org>; +Cc: pgsql-performance
>
> On 24/07/2012 14:51, Laszlo Nagy wrote:
> >
> > Hello,
> >
> > Under FreeBSD 9, what filesystem should I use for PostgreSQL? (Dell
> > PowerEdge 2900, 24G mem, 10x2T SATA2 disk, Intel RAID controller.)
> >
> > * ZFS is journaled, and it is more independent of the hardware. So if
> > the computer goes wrong, I can move the zfs array to a different
> server.
> > * UFS is not journaled. Also I have to rely on the RAID card to build
> > the RAID array. If there is a hw problem with it, then I won't be
> > able to recover the data easily.
> >
> > I wonder if UFS has better performance or not. Or can you suggest
> > another fs? Just of the PGDATA directory.
>
Relying on physically moving a disk isn't a good backup/recovery strategy.
Disks are the least reliable single component in a modern computer. You
should figure out the best file system for your application, and separately
figure out a recovery strategy, one that can survive the failure of *any*
component in your system, including the disk itself.
Craig
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 13:18 ` Re: ZFS vs. UFS Ivan Voras <ivoras@freebsd.org>
2012-07-24 14:34 ` Re: ZFS vs. UFS Craig James <cjames@emolecules.com>
@ 2012-07-24 18:27 ` Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 19:14 ` Re: ZFS vs. UFS Craig James <cjames@emolecules.com>
0 siblings, 1 reply; 12+ messages in thread
From: Laszlo Nagy @ 2012-07-24 18:27 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; pgsql-performance
> > I wonder if UFS has better performance or not. Or can you suggest
> > another fs? Just of the PGDATA directory.
>
>
> Relying on physically moving a disk isn't a good backup/recovery
> strategy. Disks are the least reliable single component in a modern
> computer. You should figure out the best file system for your
> application, and separately figure out a recovery strategy, one that
> can survive the failure of *any* component in your system, including
> the disk itself.
This is why I use a RAID array of 10 disks. So there is no single point
of failure. What else could I do? (Yes, I can make regular backups, but
that is not the same. I can still loose data...)
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 13:18 ` Re: ZFS vs. UFS Ivan Voras <ivoras@freebsd.org>
2012-07-24 14:34 ` Re: ZFS vs. UFS Craig James <cjames@emolecules.com>
2012-07-24 18:27 ` Re: ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
@ 2012-07-24 19:14 ` Craig James <cjames@emolecules.com>
0 siblings, 0 replies; 12+ messages in thread
From: Craig James @ 2012-07-24 19:14 UTC (permalink / raw)
To: Laszlo Nagy <gandalf@shopzeus.com>; +Cc: pgsql-performance
On Tue, Jul 24, 2012 at 11:27 AM, Laszlo Nagy <gandalf@shopzeus.com> wrote:
>
> > I wonder if UFS has better performance or not. Or can you suggest
>> > another fs? Just of the PGDATA directory.
>>
>
> Relying on physically moving a disk isn't a good backup/recovery
> strategy. Disks are the least reliable single component in a modern
> computer. You should figure out the best file system for your application,
> and separately figure out a recovery strategy, one that can survive the
> failure of *any* component in your system, including the disk itself.
>
> This is why I use a RAID array of 10 disks. So there is no single point of
> failure. What else could I do? (Yes, I can make regular backups, but that
> is not the same. I can still loose data...)
>
Only you can answer that because it depends on your application. If you're
operating PayPal, you probably want 24/7 100% reliability. If you're
operating a social networking site for teenagers, losing data is probably
not a catastrophe.
In my experience, most data loss is NOT from equipment failure. It's from
software bugs and operator errors. If your recovery plan doesn't cover
this, you have a problem.
Craig
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
@ 2012-07-24 14:23 ` Torsten Zuehlsdorff <foo@meisterderspiele.de>
2012-07-24 18:35 ` Re: ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
3 siblings, 1 reply; 12+ messages in thread
From: Torsten Zuehlsdorff @ 2012-07-24 14:23 UTC (permalink / raw)
To: pgsql-performance
On 24.07.2012 14:51, Laszlo Nagy wrote:
> * UFS is not journaled.
There is journal support for UFS as far as i know. Please have a look at
the gjournal manpage.
Greetings,
Torsten
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 14:23 ` Re: ZFS vs. UFS Torsten Zuehlsdorff <foo@meisterderspiele.de>
@ 2012-07-24 18:35 ` Laszlo Nagy <gandalf@shopzeus.com>
2012-07-25 07:00 ` Re: ZFS vs. UFS Torsten Zuehlsdorff <foo@meisterderspiele.de>
0 siblings, 1 reply; 12+ messages in thread
From: Laszlo Nagy @ 2012-07-24 18:35 UTC (permalink / raw)
To: Torsten Zuehlsdorff <foo@meisterderspiele.de>; pgsql-performance
> On 24.07.2012 14:51, Laszlo Nagy wrote:
>
>> * UFS is not journaled.
>
> There is journal support for UFS as far as i know. Please have a look
> at the gjournal manpage.
Yes, but gjournal works for disk devices. I would have rely on the hw
card for RAID. When the card goes wrong I won't be able to access my data.
I could also buy an identical RAID card. In fact I could buy a complete
backup server. But right now I don't have the money for that. So I would
like to use a solution that allows me to recover from a failure even if
the RAID card goes wrong.
It might also be possible to combine gmirror + gjournal, but that is not
good enough. Performance and stability of a simple gmirror with two
disks is much worse then a raidz array with 10 disks (and hot spare), or
even a raid 1+0 (and hot spare) that is supported by the hw RAID card.
So I would like to stick with UFS+hw card support (and then I need to
buy an identical RAID card if I can), or ZFS.
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 14:23 ` Re: ZFS vs. UFS Torsten Zuehlsdorff <foo@meisterderspiele.de>
2012-07-24 18:35 ` Re: ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
@ 2012-07-25 07:00 ` Torsten Zuehlsdorff <foo@meisterderspiele.de>
0 siblings, 0 replies; 12+ messages in thread
From: Torsten Zuehlsdorff @ 2012-07-25 07:00 UTC (permalink / raw)
To: pgsql-performance
>> On 24.07.2012 14:51, Laszlo Nagy wrote:
>>
>>> * UFS is not journaled.
>>
>> There is journal support for UFS as far as i know. Please have a look
>> at the gjournal manpage.
>
> Yes, but gjournal works for disk devices.
That isn't completly correct! gjournal works with all GEOM-devices,
which could be not only disk devices, but also (remote) disk devices,
(remote) files, (remote) software-raids etc.
It is very easy to mirror the *complete* disk from one *server* to
another. I use this technic for customers which need cheap backups of
their complete server.
But a RAID card will be much faster than this. I just wanted to make
this clear.
Greetings,
Torsten
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
@ 2012-07-27 05:24 ` Greg Smith <greg@2ndQuadrant.com>
2012-07-31 08:50 ` Re: ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
3 siblings, 1 reply; 12+ messages in thread
From: Greg Smith @ 2012-07-27 05:24 UTC (permalink / raw)
To: Laszlo Nagy <gandalf@shopzeus.com>; +Cc: pgsql-performance
On 07/24/2012 08:51 AM, Laszlo Nagy wrote:
> Under FreeBSD 9, what filesystem should I use for PostgreSQL? (Dell
> PowerEdge 2900, 24G mem, 10x2T SATA2 disk, Intel RAID controller.)
When Intel RAID controller is that? All of the ones on the motherboard
are pretty much useless if that's what you have. Those are slower than
software RAID and it's going to add driver issues you could otherwise
avoid. Better to connect the drives to the non-RAID ports or configure
the controller in JBOD mode first.
Using one of the better RAID controllers, one of Dell's good PERC models
for example, is one of the biggest hardware upgrades you could make to
this server. If your database is mostly read traffic, it won't matter
very much. Write-heavy loads really benefit from a good RAID
controller's write cache.
> * ZFS is journaled, and it is more independent of the hardware. So
> if the computer goes wrong, I can move the zfs array to a
> different server.
> * UFS is not journaled. Also I have to rely on the RAID card to
> build the RAID array. If there is a hw problem with it, then I
> won't be able to recover the data easily.
>
You should be able to get UFS working with a software mirror and
journaling using gstripe/gmirror or vinum. It doesn't matter that much
for PostgreSQL though. The data writes are journaled by the database,
and it tries to sync data to disk after updating metadata too. There
are plenty of PostgreSQL installs on FreeBSD/UFS that work fine.
ZFS needs more RAM and has higher CPU overhead than UFS does. It's a
heavier filesystem all around than UFS is. Your server is fast enough
that you should be able to afford it though, and the feature set is
nice. In addition to the RAID setup being simple to handle, having
checksums on your data is a good safety feature for PostgreSQL.
ZFS will heavily use server RAM for caching by default, much more so
than UFS. Make sure you check into that, and leave enough RAM for the
database to run too. (Doing *some* caching that way is good for
Postgres; you just don't want *all* the memory to be used for that)
Moving disks to another server is a very low probability fix for a
broken system. The disks are a likely place for the actual failure to
happen at in the first place. I like to think more in terms of "how can
I create a real-time replica of this data?" to protect databases, and
the standby server for that doesn't need to be an expensive system.
That said, there is no reason to set things up so that they only work
with that Intel RAID controller, given that it's not a very good piece
of hardware anyway.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
2012-07-27 05:24 ` Re: ZFS vs. UFS Greg Smith <greg@2ndQuadrant.com>
@ 2012-07-31 08:50 ` Laszlo Nagy <gandalf@shopzeus.com>
2012-07-31 14:33 ` Re: ZFS vs. UFS Craig James <cjames@emolecules.com>
0 siblings, 1 reply; 12+ messages in thread
From: Laszlo Nagy @ 2012-07-31 08:50 UTC (permalink / raw)
To: Greg Smith <greg@2ndQuadrant.com>; +Cc: pgsql-performance
> When Intel RAID controller is that? All of the ones on the
> motherboard are pretty much useless if that's what you have. Those are
> slower than software RAID and it's going to add driver issues you
> could otherwise avoid. Better to connect the drives to the non-RAID
> ports or configure the controller in JBOD mode first.
>
> Using one of the better RAID controllers, one of Dell's good PERC
> models for example, is one of the biggest hardware upgrades you could
> make to this server. If your database is mostly read traffic, it
> won't matter very much. Write-heavy loads really benefit from a good
> RAID controller's write cache.
Actually, it is a PERC with write-cache and BBU.
>
> ZFS will heavily use server RAM for caching by default, much more so
> than UFS. Make sure you check into that, and leave enough RAM for the
> database to run too. (Doing *some* caching that way is good for
> Postgres; you just don't want *all* the memory to be used for that)
Right now, the size of the database is below 5GB. So I guess it will fit
into memory. I'm concerned about data safety and availability. I have
been in a situation where the RAID card went wrong and I was not able to
recover the data because I could not get an identical RAID card in time.
I have also been in a situation where the system was crashing two times
a day, and we didn't know why. (As it turned out, it was a bug in the
"stable" kernel and we could not identify this for two weeks.) However,
we had to do fsck after every crash. With a 10TB disk array, it was
extremely painful. ZFS is much better: short recovery time and it is
RAID card independent. So I think I have answered my own question - I'm
going to use ZFS to have better availability, even if it leads to poor
performance. (That was the original question: how bad it it to use ZFS
for PostgreSQL, instead of the native UFS.)
>
> Moving disks to another server is a very low probability fix for a
> broken system. The disks are a likely place for the actual failure to
> happen at in the first place.
Yes, but we don't have to worry about that. raidz2 + hot spare is safe
enough. The RAID card is the only single point of failure.
> I like to think more in terms of "how can I create a real-time replica
> of this data?" to protect databases, and the standby server for that
> doesn't need to be an expensive system. That said, there is no reason
> to set things up so that they only work with that Intel RAID
> controller, given that it's not a very good piece of hardware anyway.
I'm not sure how to create a real-time replica. This database is updated
frequently. There is always a process that reads/writes into the
database. I was thinking about using slony to create slave databases. I
have no experience with that. We have a 100Mbit connection. I'm not sure
how much bandwidth we need to maintain a real-time slave database. It
might be a good idea.
I'm sorry, I feel I'm being off-topic.
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: ZFS vs. UFS
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
2012-07-27 05:24 ` Re: ZFS vs. UFS Greg Smith <greg@2ndQuadrant.com>
2012-07-31 08:50 ` Re: ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
@ 2012-07-31 14:33 ` Craig James <cjames@emolecules.com>
0 siblings, 0 replies; 12+ messages in thread
From: Craig James @ 2012-07-31 14:33 UTC (permalink / raw)
To: Laszlo Nagy <gandalf@shopzeus.com>; +Cc: Greg Smith <greg@2ndquadrant.com>; pgsql-performance
On Tue, Jul 31, 2012 at 1:50 AM, Laszlo Nagy <gandalf@shopzeus.com> wrote:
>
> When Intel RAID controller is that? All of the ones on the motherboard
>> are pretty much useless if that's what you have. Those are slower than
>> software RAID and it's going to add driver issues you could otherwise
>> avoid. Better to connect the drives to the non-RAID ports or configure the
>> controller in JBOD mode first.
>>
>> Using one of the better RAID controllers, one of Dell's good PERC models
>> for example, is one of the biggest hardware upgrades you could make to this
>> server. If your database is mostly read traffic, it won't matter very
>> much. Write-heavy loads really benefit from a good RAID controller's write
>> cache.
>>
> Actually, it is a PERC with write-cache and BBU.
>
Last time I checked, "PERC" was a meaningless name. Dell put that label on
a variety of different controllers ... some were quite good, some were
terrible. The latest PERC controllers are pretty good. If your machine is
a few years old, the PERC controller may be a piece of junk.
Craig
>
>> ZFS will heavily use server RAM for caching by default, much more so than
>> UFS. Make sure you check into that, and leave enough RAM for the database
>> to run too. (Doing *some* caching that way is good for Postgres; you just
>> don't want *all* the memory to be used for that)
>>
> Right now, the size of the database is below 5GB. So I guess it will fit
> into memory. I'm concerned about data safety and availability. I have been
> in a situation where the RAID card went wrong and I was not able to recover
> the data because I could not get an identical RAID card in time. I have
> also been in a situation where the system was crashing two times a day, and
> we didn't know why. (As it turned out, it was a bug in the "stable" kernel
> and we could not identify this for two weeks.) However, we had to do fsck
> after every crash. With a 10TB disk array, it was extremely painful. ZFS is
> much better: short recovery time and it is RAID card independent. So I
> think I have answered my own question - I'm going to use ZFS to have better
> availability, even if it leads to poor performance. (That was the original
> question: how bad it it to use ZFS for PostgreSQL, instead of the native
> UFS.)
>
>>
>> Moving disks to another server is a very low probability fix for a broken
>> system. The disks are a likely place for the actual failure to happen at
>> in the first place.
>>
> Yes, but we don't have to worry about that. raidz2 + hot spare is safe
> enough. The RAID card is the only single point of failure.
>
>> I like to think more in terms of "how can I create a real-time replica of
>> this data?" to protect databases, and the standby server for that doesn't
>> need to be an expensive system. That said, there is no reason to set
>> things up so that they only work with that Intel RAID controller, given
>> that it's not a very good piece of hardware anyway.
>>
> I'm not sure how to create a real-time replica. This database is updated
> frequently. There is always a process that reads/writes into the database.
> I was thinking about using slony to create slave databases. I have no
> experience with that. We have a 100Mbit connection. I'm not sure how much
> bandwidth we need to maintain a real-time slave database. It might be a
> good idea.
>
> I'm sorry, I feel I'm being off-topic.
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.**
> org <pgsql-performance@postgresql.org>)
> To make changes to your subscription:
> http://www.postgresql.org/**mailpref/pgsql-performance<http://www.postgresql.org/mailpref/pgsql-p...;
>
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2012-07-31 14:33 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-07-24 12:51 ZFS vs. UFS Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 13:03 ` Georgi Naplatanov <gosho@oles.biz>
2012-07-24 13:18 ` Ivan Voras <ivoras@freebsd.org>
2012-07-24 14:34 ` Craig James <cjames@emolecules.com>
2012-07-24 18:27 ` Laszlo Nagy <gandalf@shopzeus.com>
2012-07-24 19:14 ` Craig James <cjames@emolecules.com>
2012-07-24 14:23 ` Torsten Zuehlsdorff <foo@meisterderspiele.de>
2012-07-24 18:35 ` Laszlo Nagy <gandalf@shopzeus.com>
2012-07-25 07:00 ` Torsten Zuehlsdorff <foo@meisterderspiele.de>
2012-07-27 05:24 ` Greg Smith <greg@2ndQuadrant.com>
2012-07-31 08:50 ` Laszlo Nagy <gandalf@shopzeus.com>
2012-07-31 14:33 ` Craig James <cjames@emolecules.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox