Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uckdJ-00GeUQ-60 for pgsql-general@arkaria.postgresql.org; Fri, 18 Jul 2025 12:56:13 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1uckdH-00CB2a-7x for pgsql-general@arkaria.postgresql.org; Fri, 18 Jul 2025 12:56:11 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uckdG-00CB2S-L7 for pgsql-general@lists.postgresql.org; Fri, 18 Jul 2025 12:56:11 +0000 Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1uckdE-007wYB-2z for pgsql-general@lists.postgresql.org; Fri, 18 Jul 2025 12:56:10 +0000 Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-b34a78bb6e7so1563555a12.3 for ; Fri, 18 Jul 2025 05:56:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1752843368; x=1753448168; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/tK8ruGNzf5ENygo6lP3q372r8uiajmflF0DfrERlXw=; b=CZW1DAluUWpkVDZR3zjSy2myYxRtVLKQCR8xirtFX9CQVuVATnVa+FdkSKSZ9n4Qn6 hwiMeveIU6H2PUauPoDA/J3/VQGQLSm1gaO5emAQxBagwHTzWUtJqI9pimW4eckZnixe rE1sW6xgiam+MZADjLe/j9JlS5i+WhRBeA/MVCoeXS13biv6QrFH0G8QdIzjlrtTVJoH Dv0hbqDCSz07ipYTt0463fUxSlRP3g/31IsaqVoWYrMXwVMDPgt7AB0JfDDB/F26oBgs 6jfPYZsGbrs+8qa12NfF9pbSpeV9P8auJ4iz7iVCUUNrRP72eSwxRf1FS9estDNn02w7 FODQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752843368; x=1753448168; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/tK8ruGNzf5ENygo6lP3q372r8uiajmflF0DfrERlXw=; b=ad0idtw3jglBi4GeJR98Ayt4wA/2o7dRDhsDLljTbxW+bmT36O+VStU4weTGE8zakC stDg4QG1bQojnF4yBN04dZ1bwIm+PU2dQicBQ8PMjkaHio7gJnyA1QQXVMrVP5Ev4Kt2 Obevzt9akWZQjOO5Ndt6hLsvxohfOEi1XAPO43RzoX69zjKWzHGGowJEmasCo5K7NUyr PBwwe4ccUqZ2dKE+o/g8kVMkhI79NGHP8Mw8F0cwdQ0EIPiWDa1HSoff0aviyUiNnJIC 1NwIw54QzeuXG/X28BoNkceC+w23nMPPg+whBhEKyBnFOc2dEegAv4CHLxgCtnzXRoi3 MsSw== X-Gm-Message-State: AOJu0Yza2GwmF9ZdC3SAsJ+kHjiRXTKkzOXI+z6VL1p4yO8TeZO7EVok qViX4n1c9f3FmLO8CGwfFWit2Fy+wDWGrnXMeZtN4xya9nPy9ca2kJ7hIFB7KaX2QC8rHPg7Xmm A/oFvZYFRNQsdnQABK3wZLzQr+lbDZ3Q= X-Gm-Gg: ASbGncvlM70HYdTPOPbxkhZCnR5aZKdI2l+BCdkwfkSCKxA2CvBMhnp9FRIluLT1aFm G1Onucmof9cR0O7pnBFvTo/Tt4yel+3oJLmXBClzniqJtQXj1wpkQ1YehX9z1EgX9XPYRJ/0Qba cLmYXPW6s/ObktaIZZRQgpJzFxbK3sg/JQlf7D+KjHpFROjX27sPGcaOWbxLBCgheknIe03VO6A 1gv X-Google-Smtp-Source: AGHT+IHYHh6mQuTuTYtqhj/B7rpkm7wHKH09Q0ZsXMvQJqKw8o89MwVnbKDdUYOmSO6IOodDv1AwBtjWHr49vZfsYIU= X-Received: by 2002:a17:90b:53cf:b0:311:b0ec:1360 with SMTP id 98e67ed59e1d1-31c9f447e13mr15085724a91.29.1752843367726; Fri, 18 Jul 2025 05:56:07 -0700 (PDT) MIME-Version: 1.0 References: <8188513c-e089-4273-b2be-16dd0a5a0a80@app.fastmail.com> In-Reply-To: <8188513c-e089-4273-b2be-16dd0a5a0a80@app.fastmail.com> From: Seref Arikan Date: Fri, 18 Jul 2025 13:55:56 +0100 X-Gm-Features: Ac12FXyTehYVJcgpMueCbSlytNvQvP375STDe_b9DkWpGdkssS1Q8k5OTuqa8Ok Message-ID: Subject: Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance To: Pierre Barre Cc: pgsql-general@lists.postgresql.org Content-Type: multipart/alternative; boundary="00000000000047c2ae063a33a653" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --00000000000047c2ae063a33a653 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks, I learned something else: I didn't know Hetzner offered S3 compatible storage. The interesting thing is, a few searches about the performance return mostly negative impressions about their object storage in comparison to the original S3. Finding out what kind of performance your benchmarks would yield on a pure AWS setting would be interesting. I am not asking you to do that, but you may get even better performance in that case :) Cheers, Seref On Fri, Jul 18, 2025 at 11:58=E2=80=AFAM Pierre Barre wro= te: > Hi Seref, > > For the benchmarks, I used Hetzner's cloud service with the following > setup: > > - A Hetzner s3 bucket in the FSN1 region > - A virtual machine of type ccx63 48 vCPU 192 GB memory > - 3 ZeroFS nbd devices (same s3 bucket) > - A ZFS stripped pool with the 3 devices > - 200GB zfs L2ARC > - Postgres configured accordingly memory-wise as well as with > synchronous_commit =3D off, wal_init_zero =3D off and wal_recycle =3D off= . > > Best, > Pierre > > On Fri, Jul 18, 2025, at 12:42, Seref Arikan wrote: > > Sorry, this was meant to go to the whole group: > > Very interesting!. Great work. Can you clarify how exactly you're running > postgres in your tests? A specific AWS service? What's the test > infrastructure that sits above the file system? > > On Thu, Jul 17, 2025 at 11:59=E2=80=AFPM Pierre Barre w= rote: > > Hi everyone, > > I wanted to share a project I've been working on that enables PostgreSQL > to run on S3 storage while maintaining performance comparable to local > NVMe. The approach uses block-level access rather than trying to map > filesystem operations to S3 objects. > > ZeroFS: https://github.com/Barre/ZeroFS > > # The Architecture > > ZeroFS provides NBD (Network Block Device) servers that expose S3 storage > as raw block devices. PostgreSQL runs unmodified on ZFS pools built on > these block devices: > > PostgreSQL -> ZFS -> NBD -> ZeroFS -> S3 > > By providing block-level access and leveraging ZFS's caching capabilities > (L2ARC), we can achieve microsecond latencies despite the underlying > storage being in S3. > > ## Performance Results > > Here are pgbench results from PostgreSQL running on this setup: > > ### Read/Write Workload > > ``` > postgres@ubuntu-16gb-fsn1-1:/root$ pgbench -c 50 -j 15 -t 100000 example > pgbench (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1)) > starting vacuum...end. > transaction type: > scaling factor: 50 > query mode: simple > number of clients: 50 > number of threads: 15 > maximum number of tries: 1 > number of transactions per client: 100000 > number of transactions actually processed: 5000000/5000000 > number of failed transactions: 0 (0.000%) > latency average =3D 0.943 ms > initial connection time =3D 48.043 ms > tps =3D 53041.006947 (without initial connection time) > ``` > > ### Read-Only Workload > > ``` > postgres@ubuntu-16gb-fsn1-1:/root$ pgbench -c 50 -j 15 -t 100000 -S > example > pgbench (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1)) > starting vacuum...end. > transaction type: > scaling factor: 50 > query mode: simple > number of clients: 50 > number of threads: 15 > maximum number of tries: 1 > number of transactions per client: 100000 > number of transactions actually processed: 5000000/5000000 > number of failed transactions: 0 (0.000%) > latency average =3D 0.121 ms > initial connection time =3D 53.358 ms > tps =3D 413436.248089 (without initial connection time) > ``` > > These numbers are with 50 concurrent clients and the actual data stored i= n > S3. Hot data is served from ZFS L2ARC and ZeroFS's memory caches, while > cold data comes from S3. > > ## How It Works > > 1. ZeroFS exposes NBD devices (e.g., /dev/nbd0) that PostgreSQL/ZFS can > use like any other block device > 2. Multiple cache layers hide S3 latency: > a. ZFS ARC/L2ARC for frequently accessed blocks > b. ZeroFS memory cache for metadata and hot dataZeroFS exposes NBD > devices (e.g., /dev/nbd0) that PostgreSQL/ZFS can use like any other bloc= k > device > c. Optional local disk cache > 3. All data is encrypted (ChaCha20-Poly1305) before hitting S3 > 4. Files are split into 128KB chunks for insertion into ZeroFS' LSM-tree > > ## Geo-Distributed PostgreSQL > > Since each region can run its own ZeroFS instance, you can create > geographically distributed PostgreSQL setups. > > Example architectures: > > Architecture 1 > > > PostgreSQL Client > | > | SQL queries > | > +--------------+ > | PG Proxy | > | (HAProxy/ | > | PgBouncer) | > +--------------+ > / \ > / \ > Synchronous Synchronous > Replication Replication > / \ > / \ > +---------------+ +---------------+ > | PostgreSQL 1 | | PostgreSQL 2 | > | (Primary) |=E2=97=84------=E2=96=BA| (Standby) | > +---------------+ +---------------+ > | | > | POSIX filesystem ops | > | | > +---------------+ +---------------+ > | ZFS Pool 1 | | ZFS Pool 2 | > | (3-way mirror)| | (3-way mirror)| > +---------------+ +---------------+ > / | \ / | \ > / | \ / | \ > NBD:10809 NBD:10810 NBD:10811 NBD:10812 NBD:10813 NBD:10814 > | | | | | | > +--------++--------++--------++--------++--------++--------+ > |ZeroFS 1||ZeroFS 2||ZeroFS 3||ZeroFS 4||ZeroFS 5||ZeroFS 6| > +--------++--------++--------++--------++--------++--------+ > | | | | | | > | | | | | | > S3-Region1 S3-Region2 S3-Region3 S3-Region4 S3-Region5 S3-Region6 > (us-east) (eu-west) (ap-south) (us-west) (eu-north) (ap-east) > > Architecture 2: > > PostgreSQL Primary (Region 1) =E2=86=90=E2=86=92 PostgreSQL Standby (Regi= on 2) > \ / > \ / > Same ZFS Pool (NBD) > | > 6 Global ZeroFS > | > S3 Regions > > > The main advantages I see are: > 1. Dramatic cost reduction for large datasets > 2. Simplified geo-distribution > 3. Infinite storage capacity > 4. Built-in encryption and compression > > Looking forward to your feedback and questions! > > Best, > Pierre > > P.S. The full project includes a custom NFS filesystem too. > > > --00000000000047c2ae063a33a653 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks, I learned something else: I didn't know Hetzne= r offered S3 compatible storage.=C2=A0

The interesting t= hing is, a few searches about the performance return mostly negative impres= sions about their object storage in comparison to the original S3.=C2=A0

Finding out what kind of performance your benchmarks= would yield on a pure AWS setting would be interesting. I am not asking yo= u to do that, but you may get even better performance in that case :)=C2=A0=

Cheers,
Seref


On Fri, Jul 18, 2025 at 11:58=E2=80=AFAM Pierre Barre <= pierre@barre.sh> wrote:
=
Hi Seref= ,

For the benchmarks, I used Hetzner's cloud s= ervice with the following setup:

- A Hetzner s3 bu= cket in the FSN1 region
- A virtual machine of type=C2=A0ccx63 48= vCPU 192 GB memory
- 3 ZeroFS nbd devices (same s3 bucket)
=
- A ZFS stripped pool with the 3 devices
- 200GB zfs L2ARC
- Postgres configured accordingly memory-wise as well as with sync= hronous_commit =3D off, wal_init_zero =3D off and=C2=A0wal_recycle =3D off.=

Best,
Pierre

On Fri, Jul 18, 2025, at 12:42, Seref = Arikan wrote:
Sorry, this was meant to go to the whole group:

Very interesting!. Great work. Can you clarify how exa= ctly you're running postgres in your tests? A specific=C2=A0AWS service= ? What's the test infrastructure that sits above the file system?
=

On Thu, Jul 17, 2025 at 11:59=E2= =80=AFPM Pierre Barre <pierre@barre.sh> wrote:
Hi= everyone,

I wanted to share a project I've = been working on that enables PostgreSQL to run on S3 storage while maintain= ing performance comparable to local NVMe. The approach uses block-level acc= ess rather than trying to map filesystem operations to S3 objects.

=

# The Architecture

ZeroFS = provides NBD (Network Block Device) servers that expose S3 storage as raw b= lock devices. PostgreSQL runs unmodified on ZFS pools built on these block = devices:

PostgreSQL -> ZFS -> NBD -> Ze= roFS -> S3

By providing block-level access an= d leveraging ZFS's caching capabilities (L2ARC), we can achieve microse= cond latencies despite the underlying storage being in S3.

<= /div>
## Performance Results

Here are pgben= ch results from PostgreSQL running on this setup:

### Read/Write Workload

```
postgre= s@ubuntu-16gb-fsn1-1:/root$ pgbench -c 50 -j 15 -t 100000 example
pgbench (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1))
starting vacuum.= ..end.
transaction type: <builtin: TPC-B (sort of)>
<= div> scaling factor: 50
query mode: simple
number of = clients: 50
number of threads: 15
maximum number of t= ries: 1
number of transactions per client: 100000
num= ber of transactions actually processed: 5000000/5000000
number o= f failed transactions: 0 (0.000%)
latency average =3D 0.943 ms
initial connection time =3D 48.043 ms
tps =3D 53041.00= 6947 (without initial connection time)
```

=
### Read-Only Workload

```
post= gres@ubuntu-16gb-fsn1-1:/root$ pgbench -c 50 -j 15 -t 100000 -S example
pgbench (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1))
starting v= acuum...end.
transaction type: <builtin: select only>
scaling factor: 50
query mode: simple
number o= f clients: 50
number of threads: 15
maximum number of= tries: 1
number of transactions per client: 100000
n= umber of transactions actually processed: 5000000/5000000
number= of failed transactions: 0 (0.000%)
latency average =3D 0.121 ms=
initial connection time =3D 53.358 ms
tps =3D 413436= .248089 (without initial connection time)
```

These numbers are with 50 concurrent clients and the actual data s= tored in S3. Hot data is served from ZFS L2ARC and ZeroFS's memory cach= es, while cold data comes from S3.

## How It Wor= ks

1. ZeroFS exposes NBD devices (e.g., /dev/nbd= 0) that PostgreSQL/ZFS can use like any other block device
2. Mu= ltiple cache layers hide S3 latency:
=C2=A0 =C2=A0a. ZFS ARC/L2A= RC for frequently accessed blocks
=C2=A0 =C2=A0b. ZeroFS memory = cache for metadata and hot dataZeroFS exposes NBD devices (e.g., /dev/nbd0)= that PostgreSQL/ZFS can use like any other block device
=C2=A0 = =C2=A0c. Optional local disk cache
3. All data is encrypted (Cha= Cha20-Poly1305) before hitting S3
4. Files are split into 128KB = chunks for insertion into ZeroFS' LSM-tree

#= # Geo-Distributed PostgreSQL

Since each region c= an run its own ZeroFS instance, you can create geographically distributed P= ostgreSQL setups.

Example architectures:

Architecture 1


=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0PostgreSQL Client
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0| SQL queries
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 +--------------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 PG Proxy=C2=A0 =C2=A0 |
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 | (HAProxy/=C2=A0 =C2=A0 |
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 |=C2=A0 PgBouncer)=C2=A0 |
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 +--------------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/=C2= =A0 =C2=A0 =C2=A0 =C2=A0 \
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 \
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Synchronous=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 Synchronous
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Replication=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Replication
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 \
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0/=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 \
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 +---------------+=C2=A0 = =C2=A0 =C2=A0 =C2=A0 +---------------+
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 | PostgreSQL 1=C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 = | PostgreSQL 2=C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 | (Primary)=C2=A0 =C2=A0 =C2=A0|=E2=97=84------=E2=96=BA| (Standby)= =C2=A0 =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 +---------------+=C2=A0 =C2=A0 =C2=A0 =C2=A0 +---------------+
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 POSIX filesystem ops=C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 +---------------+=C2=A0 =C2=A0 =C2=A0 =C2=A0 +---------------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0ZFS P= ool 1=C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0ZFS Pool 2=C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | (3-way mirror)= |=C2=A0 =C2=A0 =C2=A0 =C2=A0 | (3-way mirror)|
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 +---------------+=C2=A0 =C2=A0 =C2=A0 =C2= =A0 +---------------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0/=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 \=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 /=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 \
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /=C2=A0 =C2=A0 =C2=A0 =C2= =A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0\=C2=A0 =C2=A0 =C2=A0 =C2=A0 /=C2=A0 =C2=A0 = =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0\
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 NBD:10809 NBD:10810 NBD:10811=C2=A0 NBD:10812 NBD:10813 NBD:10814
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2= =A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 +--------++--------++--------++--------++---= -----++--------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 |ZeroFS 1||ZeroFS 2= ||ZeroFS 3||ZeroFS 4||ZeroFS 5||ZeroFS 6|
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 +--------++--------++--------++--------++--------++--------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 S3-Region1 S3-= Region2 S3-Region3 S3-Region4 S3-Region5 S3-Region6
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 (us-east) (eu-west) (ap-south) (us-west) (eu-north) (ap-e= ast)

Architecture 2:

= PostgreSQL Primary (Region 1) =E2=86=90=E2=86=92 PostgreSQL Standby (Region= 2)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 \=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /
<= div> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0\=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /
=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Same ZFS Pool (NBD= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 6 Global ZeroFS
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|<= /div>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 S3 Regions


The mai= n advantages I see are:
1. Dramatic cost reduction for large dat= asets
2. Simplified geo-distribution
3. Infinite sto= rage capacity
4. Built-in encryption and compression
=
Looking forward to your feedback and questions!
=
Best,
Pierre

P.S. The= full project includes a custom NFS filesystem too.


--00000000000047c2ae063a33a653--