public inbox for [email protected]
help / color / mirror / Atom feedPostgreSQL - HA Cluster / Etcd Issue
7+ messages / 4 participants
[nested] [flat]
* PostgreSQL - HA Cluster / Etcd Issue
@ 2025-08-20 07:40 SOzcn <[email protected]>
2025-08-20 12:46 ` Re: PostgreSQL - HA Cluster / Etcd Issue Scott Ribe <[email protected]>
2025-08-20 16:51 ` Re: PostgreSQL - HA Cluster / Etcd Issue Ozgur Kulu <[email protected]>
0 siblings, 2 replies; 7+ messages in thread
From: SOzcn @ 2025-08-20 07:40 UTC (permalink / raw)
To: Pgsql-admin <[email protected]>
Hello,
In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
Although many things in this architecture work as expected, Etcd encounters
the error "memberID:5960773920904691185 alarm:NOSPACE" once every 2 or 3
months. The root cause is that the database size reaches the specified
limit and Etcd does not clean it up through rotation. I am aware of this,
but despite reviewing the maintenance documentation, I could not find the
most appropriate parameter values for this rotation. I am sharing an
example of my own Etcd configuration below. I kindly ask you to share your
experience on this matter. I am kinda stuck on this issue.
In particular, is it possible to schedule the parameter that performs this
cleanup with a time setting? For example, can I tell it to run the compact
operation at around 01:00 AM?
Best regards.
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
[Service]
User=etcd
Type=notify
ExecStart=/usr/local/sbin/etcd \
--name etcd1 \
--data-dir=/etc/etcd \
--initial-advertise-peer-urls=http://10.115.208.168:2380 \
--listen-peer-urls=http://192.168.1.1:2380 \
--listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
--advertise-client-urls=192.168.1.1:2379 \
--initial-cluster-token="etcd-cluster" \
--initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=
http://192.168.1.2:2380,etcd3=http://3:2380"; \
--initial-cluster-state="new" \
--log-level="error"
[Install]
WantedBy=multi-user.target
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL - HA Cluster / Etcd Issue
2025-08-20 07:40 PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
@ 2025-08-20 12:46 ` Scott Ribe <[email protected]>
2025-08-20 13:52 ` Re: PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
1 sibling, 1 reply; 7+ messages in thread
From: Scott Ribe @ 2025-08-20 12:46 UTC (permalink / raw)
To: SOzcn <[email protected]>; +Cc: Pgsql-admin <[email protected]>
This is a PostgreSQL mailing list, and your question has nothing to do with PostgreSQL.
--
Scott Ribe
[email protected]
https://www.linkedin.com/in/scottribe/
> On Aug 20, 2025, at 1:40 AM, SOzcn <[email protected]> wrote:
>
> Hello,
>
> In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
>
> Although many things in this architecture work as expected, Etcd encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once every 2 or 3 months. The root cause is that the database size reaches the specified limit and Etcd does not clean it up through rotation. I am aware of this, but despite reviewing the maintenance documentation, I could not find the most appropriate parameter values for this rotation. I am sharing an example of my own Etcd configuration below. I kindly ask you to share your experience on this matter. I am kinda stuck on this issue.
>
> In particular, is it possible to schedule the parameter that performs this cleanup with a time setting? For example, can I tell it to run the compact operation at around 01:00 AM?
>
> Best regards.
>
>
> [Unit]
> Description=etcd service
> Documentation=https://github.com/etcd-io/etcd
> After=network.target
>
> [Service]
> User=etcd
> Type=notify
> ExecStart=/usr/local/sbin/etcd \
> --name etcd1 \
> --data-dir=/etc/etcd \
> --initial-advertise-peer-urls=http://10.115.208.168:2380 \
> --listen-peer-urls=http://192.168.1.1:2380 \
> --listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
> --advertise-client-urls=192.168.1.1:2379 \
> --initial-cluster-token="etcd-cluster" \
> --initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=http://192.168.1.2:2380,etcd3=http://3:2380"; \
> --initial-cluster-state="new" \
> --log-level="error"
>
> [Install]
> WantedBy=multi-user.target
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL - HA Cluster / Etcd Issue
2025-08-20 07:40 PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
2025-08-20 12:46 ` Re: PostgreSQL - HA Cluster / Etcd Issue Scott Ribe <[email protected]>
@ 2025-08-20 13:52 ` SOzcn <[email protected]>
2025-08-20 14:03 ` Re: PostgreSQL - HA Cluster / Etcd Issue Mike Artz <[email protected]>
2025-08-20 16:45 ` Re: PostgreSQL - HA Cluster / Etcd Issue Scott Ribe <[email protected]>
0 siblings, 2 replies; 7+ messages in thread
From: SOzcn @ 2025-08-20 13:52 UTC (permalink / raw)
To: Scott Ribe <[email protected]>; +Cc: Pgsql-admin <[email protected]>
This is high availability environment and etcd is one management tools of
them . Many PostgreSQL managed by this architec. I advice to you research.
Etcs - Patrono - Postgresql- Haproxy- Keepalived. Anyway if you doesn't
have experience of it, that's ok.
20 Ağu 2025 Çar 15:46 tarihinde Scott Ribe <[email protected]>
şunu yazdı:
> This is a PostgreSQL mailing list, and your question has nothing to do
> with PostgreSQL.
>
> --
> Scott Ribe
> [email protected]
> https://www.linkedin.com/in/scottribe/
>
>
>
> > On Aug 20, 2025, at 1:40 AM, SOzcn <[email protected]> wrote:
> >
> > Hello,
> >
> > In my database architecture, I use Etcd, Patroni, HAProxy, and
> Keepalived.
> >
> > Although many things in this architecture work as expected, Etcd
> encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once
> every 2 or 3 months. The root cause is that the database size reaches the
> specified limit and Etcd does not clean it up through rotation. I am aware
> of this, but despite reviewing the maintenance documentation, I could not
> find the most appropriate parameter values for this rotation. I am sharing
> an example of my own Etcd configuration below. I kindly ask you to share
> your experience on this matter. I am kinda stuck on this issue.
> >
> > In particular, is it possible to schedule the parameter that performs
> this cleanup with a time setting? For example, can I tell it to run the
> compact operation at around 01:00 AM?
> >
> > Best regards.
> >
> >
> > [Unit]
> > Description=etcd service
> > Documentation=https://github.com/etcd-io/etcd
> > After=network.target
> >
> > [Service]
> > User=etcd
> > Type=notify
> > ExecStart=/usr/local/sbin/etcd \
> > --name etcd1 \
> > --data-dir=/etc/etcd \
> > --initial-advertise-peer-urls=http://10.115.208.168:2380 \
> > --listen-peer-urls=http://192.168.1.1:2380 \
> > --listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
> > --advertise-client-urls=192.168.1.1:2379 \
> > --initial-cluster-token="etcd-cluster" \
> > --initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=
> http://192.168.1.2:2380,etcd3=http://3:2380"; \
> > --initial-cluster-state="new" \
> > --log-level="error"
> >
> > [Install]
> > WantedBy=multi-user.target
>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL - HA Cluster / Etcd Issue
2025-08-20 07:40 PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
2025-08-20 12:46 ` Re: PostgreSQL - HA Cluster / Etcd Issue Scott Ribe <[email protected]>
2025-08-20 13:52 ` Re: PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
@ 2025-08-20 14:03 ` Mike Artz <[email protected]>
1 sibling, 0 replies; 7+ messages in thread
From: Mike Artz @ 2025-08-20 14:03 UTC (permalink / raw)
To: SOzcn <[email protected]>; +Cc: Scott Ribe <[email protected]>; Pgsql-admin <[email protected]>
Are you saying that you already have a compaction job running (e.g.
etcdctl compact $(etcdctl endpoint status --write-out=json | jq -r
'.[0].Status.header.revision - 1000')
or similar), but you’re hoping there’s a parameter in etcd itself to handle
that scheduling?
On Wed, Aug 20, 2025 at 8:53 AM SOzcn <[email protected]> wrote:
> This is high availability environment and etcd is one management tools of
> them . Many PostgreSQL managed by this architec. I advice to you research.
> Etcs - Patrono - Postgresql- Haproxy- Keepalived. Anyway if you doesn't
> have experience of it, that's ok.
>
> 20 Ağu 2025 Çar 15:46 tarihinde Scott Ribe <[email protected]>
> şunu yazdı:
>
>> This is a PostgreSQL mailing list, and your question has nothing to do
>> with PostgreSQL.
>>
>> --
>> Scott Ribe
>> [email protected]
>> https://www.linkedin.com/in/scottribe/
>>
>>
>>
>> > On Aug 20, 2025, at 1:40 AM, SOzcn <[email protected]> wrote:
>> >
>> > Hello,
>> >
>> > In my database architecture, I use Etcd, Patroni, HAProxy, and
>> Keepalived.
>> >
>> > Although many things in this architecture work as expected, Etcd
>> encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once
>> every 2 or 3 months. The root cause is that the database size reaches the
>> specified limit and Etcd does not clean it up through rotation. I am aware
>> of this, but despite reviewing the maintenance documentation, I could not
>> find the most appropriate parameter values for this rotation. I am sharing
>> an example of my own Etcd configuration below. I kindly ask you to share
>> your experience on this matter. I am kinda stuck on this issue.
>> >
>> > In particular, is it possible to schedule the parameter that performs
>> this cleanup with a time setting? For example, can I tell it to run the
>> compact operation at around 01:00 AM?
>> >
>> > Best regards.
>> >
>> >
>> > [Unit]
>> > Description=etcd service
>> > Documentation=https://github.com/etcd-io/etcd
>> > After=network.target
>> >
>> > [Service]
>> > User=etcd
>> > Type=notify
>> > ExecStart=/usr/local/sbin/etcd \
>> > --name etcd1 \
>> > --data-dir=/etc/etcd \
>> > --initial-advertise-peer-urls=http://10.115.208.168:2380 \
>> > --listen-peer-urls=http://192.168.1.1:2380 \
>> > --listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
>> > --advertise-client-urls=192.168.1.1:2379 \
>> > --initial-cluster-token="etcd-cluster" \
>> > --initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=
>> http://192.168.1.2:2380,etcd3=http://3:2380"; \
>> > --initial-cluster-state="new" \
>> > --log-level="error"
>> >
>> > [Install]
>> > WantedBy=multi-user.target
>>
>>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL - HA Cluster / Etcd Issue
2025-08-20 07:40 PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
2025-08-20 12:46 ` Re: PostgreSQL - HA Cluster / Etcd Issue Scott Ribe <[email protected]>
2025-08-20 13:52 ` Re: PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
@ 2025-08-20 16:45 ` Scott Ribe <[email protected]>
1 sibling, 0 replies; 7+ messages in thread
From: Scott Ribe @ 2025-08-20 16:45 UTC (permalink / raw)
To: SOzcn <[email protected]>; +Cc: Pgsql-admin <[email protected]>
> On Aug 20, 2025, at 7:52 AM, SOzcn <[email protected]> wrote:
>
> This is high availability environment and etcd is one management tools of them . Many PostgreSQL managed by this architec. I advice to you research. Etcs - Patrono - Postgresql- Haproxy- Keepalived. Anyway if you doesn't have experience of it, that's ok.
I am perfectly well aware of all that. However, you have an etcd problem, not a postgres problem. You'd have a much better chance of getting answers to etcd issues on an etcd mailing list.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL - HA Cluster / Etcd Issue
2025-08-20 07:40 PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
@ 2025-08-20 16:51 ` Ozgur Kulu <[email protected]>
2025-08-20 22:17 ` Re: PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
1 sibling, 1 reply; 7+ messages in thread
From: Ozgur Kulu @ 2025-08-20 16:51 UTC (permalink / raw)
To: SOzcn <[email protected]>; +Cc: Pgsql-admin <[email protected]>
Hi there,
Etcd database should be extended from 2 gb to 8gb. On the other hand, you
can use defrag or again created the etcd db.
Have a nice day,
20 Ağu 2025 Çar 10:40 tarihinde SOzcn <[email protected]> şunu
yazdı:
> Hello,
>
> In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
>
> Although many things in this architecture work as expected, Etcd
> encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once
> every 2 or 3 months. The root cause is that the database size reaches the
> specified limit and Etcd does not clean it up through rotation. I am aware
> of this, but despite reviewing the maintenance documentation, I could not
> find the most appropriate parameter values for this rotation. I am sharing
> an example of my own Etcd configuration below. I kindly ask you to share
> your experience on this matter. I am kinda stuck on this issue.
>
> In particular, is it possible to schedule the parameter that performs this
> cleanup with a time setting? For example, can I tell it to run the compact
> operation at around 01:00 AM?
>
> Best regards.
>
>
> [Unit]
> Description=etcd service
> Documentation=https://github.com/etcd-io/etcd
> After=network.target
>
> [Service]
> User=etcd
> Type=notify
> ExecStart=/usr/local/sbin/etcd \
> --name etcd1 \
> --data-dir=/etc/etcd \
> --initial-advertise-peer-urls=http://10.115.208.168:2380 \
> --listen-peer-urls=http://192.168.1.1:2380 \
> --listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
> --advertise-client-urls=192.168.1.1:2379 \
> --initial-cluster-token="etcd-cluster" \
> --initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=
> http://192.168.1.2:2380,etcd3=http://3:2380"; \
> --initial-cluster-state="new" \
> --log-level="error"
>
> [Install]
> WantedBy=multi-user.target
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: PostgreSQL - HA Cluster / Etcd Issue
2025-08-20 07:40 PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
2025-08-20 16:51 ` Re: PostgreSQL - HA Cluster / Etcd Issue Ozgur Kulu <[email protected]>
@ 2025-08-20 22:17 ` SOzcn <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: SOzcn @ 2025-08-20 22:17 UTC (permalink / raw)
To: Ozgur Kulu <[email protected]>; +Cc: Pgsql-admin <[email protected]>
Hello Ozgur,
Thanks for the reply.
If i up to etcd size from 2 to 8 GB it's make much problem but if defrag is
fix the "no space" issue can you share your experience? By example is it
spent too much rss etc. and are you doing that with cronjob or manuel. As i
know if etcd went to error with "no space" defrag methot also doesn't work.
Have a nice day.
Ozgur Kulu <[email protected]>, 20 Ağu 2025 Çar, 19:51 tarihinde şunu
yazdı:
> Hi there,
>
> Etcd database should be extended from 2 gb to 8gb. On the other hand, you
> can use defrag or again created the etcd db.
>
> Have a nice day,
>
> 20 Ağu 2025 Çar 10:40 tarihinde SOzcn <[email protected]> şunu
> yazdı:
>
>> Hello,
>>
>> In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
>>
>> Although many things in this architecture work as expected, Etcd
>> encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once
>> every 2 or 3 months. The root cause is that the database size reaches the
>> specified limit and Etcd does not clean it up through rotation. I am aware
>> of this, but despite reviewing the maintenance documentation, I could not
>> find the most appropriate parameter values for this rotation. I am sharing
>> an example of my own Etcd configuration below. I kindly ask you to share
>> your experience on this matter. I am kinda stuck on this issue.
>>
>> In particular, is it possible to schedule the parameter that performs
>> this cleanup with a time setting? For example, can I tell it to run the
>> compact operation at around 01:00 AM?
>>
>> Best regards.
>>
>>
>> [Unit]
>> Description=etcd service
>> Documentation=https://github.com/etcd-io/etcd
>> After=network.target
>>
>> [Service]
>> User=etcd
>> Type=notify
>> ExecStart=/usr/local/sbin/etcd \
>> --name etcd1 \
>> --data-dir=/etc/etcd \
>> --initial-advertise-peer-urls=http://10.115.208.168:2380 \
>> --listen-peer-urls=http://192.168.1.1:2380 \
>> --listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
>> --advertise-client-urls=192.168.1.1:2379 \
>> --initial-cluster-token="etcd-cluster" \
>> --initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=
>> http://192.168.1.2:2380,etcd3=http://3:2380"; \
>> --initial-cluster-state="new" \
>> --log-level="error"
>>
>> [Install]
>> WantedBy=multi-user.target
>>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2025-08-20 22:17 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-08-20 07:40 PostgreSQL - HA Cluster / Etcd Issue SOzcn <[email protected]>
2025-08-20 12:46 ` Scott Ribe <[email protected]>
2025-08-20 13:52 ` SOzcn <[email protected]>
2025-08-20 14:03 ` Mike Artz <[email protected]>
2025-08-20 16:45 ` Scott Ribe <[email protected]>
2025-08-20 16:51 ` Ozgur Kulu <[email protected]>
2025-08-20 22:17 ` SOzcn <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox