Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1XPw2T-0000RR-DN for pgsql-pkg-docker@arkaria.postgresql.org; Fri, 05 Sep 2014 16:07:05 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1XPw2S-0004ex-9Q for pgsql-pkg-docker@arkaria.postgresql.org; Fri, 05 Sep 2014 16:07:04 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XPw2R-0004eq-5h for pgsql-pkg-docker@postgresql.org; Fri, 05 Sep 2014 16:07:03 +0000 Received: from out1-smtp.messagingengine.com ([66.111.4.25]) by magus.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XPw2L-0001Dg-2b for pgsql-pkg-docker@postgresql.org; Fri, 05 Sep 2014 16:07:01 +0000 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by gateway2.nyi.internal (Postfix) with ESMTP id 0CA9C20C85 for ; Fri, 5 Sep 2014 12:06:54 -0400 (EDT) Received: from web6 ([10.202.2.216]) by compute5.internal (MEProxy); Fri, 05 Sep 2014 12:06:54 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=fastmail.net; h= message-id:from:to:mime-version:content-transfer-encoding :content-type:in-reply-to:references:subject:date; s=mesmtp; bh= p4wpYfcurVcn/wctDbAe83q3ErY=; b=eIgxZ0D4XtyYyZ5MXXqVRMCLU48bREYs EHYI+hpnN88ucA2Pzj5sN5hpjv3gvzcY5kkQVRLcXKhtVL+3LvbQBSBiqrKLn2hz gUPX8y7+4zK/YkjErlu63yLzjUZD2w1NcC92NKeKrMlBJIz3yxgjEl1LrWoovTCN +UiOp3fg7rM= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:from:to:mime-version :content-transfer-encoding:content-type:in-reply-to:references :subject:date; s=smtpout; bh=p4wpYfcurVcn/wctDbAe83q3ErY=; b=VgV bIEH4aTgqHv5Oc735miTTx1UdDgJcVfMGpLZlz8CoP+mYKPAKqtoNIkvGooNEG4g zHvBX4VRxXYyY0TnT5RnVKq5MWjwB2oxqtHtJQwvzuVU89JSROYItAKisMMDwb3Q 5YgOeZYJ4MAt4Db2YY48hw+D3nsERdOCtmjN4u/c= Received: by web6.nyi.internal (Postfix, from userid 99) id E11469C20B2; Fri, 5 Sep 2014 12:06:53 -0400 (EDT) Message-Id: <1409933213.2065898.164101681.4F404E3B@webmail.messagingengine.com> X-Sasl-Enc: J3PF5kvgNPRAZ4VL+eTT0HkZfF1Jq5SWM10ONJb8Y+Qj 1409933213 From: sprin@fastmail.net To: pgsql-pkg-docker@postgresql.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: multipart/alternative; boundary="_----------=_140993321320658980"; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-f82de2e6 In-Reply-To: References: <1409718589.1136366.162988969.1D08CE01@webmail.messagingengine.com> Subject: Re: Introduction Date: Fri, 05 Sep 2014 09:06:53 -0700 X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-pkg-docker Precedence: bulk Sender: pgsql-pkg-docker-owner@postgresql.org This is a multi-part message in MIME format. --_----------=_140993321320658980 Content-Transfer-Encoding: 7bit Content-Type: text/plain Hi Jeff, I wasn't sure when the reverse DNS lookup was happening, since I didn't run in to that problem myself, but it's documented here: [1]http://www.postgresql.org/docs/9.1/static/auth-pg-hba-conf.h tml > If a host name is specified (anything that is not an IP address or a > special key word is processed as a potential host name), that name is > compared with the result of a reverse name resolution of the client's > IP address (e.g., reverse DNS lookup, if DNS is used). In my testing so far, I have been simply allowing all connections for replication. For my case, this may also be acceptable for production since I can trust all the hosts on the private subnet where the Postgres clusters live. But I agree that for the general case, a solution is needed. I will +1 the PTR record request for Consul. It would be a good PR for someone who wants to learn a little golang ;) It looks pretty hackable: [2]https://github.com/hashicorp/consul/blob/HEAD/command/agent/ dns.go Congrats on shipping your product soon! -- Steffen Prince On Thu, Sep 4, 2014, at 06:33 AM, McCormick, Jeff wrote: Hi Steffen, since I'm working on something very related to your post, thought I would chime in with my experiences in building a soon to be announced postgres-docker related product (next week the product is announced). When I went to build a DNS solution for the postgres docker stuff I'm doing, I found that [3]consul.io doesn't support PTR records in it's current release, this is used for doing reverse DNS lookups. Postgres utilizes reverse DNS lookups when the master postgres receives a client request from a standby postgres. I have entered a feature request with the consul team regarding PTR support. So I had to move off of [4]consul.io (and registrator) and roll my own DNS-to-Docker solution. This amounted to utilizing named/bind for the DNS server and writing my own DNS bridge loosely based on registrator, in particular the docker client API library that registrator used. With a small amount of code, I now can register the Docker container IP addresses along with PTR records when a postgres container is instantiated. regards Jeff McCormick [5]jeff.mccormick@crunchydatasolutions.com On Wed, Sep 3, 2014 at 12:29 AM, <[6]sprin@fastmail.net> wrote: Hi! I am really excited to see some good discussion and involvement from the Postgres community on the Postgres Docker image. My team is in the process of "Dockerizing all the things" at $WORK, and most of "the things" have Postgres at their core. Some of our backends are little more than some Python HTTP/auth glue on top of Postgres. We make heavy use of views, a little PLPGSQL here and there, and other Postgres niceties such as JSON types. I have been developing with Postgres for a few years now, but after a minor upheaval, I am leading the overhaul of our platform and taking on more of an "ops" role. This brings with it DBA responsibilities, so I am quickly learning about replication, backups, and high availability mechanisms. Coincident with our containerization effort, we are moving towards a more "microservices" platform to alleviate the pain of long-lived, monolithic applications, lower ramp-up time for devs, and generally make the development process simpler. The downside is increased operational complexity of tying together all these services across disparate hosts and making them highly available. Docker is what we use to isolate services from each other on hosts and is a pretty complete tool for this task. To construct our "routing mesh", we found some great complements to Docker: HAProxy[1] and Consul[2]. With some glue provided by Registrator[3] and Consul-HAProxy[4], we were able to build a general purpose, zero-conf TCP routing mesh for Docker services with no new code. I see this routing layer as essential to Postgres replication and failover when dealing with arbitrary numbers of containerized clusters. So I am highly motivated to see Postgres images that are satisfactory for production use in a microservices environment *and* are very approachable to the casual Postgres user. I began some work on Postgres images for our use at $WORK, but Josh pointed me to this list when I asked for feedback. I've just finished reading through the the mailing list archives here and reviewing the commit history/issues at docker-library/postgres, and I think this list is a great place to drive further development of images and guide the use of Dockerized Postgres. I want to ask some pretty high-level questions on the overall approach since I think that will guide the direction of the images we build - tomorrow. -- Steffen Prince [1] [7]http://www.haproxy.org/ [2] [8]http://www.consul.io/ [3] [9]https://github.com/progrium/registrator [4] [10]https://github.com/hashicorp/consul-haproxy -- Sent via pgsql-pkg-docker mailing list ([11]pgsql-pkg-docker@postgresql.org) To make changes to your subscription: [12]http://www.postgresql.org/mailpref/pgsql-pkg-docker -- Jeff McCormick (843) 327-5754 Crunchy Data Solutions [13]jeff.mccormick@crunchydatasolutions.com References 1. http://www.postgresql.org/docs/9.1/static/auth-pg-hba-conf.html 2. https://github.com/hashicorp/consul/blob/HEAD/command/agent/dns.go 3. http://consul.io/ 4. http://consul.io/ 5. mailto:jeff.mccormick@crunchydatasolutions.com 6. mailto:sprin@fastmail.net 7. http://www.haproxy.org/ 8. http://www.consul.io/ 9. https://github.com/progrium/registrator 10. https://github.com/hashicorp/consul-haproxy 11. mailto:pgsql-pkg-docker@postgresql.org 12. http://www.postgresql.org/mailpref/pgsql-pkg-docker 13. mailto:jeff.mccormick@crunchydatasolutions.com --_----------=_140993321320658980 Content-Transfer-Encoding: 7bit Content-Type: text/html
Hi Jeff,
 
I wasn't sure when the reverse DNS lookup was happening, since I didn't
run in to that problem myself, but it's documented here:
 
> If a host name is specified (anything that is not an IP address or a
> special key word is processed as a potential host name), that name is
> compared with the result of a reverse name resolution of the client's
> IP address (e.g., reverse DNS lookup, if DNS is used).
 
In my testing so far, I have been simply allowing all connections for
replication. For my case, this may also be acceptable for production
since I can trust all the hosts on the private subnet where the
Postgres clusters live.
 
But I agree that for the general case, a solution is needed. I will
+1 the PTR record request for Consul. It would be a good PR for someone
who wants to learn a little golang ;)
 
It looks pretty hackable:
 
Congrats on shipping your product soon!
 
--
Steffen Prince
 
On Thu, Sep 4, 2014, at 06:33 AM, McCormick, Jeff wrote:
Hi Steffen,
 
since I'm working on something very related to your post, thought I would chime in with my experiences in building a soon to be announced postgres-docker related product (next week the product is announced).
 
When I went to build a DNS solution for the postgres docker stuff I'm doing, I found that consul.io doesn't support PTR records in it's current release, this is used for doing reverse DNS lookups.  Postgres utilizes reverse DNS lookups when the master postgres receives a client request from a standby postgres.  I have entered a feature request with the consul team regarding PTR support.
 
So I had to move off of consul.io (and registrator) and roll my own DNS-to-Docker solution.  This amounted to utilizing named/bind for the DNS server and writing my own DNS bridge loosely based on registrator, in particular the docker client API library that registrator used.  With a small amount of code, I now can register the Docker container IP addresses along with PTR records when a postgres container is instantiated.
 
regards
 
Jeff McCormick
 
 
On Wed, Sep 3, 2014 at 12:29 AM, <sprin@fastmail.net> wrote:
Hi!

I am really excited to see some good discussion and involvement from the
Postgres community on the Postgres Docker image. My team is in the
process of
"Dockerizing all the things" at $WORK, and most of "the things" have
Postgres
at their core. Some of our backends are little more than some Python
HTTP/auth glue on top of Postgres. We make heavy use of views, a little
PLPGSQL here and there, and other Postgres niceties such as JSON types.

I have been developing with Postgres for a few years now, but after a
minor
upheaval, I am leading the overhaul of our platform and taking on more
of
an "ops" role. This brings with it DBA responsibilities, so I am quickly
learning about replication, backups, and high availability mechanisms.
Coincident with our containerization effort, we are moving towards a
more
"microservices" platform to alleviate the pain of long-lived, monolithic
applications, lower ramp-up time for devs, and generally make the
development
process simpler. The downside is increased operational complexity of
tying
together all these services across disparate hosts and making them
highly
available. Docker is what we use to isolate services from each other on
hosts
and is a pretty complete tool for this task. To construct our "routing
mesh",
we found some great complements to Docker: HAProxy[1] and Consul[2].
With some
glue provided by Registrator[3] and Consul-HAProxy[4], we were able to
build a
general purpose, zero-conf TCP routing mesh for Docker services with no
new
code. I see this routing layer as essential to Postgres replication and
failover when dealing with arbitrary numbers of containerized clusters.

So I am highly motivated to see Postgres images that are satisfactory
for
production use in a microservices environment *and* are very
approachable to
the casual Postgres user. I began some work on Postgres images for our
use
at $WORK, but Josh pointed me to this list when I asked for feedback.
I've
just finished reading through the the mailing list archives here and
reviewing
the commit history/issues at docker-library/postgres, and I think this
list
is a great place to drive further development of images and guide the
use
of Dockerized Postgres. I want to ask some pretty high-level questions
on
the overall approach since I think that will guide the direction of the
images
we build - tomorrow.

--
Steffen Prince

[1] http://www.haproxy.org/
[2] http://www.consul.io/
[3] https://github.com/progrium/registrator
[4] https://github.com/hashicorp/consul-haproxy


--
Sent via pgsql-pkg-docker mailing list (pgsql-pkg-docker@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-pkg-docker
 
 
 
--
 
Jeff McCormick
 
(843) 327-5754
Crunchy Data Solutions
 
--_----------=_140993321320658980--