pgpool2.git / summary / log / commit / refs
commit f363b1acbfe1ee4867995ff72cc81bb09cf66767
Author: Tatsuo Ishii <ishii@postgresql.org>
Date: Sun Oct 04 14:22:54 2015 +0000
Add new parameter "serialize_accept".
This parameter defines whether to serialize accept() call for incoming
client connections. Default is off, which means no serializing (same
behavior as pgpool-II 3.4 or before). If this is off, the kernel
wakes up all of pgpool-II child process to execute accept() and one of
them actually accepts the incoming connection. Problem here is,
because so my child process wake up at a same time, heavy context
switching occurred and the performance is affected. This phenomena is
a classic problem called "the thundering herd problem". By enabling
serialize_accept, only one of pgpool-II child process is woken up and
executes accept() and the problem can be avoided.
If child_life_time is enabled, serialize_accept has no effect. If you
are care about pgpool-II process memory leak or whatever potential
issue, you could use child_max_connections instead. This is purely an
implementation limitation and maybe we could eliminate it in the
future.
doc/pgpool-en.html | 40 ++++++++++++++++++++++++++++
doc/pgpool-ja.html | 42 ++++++++++++++++++++++++++++++
src/config/pool_config.c | 15 +++++++++++
src/config/pool_config.l | 15 +++++++++++
src/include/pool.h | 3 ++-
src/include/pool_config.h | 1 +
src/protocol/child.c | 19 ++++++++++++++
src/sample/pgpool.conf.sample | 3 +++
src/sample/pgpool.conf.sample-master-slave | 3 +++
src/sample/pgpool.conf.sample-replication | 3 +++
src/sample/pgpool.conf.sample-stream | 3 +++
src/utils/pool_process_reporting.c | 5 ++++
12 files changed, 151 insertions(+), 1 deletion(-)
diff --git a/doc/pgpool-en.html b/doc/pgpool-en.html
index ab6b3fb67..23dda0a4d 100644
--- a/doc/pgpool-en.html
+++ b/doc/pgpool-en.html
@@ -635,6 +635,43 @@ max_pool*num_init_children*2 <= (max_connections - superuser_reserved_connect
<p>
This parameter can only be set at server start.</p>
</dd>
+
+<dt><a name="SERIALIZE_ACCEPT"></a>serialize_accept <span class="version">V3.5 -</span></dt>
+ <dd>
+ <p>
+ Whether to serialize accept() call for incoming client connections.
+ Default is off, which means no serializing (same behavior as pgpool-II 3.4 or before).
+ If this is off, the kernel wakes up all of pgpool-II child process to execute accept() and one of them actually accepts the incoming connection.
+ Problem here is, because so my child process wake up at a same time, heavy context switching occurred and
+ the performance is affected.
+ This phenomena is a classic problem called "the thundering herd problem".
+ By enabling serialize_accept, only one of pgpool-II child process is woken up and executes accept() and the problem can be avoided.
+ When you should turn on serialize_accept?
+ For large number of num_init_children, it is recommended to turn on serialize_accept.
+ For small number of num_init_children, it may not enhance the performance rather, it may degrade the performance because of serializing overhead.
+ How large is actually large is depending on the environment.
+ It is recommended to do a bench mark test to make the decision.
+ Here is a example pgbench run:
+ </p>
+ <p>
+ <pre>
+ pgbench -n -S -p 9999 -c 32 -C -S -T 300 test
+ </pre>
+ </p>
+ <p>
+ Here, -C tells pgbench to connect to database each time a transaction gets executed.
+ -c 32 specifies the concurrent sessions to pgpool-II. You should change this according to your system's requirement.
+ When pgbench finishes, you want to check the number from "including connections establishing".
+ </p>
+ <p>
+ Please note that if <a href="#CHILD_LIFE_TIME">child_life_time</a> is enabled, serialize_accept has no effect.
+ Make sure that you set child_life_time to 0 if you want to turn on serialize_accept.
+ If you are care about pgpool-II process memory leak or whatever potential issue, you could use <a href="#CHILD_MAX_CONNECTIONS">child_max_connections</a> instead.
+ This is purely an implementation limitation and maybe removed in the future.
+ </p>
+ <p>
+ This parameter can only be set at server start.</p>
+ </dd>
<dt><a name="CHILD_LIFE_TIME"></a>child_life_time</dt>
<dd>
@@ -645,6 +682,9 @@ max_pool*num_init_children*2 <= (max_connections - superuser_reserved_connect
0 disables this feature. Note that this doesn't apply for processes
that have not accepted any connection yet.
</p>
+ <p>
+ Note: if this parameter is non 0, <a href="#SERIALIZE_ACCEPT">serialize_accept</a> has no effect.
+ </p>
<p>
You need to reload pgpool.conf if you change this value.
</p>
diff --git a/doc/pgpool-ja.html b/doc/pgpool-ja.html
index 72147c3dd..b6fbba747 100644
--- a/doc/pgpool-ja.html
+++ b/doc/pgpool-ja.html
@@ -659,6 +659,45 @@ net.core.somaxconn = 256
このパラメータを変更した時には pgpool-II を再起動してください。
</p>
</dd>
+
+<dt><a name="SERIALIZE_ACCEPT"></a>serialize_accept <span class="version">V3.5 -</span></dt>
+ <dd>
+ <p>
+ クライアントからの接続を受け付ける際に accept() の呼び出しをシリアライズするかどうかを指定します。
+ デフォルトはoffです(シリアライズしません)で、これは pgpool-II 3.4 以前と同じ挙動です。
+ このパラメータがoffの場合、カーネルはすべてのpgpool-II子プロセスを起こして accept() を実行させます。
+ そして子プロセスのうちひとつだけが実際に接続を受け付けます。
+ 問題は、ここで多くの子プロセスが一度に起こされるため、重いコンテキストスイッチングが起こり、性能に影響がでることです。
+ この現象は「thundering herd problem」と呼ばれる古典的な問題です。
+ serialize_accept を有効にすることにより、pgpool-II子プロセスのうちひとつだけが起こされて accept() を実行するようになり、この問題は回避されます。
+ ではどんなときに serialize_accept を有効にすべきでしょう？
+ num_init_children が大きい時はserialize_accept 有効にすることをおすすめします。
+ num_init_children が小さい時はserialize_accept 有効にしても効果がないかもしれません。
+ むしろシリアライズのオーバヘッドのために性能が低下するかもしれません。
+ どの位の数が「大きい」と言えるかは環境によります。
+ どうするか決める前に、ベンチマークテストを行ってみることをおすすめします。
+ 例として以下のような方法でpgbenchを実行します。
+ </p>
+ <p>
+ <pre>
+ pgbench -n -S -p 9999 -c 32 -C -S -T 300 test
+ </pre>
+ </p>
+ <p>
+ ここで、 -C は pgbench にトランザクションの実行の度に毎回データベースに接続することを指示します。
+ -c 32 は、pgpool-II への同時接続数です。これはあなたのシステム環境にあわせて変更しましょう。
+ pgbenchが終了すると、"including connections establishing" のところに数字が出てくるのでそれをチェックします。
+ </p>
+ <p>
+ なお、<a href="#CHILD_LIFE_TIME">child_life_time</a> が有効だと、serialize_accept は効果がありません。
+ serialize_accept を有効にしたい場合は、child_life_timeが 0 であることを確認してください。
+ pgpool-IIプロセスのメモリーリークなどの潜在的な問題を気にする場合は、代わりに<a href="#CHILD_MAX_CONNECTIONS">child_max_connections</a> を使ってください。
+ この制限は純粋に実装上の問題であり、将来はなくなるかもしれません。
+ </p>
+ <p>
+ このパラメータを変更した時には pgpool-II を再起動してください。
+ </p>
+ </dd>
<dt><a name="CHILD_LIFE_TIME"></a>child_life_time</dt>
<dd>
@@ -670,6 +709,9 @@ net.core.somaxconn = 256
0を指定するとこの機能は働きません（すなわち起動しっ放し）。
なお、まだ一度もコネクションを受け付けていないプロセスにはchild_life_timeは適用されません。
</p>
+ <p>
+ 注意: このパラメータが0以外の場合、<a href="#SERIALIZE_ACCEPT">serialize_accept</a> の効果はなくなります。
+ </p>
<p>
このパラメータを変更した時には設定ファイルを再読み込みしてください。
</p>
diff --git a/src/config/pool_config.c b/src/config/pool_config.c
index 8be91870a..68218fe56 100644
--- a/src/config/pool_config.c
+++ b/src/config/pool_config.c
@@ -1873,6 +1873,7 @@ int pool_init_config(void)
pool_config->pcp_socket_dir = DEFAULT_SOCKET_DIR;
pool_config->num_init_children = 32;
pool_config->listen_backlog_multiplier = 2;
+ pool_config->serialize_accept = 0;
pool_config->max_pool = 4;
pool_config->child_life_time = 300;
pool_config->client_idle_limit = 0;
@@ -2387,6 +2388,20 @@ int pool_get_config(char *confpath, POOL_CONFIG_CONTEXT context)
}
pool_config->listen_backlog_multiplier = v;
}
+ else if (!strcmp(key, "serialize_accept") && CHECK_CONTEXT(INIT_CONFIG, context))
+ {
+ int v = eval_logical(yytext);
+
+ if (v < 0)
+ {
+ fclose(fd);
+ ereport(error_level,
+ (errmsg("invalid configuration for key \"%s\"",key),
+ errdetail("invalid value:\"%s\" for key:\"%s\"",yytext,key)));
+ return(-1);
+ }
+ pool_config->serialize_accept = v;
+ }
else if (!strcmp(key, "child_life_time") &&
CHECK_CONTEXT(INIT_CONFIG|RELOAD_CONFIG, context))
{
diff --git a/src/config/pool_config.l b/src/config/pool_config.l
index f9cdd8767..05c47da22 100644
--- a/src/config/pool_config.l
+++ b/src/config/pool_config.l
@@ -134,6 +134,7 @@ int pool_init_config(void)
pool_config->pcp_socket_dir = DEFAULT_SOCKET_DIR;
pool_config->num_init_children = 32;
pool_config->listen_backlog_multiplier = 2;
+ pool_config->serialize_accept = 0;
pool_config->max_pool = 4;
pool_config->child_life_time = 300;
pool_config->client_idle_limit = 0;
@@ -648,6 +649,20 @@ int pool_get_config(char *confpath, POOL_CONFIG_CONTEXT context)
}
pool_config->listen_backlog_multiplier = v;
}
+ else if (!strcmp(key, "serialize_accept") && CHECK_CONTEXT(INIT_CONFIG, context))
+ {
+ int v = eval_logical(yytext);
+
+ if (v < 0)
+ {
+ fclose(fd);
+ ereport(error_level,
+ (errmsg("invalid configuration for key \"%s\"",key),
+ errdetail("invalid value:\"%s\" for key:\"%s\"",yytext,key)));
+ return(-1);
+ }
+ pool_config->serialize_accept = v;
+ }
else if (!strcmp(key, "child_life_time") &&
CHECK_CONTEXT(INIT_CONFIG|RELOAD_CONFIG, context))
{
diff --git a/src/include/pool.h b/src/include/pool.h
index 83c2a143b..9f3c117e5 100644
--- a/src/include/pool.h
+++ b/src/include/pool.h
@@ -357,12 +357,13 @@ extern int my_master_node_id;
#define NO_LOAD_BALANCE "/*NO LOAD BALANCE*/"
#define NO_LOAD_BALANCE_COMMENT_SZ (sizeof(NO_LOAD_BALANCE)-1)
-#define MAX_NUM_SEMAPHORES 5
+#define MAX_NUM_SEMAPHORES 6
#define CONN_COUNTER_SEM 0
#define REQUEST_INFO_SEM 1
#define SHM_CACHE_SEM 2
#define QUERY_CACHE_STATS_SEM 3
#define PCP_REQUEST_SEM 4
+#define ACCEPT_FD_SEM 5
#define MAX_REQUEST_QUEUE_SIZE 10
/*
diff --git a/src/include/pool_config.h b/src/include/pool_config.h
index 6f7540a64..0ccb4d2b1 100644
--- a/src/include/pool_config.h
+++ b/src/include/pool_config.h
@@ -77,6 +77,7 @@ typedef struct {
char *pcp_socket_dir; /* PCP socket directory */
int num_init_children; /* # of children initially pre-forked */
int listen_backlog_multiplier; /* determines the size of the connection queue */
+ int serialize_accept; /* if non 0, serialize call to accept() to avoid thundering herd problem */
int child_life_time; /* if idle for this seconds, child exits */
int connection_life_time; /* if idle for this seconds, connection closes */
int child_max_connections; /* if max_connections received, child exits */
diff --git a/src/protocol/child.c b/src/protocol/child.c
index 00d0ae2ca..8bcc5f829 100644
--- a/src/protocol/child.c
+++ b/src/protocol/child.c
@@ -1832,9 +1832,28 @@ wait_for_new_connections(int *fds, struct timeval *timeout, SockAddr *saddr)
#endif
}
+ /*
+ * If child life time is disabled and serialize_accept is on, we serialize
+ * select() and accept() to avoid the "Thundering herd" problem.
+ */
+ if (pool_config->child_life_time == 0 && pool_config->serialize_accept != 0)
+ {
+ pool_semaphore_lock(ACCEPT_FD_SEM);
+ ereport(DEBUG1,
+ (errmsg("LOCKING select()")));
+ }
+
numfds = select(nsocks, &rmask, NULL, NULL, timeoutval);
save_errno = errno;
+
+ if (pool_config->child_life_time == 0 && pool_config->serialize_accept != 0)
+ {
+ pool_semaphore_unlock(ACCEPT_FD_SEM);
+ ereport(DEBUG1,
+ (errmsg("UNLOCKING select()")));
+ }
+
/* check backend timer is expired */
if (backend_timer_expired)
{
diff --git a/src/sample/pgpool.conf.sample b/src/sample/pgpool.conf.sample
index e89bcf2ca..da95f4710 100644
--- a/src/sample/pgpool.conf.sample
+++ b/src/sample/pgpool.conf.sample
@@ -40,6 +40,9 @@ listen_backlog_multiplier = 2
# Set the backlog parameter of listen(2) to
# num_init_children * listen_backlog_multiplier.
# (change requires restart)
+serialize_accept = off
+ # whether to serialize accept() call to avoid thundering herd problem
+ # (change requires restart)
# - pgpool Communication Manager Connection Settings -
diff --git a/src/sample/pgpool.conf.sample-master-slave b/src/sample/pgpool.conf.sample-master-slave
index 0444f7d08..833a72fc4 100644
--- a/src/sample/pgpool.conf.sample-master-slave
+++ b/src/sample/pgpool.conf.sample-master-slave
@@ -40,6 +40,9 @@ listen_backlog_multiplier = 2
# Set the backlog parameter of listen(2) to
# num_init_children * listen_backlog_multiplier.
# (change requires restart)
+serialize_accept = off
+ # whether to serialize accept() call to avoid thundering herd problem
+ # (change requires restart)
# - pgpool Communication Manager Connection Settings -
diff --git a/src/sample/pgpool.conf.sample-replication b/src/sample/pgpool.conf.sample-replication
index dd582f11d..f26c4736a 100644
--- a/src/sample/pgpool.conf.sample-replication
+++ b/src/sample/pgpool.conf.sample-replication
@@ -40,6 +40,9 @@ listen_backlog_multiplier = 2
# Set the backlog parameter of listen(2) to
# num_init_children * listen_backlog_multiplier.
# (change requires restart)
+serialize_accept = off
+ # whether to serialize accept() call to avoid thundering herd problem
+ # (change requires restart)
# - pgpool Communication Manager Connection Settings -
diff --git a/src/sample/pgpool.conf.sample-stream b/src/sample/pgpool.conf.sample-stream
index 9763c607e..f52bdb9e9 100644
--- a/src/sample/pgpool.conf.sample-stream
+++ b/src/sample/pgpool.conf.sample-stream
@@ -56,6 +56,9 @@ listen_backlog_multiplier = 2
# Set the backlog parameter of listen(2) to
# num_init_children * listen_backlog_multiplier.
# (change requires restart)
+serialize_accept = off
+ # whether to serialize accept() call to avoid thundering herd problem
+ # (change requires restart)
# - Backend Connection Settings -
diff --git a/src/utils/pool_process_reporting.c b/src/utils/pool_process_reporting.c
index 9745ba7ad..cc0f0443a 100644
--- a/src/utils/pool_process_reporting.c
+++ b/src/utils/pool_process_reporting.c
@@ -242,6 +242,11 @@ POOL_REPORT_CONFIG* get_config(int *nrows)
StrNCpy(status[i].desc, "determines the size of the queue for pending connections", POOLCONFIG_MAXDESCLEN);
i++;
+ StrNCpy(status[i].name, "serialize_accept", POOLCONFIG_MAXNAMELEN);
+ snprintf(status[i].value, POOLCONFIG_MAXVALLEN, "%d", pool_config->serialize_accept);
+ StrNCpy(status[i].desc, "whether to serialize accept() call", POOLCONFIG_MAXDESCLEN);
+ i++;
+
StrNCpy(status[i].name, "max_pool", POOLCONFIG_MAXNAMELEN);
snprintf(status[i].value, POOLCONFIG_MAXVALLEN, "%d", pool_config->max_pool);
StrNCpy(status[i].desc, "max # of connection pool per child", POOLCONFIG_MAXDESCLEN);
[parent: 3e930d79401d]