X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id DB8DEDA573 for ; Sat, 22 Oct 2005 18:12:07 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 99600-01 for ; Sat, 22 Oct 2005 21:12:03 +0000 (GMT) Received: from web32902.mail.mud.yahoo.com (web32902.mail.mud.yahoo.com [68.142.206.49]) by svr1.postgresql.org (Postfix) with SMTP id BE3CBD7797 for ; Sat, 22 Oct 2005 18:12:03 -0300 (ADT) Received: (qmail 97793 invoked by uid 60001); 22 Oct 2005 21:12:05 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=Yv48ovXuIgNS99q+cnTC/1vWznDm3SvWMytctP0sgriK9F5woW39MG25gwbtQrAIFj/Ozj5/o1wy+nBzTVl+63oFDbZTcMrgy9c3uAaSP8UPQX2uYC6defxODzFTB3U4eBhr4ygToUk3cgKyvdmbkcdCCODs79iDzhbYqSCnzPM= ; Message-ID: <20051022211205.97791.qmail@web32902.mail.mud.yahoo.com> Received: from [202.65.136.134] by web32902.mail.mud.yahoo.com via HTTP; Sat, 22 Oct 2005 14:12:05 PDT Date: Sat, 22 Oct 2005 14:12:05 -0700 (PDT) From: Kishore B Subject: Need help in setting optimal configuration for a huge database. To: pgsql-performance@postgresql.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="0-267068631-1130015525=:97706" Content-Transfer-Encoding: 8bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.402 required=5 tests=[DNS_FROM_RFC_ABUSE=0.374, HTML_60_70=0.027, HTML_MESSAGE=0.001] X-Spam-Level: X-Archive-Number: 200510/346 X-Sequence-Number: 15114 --0-267068631-1130015525=:97706 Content-Type: multipart/alternative; boundary="0-2134442436-1130015525=:97706" --0-2134442436-1130015525=:97706 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Hi All, I am Kishore doing freelance development of J2EE applications. We switched to use Postgresql recently because of the advantages it has over other commercial databases. All went well untill recently, untill we began working on an application that needs to maintain a huge database. I am describing the problem we are facing below. Can you please take a look at the case, and help me in configuring the PostgreSQL. We have only two tables, one of which contains 97% of the data and the other table which contains 2.8% of the data. All other contain only the remaining 0.2% of data and are designed to support these two big tables. Currently we have 9 million of records in the first table and 0.2 million of records in the second table. We need to insert into the bigger table almost for every second , through out the life time. In addition, we receive at least 200,000 records a day at a fixed time. We are facing a critical situation because of the performance of the database. Even a basic query like select count(*) from bigger_table is taking about 4 minutes to return. The following is the system configuration. Database : Postgresql 7.3 OS : Redhat Linux Processor : Athlon, Memory : 2 GB We are expecting that at least 200 active connections need to be maintained through out the day. I am also attaching the configuration file that we are currently using. Can any you please suggest the best configuration to satisfy the above requirements? Thanks in advance. Thank you, Kishore. --------------------------------- Yahoo! FareChase - Search multiple travel sites in one click. --0-2134442436-1130015525=:97706 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: 8bit

Hi All,

 

I am Kishore doing freelance development of J2EE applications.

 

We switched to use Postgresql recently because of the advantages it has over other commercial databases. All went well untill recently, untill we began working on an application that needs to maintain  a huge database.

 

I am describing the problem we are facing below. Can you please take a look at the case, and help me in configuring the PostgreSQL.

 

We have only two tables, one of which contains 97% of the data and the other table which contains 2.8% of the data. All other contain only the remaining 0.2% of data and are designed to support these two big tables. Currently we have 9 million of records in the first table and 0.2 million of records in the second table.

We need to insert into the bigger table almost for every second , through out the life time. In addition, we receive at least 200,000 records a day at a fixed time.

We are facing a critical situation because of the performance of the database. Even a basic query like select count(*) from bigger_table is taking about 4 minutes to return.

The following is the system configuration.

Database : Postgresql 7.3
OS : Redhat Linux
Processor : Athlon,
Memory : 2 GB

We are expecting that at least 200 active connections need to be maintained through out the day.

 

I am also attaching the configuration file that we are currently using.



Can any you please suggest the best configuration to satisfy the above requirements?

Thanks in advance.

 

Thank  you,

Kishore.

 


Yahoo! FareChase - Search multiple travel sites in one click. --0-2134442436-1130015525=:97706-- --0-267068631-1130015525=:97706 Content-Type: application/octet-stream; name="postgresql.conf" Content-Transfer-Encoding: base64 Content-Description: 3963038301-postgresql.conf Content-Disposition: attachment; filename="postgresql.conf" Iw0KIyBQb3N0Z3JlU1FMIGNvbmZpZ3VyYXRpb24gZmlsZQ0KIyAtLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KIw0KIyBUaGlzIGZpbGUgY29uc2lz dHMgb2YgbGluZXMgb2YgdGhlIGZvcm06DQojDQojICAgbmFtZSA9IHZhbHVl DQojDQojIChUaGUgJz0nIGlzIG9wdGlvbmFsLikgV2hpdGUgc3BhY2UgbWF5 IGJlIHVzZWQuIENvbW1lbnRzIGFyZSBpbnRyb2R1Y2VkDQojIHdpdGggJyMn IGFueXdoZXJlIG9uIGEgbGluZS4gVGhlIGNvbXBsZXRlIGxpc3Qgb2Ygb3B0 aW9uIG5hbWVzIGFuZA0KIyBhbGxvd2VkIHZhbHVlcyBjYW4gYmUgZm91bmQg aW4gdGhlIFBvc3RncmVTUUwgZG9jdW1lbnRhdGlvbi4gVGhlDQojIGNvbW1l bnRlZC1vdXQgc2V0dGluZ3Mgc2hvd24gaW4gdGhpcyBmaWxlIHJlcHJlc2Vu dCB0aGUgZGVmYXVsdCB2YWx1ZXMuDQojDQojIEFueSBvcHRpb24gY2FuIGFs c28gYmUgZ2l2ZW4gYXMgYSBjb21tYW5kIGxpbmUgc3dpdGNoIHRvIHRoZQ0K IyBwb3N0bWFzdGVyLCBlLmcuICdwb3N0bWFzdGVyIC1jIGxvZ19jb25uZWN0 aW9ucz1vbicuIFNvbWUgb3B0aW9ucw0KIyBjYW4gYmUgY2hhbmdlZCBhdCBy dW4tdGltZSB3aXRoIHRoZSAnU0VUJyBTUUwgY29tbWFuZC4NCiMNCiMgVGhp cyBmaWxlIGlzIHJlYWQgb24gcG9zdG1hc3RlciBzdGFydHVwIGFuZCB3aGVu IHRoZSBwb3N0bWFzdGVyDQojIHJlY2VpdmVzIGEgU0lHSFVQLiBJZiB5b3Ug ZWRpdCB0aGUgZmlsZSBvbiBhIHJ1bm5pbmcgc3lzdGVtLCB5b3UgaGF2ZSAN CiMgdG8gU0lHSFVQIHRoZSBwb3N0bWFzdGVyIGZvciB0aGUgY2hhbmdlcyB0 byB0YWtlIGVmZmVjdCwgb3IgdXNlIA0KIyAicGdfY3RsIHJlbG9hZCIuDQoN Cg0KIz09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KDQoNCiMNCiMJQ29u bmVjdGlvbiBQYXJhbWV0ZXJzDQojDQp0Y3BpcF9zb2NrZXQgPXRydWUNCiNz c2wgPSBmYWxzZQ0KDQptYXhfY29ubmVjdGlvbnMgPTUwMA0KI3N1cGVydXNl cl9yZXNlcnZlZF9jb25uZWN0aW9ucyA9IDINCg0KcG9ydCA9NTIyMyANCiNo b3N0bmFtZV9sb29rdXAgPSBmYWxzZQ0KI3Nob3dfc291cmNlX3BvcnQgPSBm YWxzZQ0KDQojdW5peF9zb2NrZXRfZGlyZWN0b3J5ID0gJycNCiN1bml4X3Nv Y2tldF9ncm91cCA9ICcnDQojdW5peF9zb2NrZXRfcGVybWlzc2lvbnMgPSAw Nzc3CSMgb2N0YWwNCg0KI3ZpcnR1YWxfaG9zdCA9ICcnDQoNCiNrcmJfc2Vy dmVyX2tleWZpbGUgPSAnJw0KDQoNCiMNCiMJU2hhcmVkIE1lbW9yeSBTaXpl DQojDQpzaGFyZWRfYnVmZmVycyA9NDgwMDAgICAgICAjIG1pbiBtYXhfY29u bmVjdGlvbnMqMiBvciAxNiwgOEtCIGVhY2gNCiNtYXhfZnNtX3JlbGF0aW9u cyA9IDEwMDAJIyBtaW4gMTAsIGZzbSBpcyBmcmVlIHNwYWNlIG1hcCwgfjQw IGJ5dGVzDQojbWF4X2ZzbV9wYWdlcyA9IDEwMDAwCQkjIG1pbiAxMDAwLCBm c20gaXMgZnJlZSBzcGFjZSBtYXAsIH42IGJ5dGVzDQojbWF4X2xvY2tzX3Bl cl90cmFuc2FjdGlvbiA9IDY0CSMgbWluIDEwDQojd2FsX2J1ZmZlcnMgPSA4 CQkjIG1pbiA0LCB0eXBpY2FsbHkgOEtCIGVhY2gNCg0KIw0KIwlOb24tc2hh cmVkIE1lbW9yeSBTaXplcw0KIw0Kc29ydF9tZW0gPTIwNDgJCSMgbWluIDY0 LCBzaXplIGluIEtCDQojdmFjdXVtX21lbSA9IDgxOTIJCSMgbWluIDEwMjQs IHNpemUgaW4gS0INCg0KDQojDQojCVdyaXRlLWFoZWFkIGxvZyAoV0FMKQ0K Iw0KI2NoZWNrcG9pbnRfc2VnbWVudHMgPSAzCSMgaW4gbG9nZmlsZSBzZWdt ZW50cywgbWluIDEsIDE2TUIgZWFjaA0KI2NoZWNrcG9pbnRfdGltZW91dCA9 IDMwMAkjIHJhbmdlIDMwLTM2MDAsIGluIHNlY29uZHMNCiMNCiNjb21taXRf ZGVsYXkgPSAwCQkjIHJhbmdlIDAtMTAwMDAwLCBpbiBtaWNyb3NlY29uZHMN CiNjb21taXRfc2libGluZ3MgPSA1CQkjIHJhbmdlIDEtMTAwMA0KIw0KZnN5 bmMgPWZhbHNlDQojd2FsX3N5bmNfbWV0aG9kID0gZnN5bmMJIyB0aGUgZGVm YXVsdCB2YXJpZXMgYWNyb3NzIHBsYXRmb3JtczoNCiMJCQkJIyBmc3luYywg ZmRhdGFzeW5jLCBvcGVuX3N5bmMsIG9yIG9wZW5fZGF0YXN5bmMNCiN3YWxf ZGVidWcgPSAwCQkJIyByYW5nZSAwLTE2DQoNCg0KIw0KIwlPcHRpbWl6ZXIg UGFyYW1ldGVycw0KIw0KI2VuYWJsZV9zZXFzY2FuID0gdHJ1ZQ0KI2VuYWJs ZV9pbmRleHNjYW4gPSB0cnVlDQojZW5hYmxlX3RpZHNjYW4gPSB0cnVlDQoj ZW5hYmxlX3NvcnQgPSB0cnVlDQojZW5hYmxlX25lc3Rsb29wID0gdHJ1ZQ0K I2VuYWJsZV9tZXJnZWpvaW4gPSB0cnVlDQojZW5hYmxlX2hhc2hqb2luID0g dHJ1ZQ0KDQplZmZlY3RpdmVfY2FjaGVfc2l6ZSA9MTAwMDAwCSMgdHlwaWNh bGx5IDhLQiBlYWNoDQpyYW5kb21fcGFnZV9jb3N0ID0gMwkJIyB1bml0cyBh cmUgb25lIHNlcXVlbnRpYWwgcGFnZSBmZXRjaCBjb3N0DQojcmFuZG9tX3Bh Z2VfY29zdCA9IDQJCSMgdW5pdHMgYXJlIG9uZSBzZXF1ZW50aWFsIHBhZ2Ug ZmV0Y2ggY29zdA0KI2NwdV90dXBsZV9jb3N0ID0gMC4wMQkJIyAoc2FtZSkN CiNjcHVfaW5kZXhfdHVwbGVfY29zdCA9IDAuMDAxCSMgKHNhbWUpDQojY3B1 X29wZXJhdG9yX2Nvc3QgPSAwLjAwMjUJIyAoc2FtZSkNCg0KI2Zyb21fY29s bGFwc2VfbGltaXQgPSA4DQojam9pbl9jb2xsYXBzZV9saW1pdCA9IDgJIyAx IGRpc2FibGVzIGNvbGxhcHNpbmcgb2YgZXhwbGljaXQgSk9JTnMNCg0KI2Rl ZmF1bHRfc3RhdGlzdGljc190YXJnZXQgPSAxMAkjIHJhbmdlIDEtMTAwMA0K DQojDQojCUdFUU8gT3B0aW1pemVyIFBhcmFtZXRlcnMNCiMNCiNnZXFvID0g dHJ1ZQ0KI2dlcW9fc2VsZWN0aW9uX2JpYXMgPSAyLjAJIyByYW5nZSAxLjUt Mi4wDQojZ2Vxb190aHJlc2hvbGQgPSAxMQ0KI2dlcW9fcG9vbF9zaXplID0g MAkJIyBkZWZhdWx0IGJhc2VkIG9uIHRhYmxlcyBpbiBzdGF0ZW1lbnQsIA0K CQkJCSMgcmFuZ2UgMTI4LTEwMjQNCiNnZXFvX2VmZm9ydCA9IDENCiNnZXFv X2dlbmVyYXRpb25zID0gMA0KI2dlcW9fcmFuZG9tX3NlZWQgPSAtMQkJIyBh dXRvLWNvbXB1dGUgc2VlZA0KDQoNCiMNCiMJTWVzc2FnZSBkaXNwbGF5DQoj DQojc2VydmVyX21pbl9tZXNzYWdlcyA9IG5vdGljZQkjIFZhbHVlcywgaW4g b3JkZXIgb2YgZGVjcmVhc2luZyBkZXRhaWw6DQoJCQkJIyAgIGRlYnVnNSwg ZGVidWc0LCBkZWJ1ZzMsIGRlYnVnMiwgZGVidWcxLA0KCQkJCSMgICBpbmZv LCBub3RpY2UsIHdhcm5pbmcsIGVycm9yLCBsb2csIGZhdGFsLA0KCQkJCSMg ICBwYW5pYw0KI2NsaWVudF9taW5fbWVzc2FnZXMgPSBub3RpY2UJIyBWYWx1 ZXMsIGluIG9yZGVyIG9mIGRlY3JlYXNpbmcgZGV0YWlsOg0KCQkJCSMgICBk ZWJ1ZzUsIGRlYnVnNCwgZGVidWczLCBkZWJ1ZzIsIGRlYnVnMSwNCgkJCQkj ICAgbG9nLCBpbmZvLCBub3RpY2UsIHdhcm5pbmcsIGVycm9yDQojc2lsZW50 X21vZGUgPSBmYWxzZQ0KDQojbG9nX2Nvbm5lY3Rpb25zID0gZmFsc2UNCiNs b2dfcGlkID0gZmFsc2UNCiNsb2dfc3RhdGVtZW50ID0gZmFsc2UNCiNsb2df ZHVyYXRpb24gPSBmYWxzZQ0KI2xvZ190aW1lc3RhbXAgPSBmYWxzZQ0KDQoj bG9nX21pbl9lcnJvcl9zdGF0ZW1lbnQgPSBwYW5pYyAjIFZhbHVlcyBpbiBv cmRlciBvZiBpbmNyZWFzaW5nIHNldmVyaXR5Og0KCQkJCSAjICAgZGVidWc1 LCBkZWJ1ZzQsIGRlYnVnMywgZGVidWcyLCBkZWJ1ZzEsDQoJCQkJICMgICBp bmZvLCBub3RpY2UsIHdhcm5pbmcsIGVycm9yLCBwYW5pYyhvZmYpDQoNCiNk ZWJ1Z19wcmludF9wYXJzZSA9IGZhbHNlDQojZGVidWdfcHJpbnRfcmV3cml0 dGVuID0gZmFsc2UNCiNkZWJ1Z19wcmludF9wbGFuID0gZmFsc2UNCiNkZWJ1 Z19wcmV0dHlfcHJpbnQgPSBmYWxzZQ0KDQojZXhwbGFpbl9wcmV0dHlfcHJp bnQgPSB0cnVlDQoNCiMgcmVxdWlyZXMgVVNFX0FTU0VSVF9DSEVDS0lORw0K I2RlYnVnX2Fzc2VydGlvbnMgPSB0cnVlDQoNCg0KIw0KIwlTeXNsb2cNCiMN CiNzeXNsb2cgPSAwCQkJIyByYW5nZSAwLTINCiNzeXNsb2dfZmFjaWxpdHkg PSAnTE9DQUwwJw0KI3N5c2xvZ19pZGVudCA9ICdwb3N0Z3JlcycNCg0KDQoj DQojCVN0YXRpc3RpY3MNCiMNCiNzaG93X3BhcnNlcl9zdGF0cyA9IGZhbHNl DQojc2hvd19wbGFubmVyX3N0YXRzID0gZmFsc2UNCiNzaG93X2V4ZWN1dG9y X3N0YXRzID0gZmFsc2UNCiNzaG93X3N0YXRlbWVudF9zdGF0cyA9IGZhbHNl DQoNCiMgcmVxdWlyZXMgQlRSRUVfQlVJTERfU1RBVFMNCiNzaG93X2J0cmVl X2J1aWxkX3N0YXRzID0gZmFsc2UNCg0KDQojDQojCUFjY2VzcyBzdGF0aXN0 aWNzIGNvbGxlY3Rpb24NCiMNCiNzdGF0c19zdGFydF9jb2xsZWN0b3IgPSB0 cnVlDQojc3RhdHNfcmVzZXRfb25fc2VydmVyX3N0YXJ0ID0gdHJ1ZQ0KI3N0 YXRzX2NvbW1hbmRfc3RyaW5nID0gZmFsc2UNCiNzdGF0c19yb3dfbGV2ZWwg PSBmYWxzZQ0KI3N0YXRzX2Jsb2NrX2xldmVsID0gZmFsc2UNCg0KDQojDQoj CUxvY2sgVHJhY2luZw0KIw0KI3RyYWNlX25vdGlmeSA9IGZhbHNlDQoNCiMg cmVxdWlyZXMgTE9DS19ERUJVRw0KI3RyYWNlX2xvY2tzID0gZmFsc2UNCiN0 cmFjZV91c2VybG9ja3MgPSBmYWxzZQ0KI3RyYWNlX2x3bG9ja3MgPSBmYWxz ZQ0KI2RlYnVnX2RlYWRsb2NrcyA9IGZhbHNlDQojdHJhY2VfbG9ja19vaWRt aW4gPSAxNjM4NA0KI3RyYWNlX2xvY2tfdGFibGUgPSAwDQoNCg0KIw0KIwlN aXNjDQojDQojYXV0b2NvbW1pdCA9IHRydWUNCiNkeW5hbWljX2xpYnJhcnlf cGF0aCA9ICckbGliZGlyJw0KI3NlYXJjaF9wYXRoID0gJyR1c2VyLHB1Ymxp YycNCiNkYXRlc3R5bGUgPSAnaXNvLCB1cycNCiN0aW1lem9uZSA9IHVua25v d24JCSMgYWN0dWFsbHksIGRlZmF1bHRzIHRvIFRaIGVudmlyb25tZW50IHNl dHRpbmcNCiNhdXN0cmFsaWFuX3RpbWV6b25lcyA9IGZhbHNlDQojY2xpZW50 X2VuY29kaW5nID0gc3FsX2FzY2lpCSMgYWN0dWFsbHksIGRlZmF1bHRzIHRv IGRhdGFiYXNlIGVuY29kaW5nDQojYXV0aGVudGljYXRpb25fdGltZW91dCA9 IDYwCSMgMS02MDAsIGluIHNlY29uZHMNCiNkZWFkbG9ja190aW1lb3V0ID0g MTAwMAkjIGluIG1pbGxpc2Vjb25kcw0KI2RlZmF1bHRfdHJhbnNhY3Rpb25f aXNvbGF0aW9uID0gJ3JlYWQgY29tbWl0dGVkJw0KI21heF9leHByX2RlcHRo ID0gMTAwMDAJCSMgbWluIDEwDQojbWF4X2ZpbGVzX3Blcl9wcm9jZXNzID0g MTAwMAkjIG1pbiAyNQ0KI3Bhc3N3b3JkX2VuY3J5cHRpb24gPSB0cnVlDQoj c3FsX2luaGVyaXRhbmNlID0gdHJ1ZQ0KI3RyYW5zZm9ybV9udWxsX2VxdWFs cyA9IGZhbHNlDQojc3RhdGVtZW50X3RpbWVvdXQgPSAwCQkjIDAgaXMgZGlz YWJsZWQsIGluIG1pbGxpc2Vjb25kcw0KI2RiX3VzZXJfbmFtZXNwYWNlID0g ZmFsc2UNCiANCg0KDQojDQojCUxvY2FsZSBzZXR0aW5ncw0KIw0KIyAoaW5p dGlhbGl6ZWQgYnkgaW5pdGRiIC0tIG1heSBiZSBjaGFuZ2VkKQ0KTENfTUVT U0FHRVMgPSAnZW5fVVMnDQpMQ19NT05FVEFSWSA9ICdlbl9VUycNCkxDX05V TUVSSUMgPSAnZW5fVVMnDQpMQ19USU1FID0gJ2VuX1VTJw0K --0-267068631-1130015525=:97706--