Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1eflaz-0003Hw-R5 for pgsql-performance@arkaria.postgresql.org; Sun, 28 Jan 2018 11:58:02 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eflax-0002el-GW for pgsql-performance@arkaria.postgresql.org; Sun, 28 Jan 2018 11:57:59 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1eflax-0002eZ-98 for pgsql-performance@lists.postgresql.org; Sun, 28 Jan 2018 11:57:59 +0000 Received: from symbm03.itasdata.no ([128.39.191.25]) by magus.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1eflat-0000vo-9U for pgsql-performance@postgresql.org; Sun, 28 Jan 2018 11:57:58 +0000 X-AuditID: 8027bf19-7afff70000000a0f-73-5a6dbac2a8f9 Received: from mail.nibio.no (pf-18461.bioforsk.no [128.39.184.61]) by symbm03.itasdata.no (symbm03.itasdata.no) with SMTP id 5C.B9.02575.2CABD6A5; Sun, 28 Jan 2018 12:57:54 +0100 (CET) Received: from NIBIOEXCH01.int.nibio.no (10.100.32.20) by NIBIOEXCH01.int.nibio.no (10.100.32.20) with Microsoft SMTP Server (TLS) id 15.1.225.42; Sun, 28 Jan 2018 12:57:53 +0100 Received: from NIBIOEXCH01.int.nibio.no ([10.100.32.20]) by NIBIOEXCH01.int.nibio.no ([10.100.32.20]) with mapi id 15.01.0225.041; Sun, 28 Jan 2018 12:57:53 +0100 From: Lars Aksel Opsahl To: Mariel Cherkassky , PostgreSQL mailing lists Subject: SV: copy csv into partitioned table with unique index Thread-Topic: copy csv into partitioned table with unique index Thread-Index: AQHTmBgVdxJRSfTyLUmkPaCstPI4rKOJLVYq Date: Sun, 28 Jan 2018 11:57:53 +0000 Message-ID: References: In-Reply-To: Accept-Language: nb-NO, en-US Content-Language: nb-NO X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [10.100.44.6] Content-Type: multipart/alternative; boundary="_000_be608b72969a4ef3a9531cd273211e4enibiono_" MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprNIsWRmVeSWpSXmKPExsXSoL7DVvfQrtwog8MPmC06761nsbg+8xyz A5PHzll32T0u7JvGHsAUxWWTkpqTWZZapG+XwJVx9PoB1oL5nhUnNkxhbGA8YdPFyMkhIWAi MWtfD0sXIxeHkMBuRoneg3cYIZxtjBJPjrYzQTjLGCU6n3czgbSwCehJvN/XxA5iiwgUSbTs OsLaxcjBISxgJ3G5KQsibC/xb9ESRpCwiICRxMRPqSBhFgFVia2TP7GB2LwC5hLffp8BmyIk ECBx49BrsOmcAoES7zt2sYO0MgrISsxt4gUJMwuIS8w4eocF4mYBiSV7zjND2KISLx//Y4Ww dSTOXn/CCGHLS2z/u5kJZAyzQLzEvr8uEFsFJU7OfMIygVF0FpKpsxCqZiGpgijRk7gxdQob hK0tsWzha2YIW1dixr9DLMjiCxjZVzEKF1fmJuUaGOtlliQWpySWJOrl5W9iBEfZfskdjN+u uxxiFOBgVOLhvXEyJ0qINbEMqOEQowQHs5II77HpuVFCvCmJlVWpRfnxRaU5qcWHGKU5WJTE ecUif0QKCaQnlqRmp6YWpBbBZJk4OKUaGDPnSLUWhe3cciV06ve/cota3jt2BN8u/aMuuer3 t63vn7N/zwzesO3+s4fijrw51SwXVkWEbO5z37S/fm3Og6N3Od3+B9fu0TyrleEwR7XjwrL3 H5u2z3f5x/fijtGEnfaXReb0rc7KOhm459CdxhtLom/37Lvgrq9uYsHm7aL7z+/niZeTni5X YinOSDTUYi4qTgQAhjeDB64CAAA= List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --_000_be608b72969a4ef3a9531cd273211e4enibiono_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi We had the same problems with performance when testing with more than 100 b= illion weather observations. We now have a solution where we can push betwe= en 300 and 400 million weather observations pr. Second into the database. We download date from NetCDF files. The date are joined based on time and g= eolocation, so data from many different NetCDF should end up in the same ro= w and table. Doing this based on database join and table update was taking a long time a= s you also have noticed. To get high performance we ended up this solution - Uses os commands like awk,sed,join,cut,.. to prepare CSV file for the dat= abase. - Use multithreads. - Insert data directly into child tables. - No triggers, constraints and indexes on working table. - Don't update rows. - Unlogged tables. We first download NetCDF and make CSV files that fits in perfect for the co= py a command and with complete files for each child tables it's created for= , this is a time consuming operation. So before the copy in into database we just do a truncate on the selected t= able. We are then able to insert between 300 and 400 mill. weather observat= ions pr. Second. We have 11 observations pr row so it means around 35 mill = rows pr second. We have one child table for each year and month. The database we working on have 16 dual core CPU's and SSD discs. When test= ing I was running 11 threads in parallel. Indexes and constraints are added later based on needs. How can you take on chance om using something like unlogged tables? Linux system are quite stable and if we keep the a copy of the CVS files it= does not take long time to insert data after crash. You can also change your table to logged later if you need to secure your d= ata in the database. Lars ________________________________ Fra: Mariel Cherkassky Sendt: 28. januar 2018 10:11 Til: PostgreSQL mailing lists Emne: copy csv into partitioned table with unique index Hi, I configured a master table that is called "year_2018" : create table year_2018(a int,b int c date); The master table has a unique constraint on those 3 columns so that I wont = have any duplicated rows. Moreover, I configured a before insert trigger on= that table that creates a child table for each day in the year. The child = should include all the data related to that specific day. Now, every day I got a csv file that I need to load to the year table. I mu= st load the data as fast as possible but I have 2 problems : 1)I must load the data as a bulk - via the copy command. However, the copy = command fails because sometimes I have duplicated rows. 2)I tried to use the pgloader extension but it fails because I have a trigg= er before each insert. -I cant load all the data into a temp table and then run insert into year_2= 018 select * from temp because it takes too much time. Any idea ? --_000_be608b72969a4ef3a9531cd273211e4enibiono_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi


We had the same problems= with performance when testing with more than 100 billion weather observati= ons. We now have a solution where we can push between 300 and 400 million w= eather observations pr. Second into the database.


We download date from Ne= tCDF files. The date are joined based on time and geolocation, so data from= many different NetCDF should end up in the same row and table.


Doing this based on data= base join and table update was taking a long time as you also have noticed.=


To get high performance = we ended up this solution

- Uses os commands like = awk,sed,join,cut,.. to prepare CSV file for the database.

- Use multithreads.

- Insert data directly i= nto child tables.

- No triggers, constrain= ts and indexes on working table.

- Don’t update row= s.

- Unlogged tables.


We first download NetCDF= and make CSV files that fits in perfect for the copy a command and with co= mplete files for each child tables it’s created for, this is a time c= onsuming operation.


So before the copy in in= to database we just do a truncate on the selected table. We are then able t= o insert between 300 and 400 mill. weather observations pr. Second. We have= 11 observations pr row so it means around 35 mill rows pr second. We have one child table for each year and m= onth.


The database we working = on have 16 dual core CPU’s and SSD discs. When testing I was running = 11 threads in parallel.


Indexes and constraints = are added later based on needs.


How can you take on chan= ce om using something like unlogged tables?


Linux system are quite s= table and if we keep the a copy of the CVS files it does not take long time= to insert data after crash.


You can also change your= table to logged later if you need to secure your data in the database.


Lars




Fra: Mariel Cherkassky <= mariel.cherkassky@gmail.com>
Sendt: 28. januar 2018 10:11
Til: PostgreSQL mailing lists
Emne: copy csv into partitioned table with unique index
 
Hi,
I configured a master table that is called "year_2018= " : 
create table year_2018(a int,b int c date); 

The master table has a unique constraint on those 3 column= s so that I wont have any duplicated rows. Moreover, I configured a before = insert trigger on that table that creates a child table for each day in the= year. The child should include all the data related to that specific day.

Now, every day I got a csv file that I need to load to the= year table. I must load the data as fast as possible but I have 2 problems= :
1)I must load the data as a bulk - via the copy command. H= owever, the copy command fails because sometimes I have duplicated rows.
2)I tried to use the pgloader extension but it fails becau= se I have a trigger before each insert.

-I cant load all the data into a temp table and then run i= nsert into year_2018 select * from temp because it takes too much time.

Any idea ?


--_000_be608b72969a4ef3a9531cd273211e4enibiono_--