public inbox for [email protected]help / color / mirror / Atom feed
Re: Is there any limit on the number of rows to import using copy command 6+ messages / 5 participants [nested] [flat]
* Re: Is there any limit on the number of rows to import using copy command @ 2025-07-23 08:50 [email protected] <[email protected]> 0 siblings, 3 replies; 6+ messages in thread From: [email protected] @ 2025-07-23 08:50 UTC (permalink / raw) To: pgsql-general Tried in PostgreSQL 11.11 , PostgreSQL 15.2 in Windows 10 Here we try to transfer data from one database to another (remote) database. Tables do have records ranging from 85000 to 3600000 along with smaller sized tables. No issues while transferring smaller sized tables. I here take one particular table [table1] which has 85000 records.The table got Primary Key, Foreign Key(s), Triggers. Trigger updates another table [table2]Table2 have 2 triggers, one to arrive a closing value and other to delete, if the closing value is zero. 1. Transfer the data from source database to a csv file. 85000 records transferred. No issues.2. Transfer the file to the remote location. No issues.3. Transfer the contents of the file to the table using Copy From command. - Fails when try to transfer all the 85000 records at once. Copy from command is Copy public.table1 From 'E:\temp\file1.csv' (FORMAT CSV, DELIMITER ',', HEADER TRUE) The above command succeeds, when1. The trigger in Table1 is disabled with all other constraints on. 2. The no. of rows is within 16000 or less, with Trigger enabled. We haven't tried with higher no of rows. The above command goes on infinite loop, when1. We try to transfer all 85000 rows at once, with Trigger and other constraints in table1 enabled. We waited for 1.5 hrs first time and 2.5 hrs second time before cancelling the operation. I read in the documentation that the fastest way to transfer data is to use Copy command. And I couldn't find any limit in transferring data using that command. One could easily transfer millions of rows using this command. Here are the triggers. Trigger function, which is called from Table1 on After Insert, Update, Delete Declare variety_code character varying(30);Declare company_code character varying(10);Declare branch_code character varying(10);Declare location_fk character varying(32);Declare opening_quantity numeric(17,3) ;Declare modified_user character varying(50) ;Declare modified_date timestamp without time zone ;Declare modified_computer character varying(50); BEGIN If (TG_OP = 'INSERT') Then company_code = NEW.companycode ; branch_code = NEW.branchcode ; location_fk = NEW.locationfk ; variety_code = NEW.barcode ; opening_quantity = Coalesce(NEW.openingquantity,0); End If ; If (TG_OP = 'UPDATE') Then company_code = NEW.companycode ; branch_code = NEW.branchcode ; location_fk = NEW.locationfk ; variety_code = NEW.barcode ; opening_quantity = Coalesce(NEW.openingquantity,0) - OLD.openingquantity ; modified_user = NEW.modifieduser ; modified_date = NEW.modifieddate ; modified_computer = NEW.modifiedcomputer ; End If ; If (TG_OP = 'DELETE') Then company_code = OLD.companycode ; branch_code = OLD.branchcode ; location_fk = OLD.locationfk ; variety_code = OLD.barcode ; opening_quantity = OLD.openingquantity * -1 ; modified_user = OLD.modifieduser ; modified_date = OLD.modifieddate ; modified_computer = OLD.modifiedcomputer ; End If ; If (Select Count(*) From table2 WHERE companycode = company_code AND branchcode = branch_code AND locationfk = location_fk AND barcode = variety_code ) > 0 Then BEGIN UPDATE table2 SET openingquantity = Coalesce(openingquantity,0) + opening_quantity, modifieduser = modified_user, modifieddate = modified_date, modifiedcomputer = modified_computer WHERE companycode = company_code AND branchcode = branch_code AND locationfk = location_fk AND barcode = variety_code ; END ; Else BEGIN INSERT INTO public.table2( barcodestockpk, companycode, branchcode, locationfk, barcode, baleopenheaderfk, lrentryheaderfk, lrentrydetailfk, baleopendetailfk, lrentrydetailsequencenumber, baleopendetailsequencenumber, barcodeopeningstockfk, sequencenumber, varietyfk, brandfk, modelfk, patternfk, shadefk, materialfk, finishfk, sizefk, meterbreakup, unit, barcodesinglebulk, barcodeitem, effectiverate, openingquantity, barcodedquantity, purchasereturnquantity, salesquantity, salescancellationquantity, salesreturnquantity, salesreturncancellationquantity, stockissuequantity, stockreceiptquantity, locationissuequantity, locationreceiptquantity, branchissuequantity, branchreceiptquantity, closingstock, salesrate, mrprate, labelrate, ratecode, discountpercent, discountrate, defectiveitem, locked, insertuser, insertdate, insertcomputer, modifieduser, modifieddate, modifiedcomputer, wsrate, reversecalculation, hsnnumber) VALUES ( replace(uuid_generate_v4()::text, '-', ''), company_code , branch_code, location_fk, variety_code, Null, Null,Null, Null, Null, Null, NEW.barcodeopeningstockpk, NEW.Sequencenumber, NEW.varietyfk, NEW.brandfk, NEW.modelfk, NEW.patternfk, NEW.shadefk, NEW.materialfk, NEW.finishfk, NEW.sizefk, NEW.meterbreakup, NEW.unit, NEW.barcodesinglebulk, NEW.barcodeitem, NEW.effectiverate, opening_quantity, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, NEW.salesrate, NEW.mrprate, NEW.labelrate, NEW.ratecode, 0, 0, 'N', NEW.locked, NEW.insertuser, NEW.insertdate, NEW.insertcomputer, NEW.modifieduser, NEW.modifieddate, NEW.modifiedcomputer, NEW.wsrate, 'N', NEW.hsnnumber); END ; End IF ; RETURN NEW ;END ; Trigger functions in Table 2First Trigger, which calculates the closing stock, on before insert, update BEGIN If (TG_OP = 'INSERT') Then NEW.closingstock = coalesce(NEW.barcodedquantity,0) + coalesce(NEW.openingquantity,0) - coalesce(NEW.salesquantity,0) + coalesce(NEW.salesreturnquantity,0) + coalesce(NEW.salescancellationquantity,0) - coalesce(NEW.salesreturncancellationquantity,0) - coalesce(NEW.purchasereturnquantity,0) - coalesce(NEW.stockissuequantity,0) + coalesce(NEW.stockreceiptquantity,0) - coalesce(NEW.locationissuequantity,0) + coalesce(NEW.locationreceiptquantity,0) - coalesce(NEW.branchissuequantity,0) + coalesce(NEW.branchreceiptquantity,0) ; Return New ; End If ; If (TG_OP = 'UPDATE') Then NEW.closingstock = coalesce(NEW.barcodedquantity,0) + coalesce(NEW.openingquantity,0) - coalesce(NEW.salesquantity,0) + coalesce(NEW.salesreturnquantity,0) + coalesce(NEW.salescancellationquantity,0) - coalesce(NEW.salesreturncancellationquantity,0) - coalesce(NEW.purchasereturnquantity,0) - coalesce(NEW.stockissuequantity,0) + coalesce(NEW.stockreceiptquantity,0) - coalesce(NEW.locationissuequantity,0) + coalesce(NEW.locationreceiptquantity,0) - coalesce(NEW.branchissuequantity,0) + coalesce(NEW.branchreceiptquantity,0) ; Return New ; End If ;END Second trigger, which deletes row, when every value is zero, after insert, update, delete Begin If Coalesce(NEW.openingquantity,0) = 0 and Coalesce(NEW.barcodedquantity,0) = 0 and Coalesce(NEW.salesquantity,0) = 0 and Coalesce(NEW.salescancellationquantity,0) = 0 and Coalesce(NEW.salesreturnquantity,0) = 0 and Coalesce(NEW.salesreturncancellationquantity,0) = 0 and Coalesce(NEW.purchasereturnquantity,0) = 0 and Coalesce(NEW.stockissuequantity,0) = 0 and Coalesce(NEW.stockreceiptquantity,0) = 0 and Coalesce(NEW.locationissuequantity,0) = 0 and Coalesce(NEW.locationreceiptquantity,0) = 0 and Coalesce(NEW.branchissuequantity,0) = 0 and Coalesce(NEW.branchreceiptquantity,0) = 0 Then Delete From tx_barcode_stock Where barcodestockpk = new.barcodestockpk ; End If ; Return New ;END Any (other) suggestion to transfer successfully is really appreciated. Happiness Always BKR Sivaprakash ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Is there any limit on the number of rows to import using copy command @ 2025-07-23 10:16 Laurenz Albe <[email protected]> parent: [email protected] <[email protected]> 2 siblings, 1 reply; 6+ messages in thread From: Laurenz Albe @ 2025-07-23 10:16 UTC (permalink / raw) To: [email protected] <[email protected]>; pgsql-general On Wed, 2025-07-23 at 08:50 +0000, [email protected] wrote: > Tried in PostgreSQL 11.11 , PostgreSQL 15.2 in Windows 10 Both of these choices are unsavory. Don't use the unsupported v11, and use 15.13 with v15. > Here we try to transfer data from one database to another (remote) database. > > Tables do have records ranging from 85000 to 3600000 along with smaller sized tables. > No issues while transferring smaller sized tables. > > I here take one particular table [table1] which has 85000 records. > The table got Primary Key, Foreign Key(s), Triggers. Trigger updates another table [table2] > Table2 have 2 triggers, one to arrive a closing value and other to delete, if the closing value is zero. > > 1. Transfer the data from source database to a csv file. 85000 records transferred. No issues. > 2. Transfer the file to the remote location. No issues. > 3. Transfer the contents of the file to the table using Copy From command. - Fails when try to transfer all the 85000 records at once. > > Copy from command is > > Copy public.table1 From 'E:\temp\file1.csv' (FORMAT CSV, DELIMITER ',', HEADER TRUE) > > The above command succeeds, when > 1. The trigger in Table1 is disabled with all other constraints on. > 2. The no. of rows is within 16000 or less, with Trigger enabled. We haven't tried with higher no of rows. > > The above command goes on infinite loop, when > 1. We try to transfer all 85000 rows at once, with Trigger and other constraints in table1 enabled. > We waited for 1.5 hrs first time and 2.5 hrs second time before cancelling the operation. > > I read in the documentation that the fastest way to transfer data is to use Copy command. > And I couldn't find any limit in transferring data using that command. > One could easily transfer millions of rows using this command. There is no limit for the number of rows that get created by a single COPY. You should research why processing fails for higher row counts: - Are there any messages on the client or the server side? - Is the backend process on the server busy (consuming CPU) when processing hangs? - Do you see locks or other wait events in "pg_stat_activity"? > Here are the triggers. > > Trigger function, which is called from Table1 on After Insert, Update, Delete One thing you could try is a BEFORE trigger. That should work the same, unless there are foreign key constraints. Do you see high memory usage or paging for the backend process when the COPY hangs? > [...] > If (Select Count(*) > From table2 > WHERE companycode = company_code > AND branchcode = branch_code > AND locationfk = location_fk > AND barcode = variety_code ) > 0 Then > [...] That may well be slow, particularly without a matching index. A better way to write that would be IF EXISTS (SELECT 1 FROM table2 WHERE ...) because that can stop processing after the first match. It still needs an index for fast processing. Yours, Laurenz Albe ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Is there any limit on the number of rows to import using copy command @ 2025-07-23 11:55 [email protected] <[email protected]> parent: Laurenz Albe <[email protected]> 0 siblings, 1 reply; 6+ messages in thread From: [email protected] @ 2025-07-23 11:55 UTC (permalink / raw) To: pgsql-general; Laurenz Albe <[email protected]> Thanks Laurenz Albe, 1. I tried running Copy From command from PGAdmin. 2. I ran pg_stat_activity also in another tab [ PGAdmin ]. What I observed,1. In about 2 min, in the Dashboard of PGAdmin, the colour changed to Orange for that particular pid.2. After few seconds, the colour again changed to Red. 3. The stat column in both Dashboard and pg_stat_activity.state shows 'active'.4. No messages relevant to this pid in postgres' log files5. CPU usage is normal, hovering around 36% overall, 15-18% for postgresql server, from the beginning and even after 3 min.6. We didn't run any other application in that machine. 7. Unique Index is there in table2 which will return only one row for that Select Count(*) query.8. No record is there in the target table when transfer started. This transfer is the first batch for that particular table.9. Once color turned into Red, I could cancel the query execuion in PGAdmin, which immediately stops the execution.10. I could not see any locks or blocking pids. Happiness Always BKR Sivaprakash On Wednesday 23 July, 2025 at 03:46:40 pm IST, Laurenz Albe <[email protected]> wrote: On Wed, 2025-07-23 at 08:50 +0000, [email protected] wrote: > Tried in PostgreSQL 11.11 , PostgreSQL 15.2 in Windows 10 Both of these choices are unsavory. Don't use the unsupported v11, and use 15.13 with v15. > Here we try to transfer data from one database to another (remote) database. > > Tables do have records ranging from 85000 to 3600000 along with smaller sized tables. > No issues while transferring smaller sized tables. > > I here take one particular table [table1] which has 85000 records. > The table got Primary Key, Foreign Key(s), Triggers. Trigger updates another table [table2] > Table2 have 2 triggers, one to arrive a closing value and other to delete, if the closing value is zero. > > 1. Transfer the data from source database to a csv file. 85000 records transferred. No issues. > 2. Transfer the file to the remote location. No issues. > 3. Transfer the contents of the file to the table using Copy From command. - Fails when try to transfer all the 85000 records at once. > > Copy from command is > > Copy public.table1 From 'E:\temp\file1.csv' (FORMAT CSV, DELIMITER ',', HEADER TRUE) > > The above command succeeds, when > 1. The trigger in Table1 is disabled with all other constraints on. > 2. The no. of rows is within 16000 or less, with Trigger enabled. We haven't tried with higher no of rows. > > The above command goes on infinite loop, when > 1. We try to transfer all 85000 rows at once, with Trigger and other constraints in table1 enabled. > We waited for 1.5 hrs first time and 2.5 hrs second time before cancelling the operation. > > I read in the documentation that the fastest way to transfer data is to use Copy command. > And I couldn't find any limit in transferring data using that command. > One could easily transfer millions of rows using this command. There is no limit for the number of rows that get created by a single COPY. You should research why processing fails for higher row counts: - Are there any messages on the client or the server side? - Is the backend process on the server busy (consuming CPU) when processing hangs? - Do you see locks or other wait events in "pg_stat_activity"? > Here are the triggers. > > Trigger function, which is called from Table1 on After Insert, Update, Delete One thing you could try is a BEFORE trigger. That should work the same, unless there are foreign key constraints. Do you see high memory usage or paging for the backend process when the COPY hangs? > [...] > If (Select Count(*) > From table2 > WHERE companycode = company_code > AND branchcode = branch_code > AND locationfk = location_fk > AND barcode = variety_code ) > 0 Then > [...] That may well be slow, particularly without a matching index. A better way to write that would be IF EXISTS (SELECT 1 FROM table2 WHERE ...) because that can stop processing after the first match. It still needs an index for fast processing. Yours, Laurenz Albe ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Is there any limit on the number of rows to import using copy command @ 2025-07-23 13:49 David Barbour <[email protected]> parent: [email protected] <[email protected]> 0 siblings, 0 replies; 6+ messages in thread From: David Barbour @ 2025-07-23 13:49 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: pgsql-general; Laurenz Albe <[email protected]> You might also consider Backup/Restore. It appears you're not concerned with data being inserted into the source table after the backup is complete? If so, you can then easily sync the two post restore. On Wed, Jul 23, 2025 at 6:55 AM [email protected] < [email protected]> wrote: > Thanks Laurenz Albe, > > 1. I tried running Copy From command from PGAdmin. > 2. I ran pg_stat_activity also in another tab [ PGAdmin ]. > > What I observed, > 1. In about 2 min, in the Dashboard of PGAdmin, the colour changed to > Orange for that particular pid. > 2. After few seconds, the colour again changed to Red. > 3. The stat column in both Dashboard and pg_stat_activity.state shows > 'active'. > 4. No messages relevant to this pid in postgres' log files > 5. CPU usage is normal, hovering around 36% overall, 15-18% for > postgresql server, from the beginning and even after 3 min. > 6. We didn't run any other application in that machine. > 7. Unique Index is there in table2 which will return only one row for > that Select Count(*) query. > 8. No record is there in the target table when transfer started. This > transfer is the first batch for that particular table. > 9. Once color turned into Red, I could cancel the query execuion in > PGAdmin, which immediately stops the execution. > 10. I could not see any locks or blocking pids. > > Happiness Always > BKR Sivaprakash > > On Wednesday 23 July, 2025 at 03:46:40 pm IST, Laurenz Albe < > [email protected]> wrote: > > > On Wed, 2025-07-23 at 08:50 +0000, [email protected] wrote: > > Tried in PostgreSQL 11.11 , PostgreSQL 15.2 in Windows 10 > > Both of these choices are unsavory. Don't use the unsupported v11, > and use 15.13 with v15. > > > Here we try to transfer data from one database to another (remote) > database. > > > > Tables do have records ranging from 85000 to 3600000 along with smaller > sized tables. > > No issues while transferring smaller sized tables. > > > > I here take one particular table [table1] which has 85000 records. > > The table got Primary Key, Foreign Key(s), Triggers. Trigger updates > another table [table2] > > Table2 have 2 triggers, one to arrive a closing value and other to > delete, if the closing value is zero. > > > > 1. Transfer the data from source database to a csv file. 85000 records > transferred. No issues. > > 2. Transfer the file to the remote location. No issues. > > 3. Transfer the contents of the file to the table using Copy From > command. - Fails when try to transfer all the 85000 records at once. > > > > Copy from command is > > > > Copy public.table1 From 'E:\temp\file1.csv' (FORMAT CSV, DELIMITER ',', > HEADER TRUE) > > > > The above command succeeds, when > > 1. The trigger in Table1 is disabled with all other constraints on. > > 2. The no. of rows is within 16000 or less, with Trigger enabled. We > haven't tried with higher no of rows. > > > > The above command goes on infinite loop, when > > 1. We try to transfer all 85000 rows at once, with Trigger and other > constraints in table1 enabled. > > We waited for 1.5 hrs first time and 2.5 hrs second time before > cancelling the operation. > > > > I read in the documentation that the fastest way to transfer data is to > use Copy command. > > And I couldn't find any limit in transferring data using that command. > > One could easily transfer millions of rows using this command. > > There is no limit for the number of rows that get created by a single COPY. > > You should research why processing fails for higher row counts: > - Are there any messages on the client or the server side? > - Is the backend process on the server busy (consuming CPU) when > processing hangs? > - Do you see locks or other wait events in "pg_stat_activity"? > > > Here are the triggers. > > > > Trigger function, which is called from Table1 on After Insert, Update, > Delete > > One thing you could try is a BEFORE trigger. That should work the same, > unless > there are foreign key constraints. Do you see high memory usage or paging > for > the backend process when the COPY hangs? > > > [...] > > If (Select Count(*) > > From table2 > > WHERE companycode = company_code > > AND branchcode = branch_code > > AND locationfk = location_fk > > AND barcode = variety_code ) > 0 Then > > [...] > > That may well be slow, particularly without a matching index. > A better way to write that would be > > IF EXISTS (SELECT 1 FROM table2 > > WHERE ...) > > > because that can stop processing after the first match. > It still needs an index for fast processing. > > Yours, > Laurenz Albe > > ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Is there any limit on the number of rows to import using copy command @ 2025-07-23 14:41 Adrian Klaver <[email protected]> parent: [email protected] <[email protected]> 2 siblings, 0 replies; 6+ messages in thread From: Adrian Klaver @ 2025-07-23 14:41 UTC (permalink / raw) To: [email protected] <[email protected]>; pgsql-general On 7/23/25 01:50, [email protected] wrote: > Tried in PostgreSQL 11.11 , PostgreSQL 15.2 in Windows 10 > > The above command succeeds, when > 1. The trigger in Table1 is disabled with all other constraints on. > 2. The no. of rows is within 16000 or less, with Trigger enabled. We > haven't tried with higher no of rows. Why not? > > The above command goes on infinite loop, when > 1. We try to transfer all 85000 rows at once, with Trigger and other > constraints in table1 enabled. We waited for 1.5 hrs first time and 2.5 > hrs second time before cancelling the operation. Try with the triggers disabled. > > I read in the documentation that the fastest way to transfer data is to > use Copy command. And I couldn't find any limit in transferring data > using that command. One could easily transfer millions of rows using > this command. It is, I have used it for much larger datasets then 85000 rows and it completed in less time. As example using Duckdb it took the NYC taxi data set yellow_tripdata_2023-09.parquet, transformed it and loaded using COPY in 5.4 secs for ~2.8 million rows. FYI, BEGIN in plpgsql is not the same as in SQL. In plpgsql it represents a block. I don't think you need the BEGIN/END around the UPDATE and INSERT queries. See https://www.postgresql.org/docs/current/plpgsql-structure.html for more information. > Any (other) suggestion to transfer successfully is really appreciated. > > Happiness Always > BKR Sivaprakash > -- Adrian Klaver [email protected] ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Is there any limit on the number of rows to import using copy command @ 2025-07-23 17:49 Merlin Moncure <[email protected]> parent: [email protected] <[email protected]> 2 siblings, 0 replies; 6+ messages in thread From: Merlin Moncure @ 2025-07-23 17:49 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: pgsql-general On Wed, Jul 23, 2025 at 2:51 AM [email protected] <[email protected]> wrote: > > Tried in PostgreSQL 11.11 , PostgreSQL 15.2 in Windows 10 > > Here we try to transfer data from one database to another (remote) database. > > Tables do have records ranging from 85000 to 3600000 along with smaller sized tables. > No issues while transferring smaller sized tables. > > I here take one particular table [table1] which has 85000 records. > The table got Primary Key, Foreign Key(s), Triggers. Trigger updates another table [table2] > Table2 have 2 triggers, one to arrive a closing value and other to delete, if the closing value is zero. > > 1. Transfer the data from source database to a csv file. 85000 records transferred. No issues. > 2. Transfer the file to the remote location. No issues. > 3. Transfer the contents of the file to the table using Copy From command. - Fails when try to transfer all the 85000 records at once. > > Copy from command is > > Copy public.table1 From 'E:\temp\file1.csv' (FORMAT CSV, DELIMITER ',', HEADER TRUE) > > The above command succeeds, when > 1. The trigger in Table1 is disabled with all other constraints on. > 2. The no. of rows is within 16000 or less, with Trigger enabled. We haven't tried with higher no of rows. > > The above command goes on infinite loop, when > 1. We try to transfer all 85000 rows at once, with Trigger and other constraints in table1 enabled. We waited for 1.5 hrs first time and 2.5 hrs second time before cancelling the operation. > > I read in the documentation that the fastest way to transfer data is to use Copy command. And I couldn't find any limit in transferring data using that command. One could easily transfer millions of rows using this command. Most likely, you are getting yourself into trouble with the trigger dependencies. Triggers are powerful, but also can be dangerous, and this could be 'wrong tool for the job' situation. Here are some general tips: * pg_trigger_depth(): can tell you if trigger A calls trigger B and back to trigger A, etc. you can use it with raise notify, and also use it to guard execution on CREATE TRIGGER * reconfiguring your logic to statement level triggers can be a good idea. this can take some thinking, but can be much more efficient when bulk processing since trigger execution can be deferred until the load completes. (one trick is to use now() to check for records inserted since it is stable though the transaction) * reconfiguring your logic to a procedure can be a better idea; COPY your data into some staging tables (perhaps temp, and indexed), then write to various tables with joins, upserts, etc. merlin ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2025-07-23 17:49 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-07-23 08:50 Re: Is there any limit on the number of rows to import using copy command [email protected] <[email protected]> 2025-07-23 10:16 ` Laurenz Albe <[email protected]> 2025-07-23 11:55 ` [email protected] <[email protected]> 2025-07-23 13:49 ` David Barbour <[email protected]> 2025-07-23 14:41 ` Adrian Klaver <[email protected]> 2025-07-23 17:49 ` Merlin Moncure <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox