pg.ddx.io pgsql-bugs@postgresql.org mailing list archive help / color / mirror / Atom feed From: ocean_li_996 <ocean_li_996@163.com>
To: pgsql-bugs@lists.postgresql.org
Subject: Re:BUG #18369: logical decoding core on AssertTXNLsnOrder()
Date: Wed, 28 Feb 2024 15:57:37 +0800 (CST)
Message-ID: <6d0e80d6.c1fc.18deeb8120a.Coremail.ocean_li_996@163.com> (raw )
In-Reply-To: <18369-ad61699bf91c5bc0@postgresql.org >
References: <18369-ad61699bf91c5bc0@postgresql.org >
At 2024-02-28 15:53:30, "PG Bug reporting form" <noreply@postgresql.org> wrote:
>1) The WAL records from restart_lsn to the corresponding lsn when the issue
>occurred,
>2) personal analysis of the problem,
>3) the steps to reproduce the issue,
>4) personal proposed solution
>will be posted later under this thread.
>
1) The WAL records from restart_lsn to the corresponding lsn when the issue occurred is supported in attachment file 1.
2) As indicated in 1), some invalidation messages are generated in 19933 top xact. After the decoding restarted, the invalidation messages will make 19933 top xact and its subtransaction(s) to be marked as containing catalog change while processing its commit record(see SnapBuildXidSetCatalogChanges() ). In this step, the corresponding subxacts which never procedded before are added into ReorderBuffer with the same first_lsn as top-level xact. Then, the check in AssertTXNLsnOrder() will failed if the number of subxact mentioned above more than 1.
3) The patch to reproduce the issue is supported in attachment file 2. DML on temporary table can consume xid and not log any WAL RECORD except it's the firtst subxact of top xact(log ASSIGNMENT record). So we use DML on temporary table to generate two "never procedded before" sunxacts in on top xact.
4) Since it is already known to be a subxact before being add into ReorderBuffer, I think an appropriate fix is extending the ReorderBufferXidSetCatalogChanges function with an is_top parameter to indicate whether the xact is a top-level xact.
For a subxact, it would not be added to the toplevel_by_lsn list and would not undergo the AssertTXNLsnOrder check. Of course, it is necessary to introduce a check to verify whether a node is in the list when attempting to remove a node from toplevel_by_lsn.
The specific fix patch is provided in Attachment 3.
Thanks
Haiyang Li
Attachments: download all (3 files, .tar.gz)
xid_19933_wal_record.txt (text/plain, 9.1K) download
rmgr: Standby len (rec/tot): 54/ 54, tx: 0, lsn: 2/F171B8A8, prev 2/F171AA38, desc: RUNNING_XACTS nextXid 19934 latestCompletedXid 19932 oldestRunningXid 19933; 1 xacts: 19933
rmgr: Transaction len (rec/tot): 38/ 38, tx: 0, lsn: 2/F171B8E0, prev 2/F171B8A8, desc: ASSIGNMENT xtop 19933: subxacts: 19934
rmgr: Heap len (rec/tot): 54/ 54, tx: 19936, lsn: 2/F171B908, prev 2/F171B8E0, desc: LOCK off 23: xid 19936: flags 0 LOCK_ONLY EXCL_LOCK KEYS_UPDATED , blkref #0: rel 1663/17298/35747 blk 0
rmgr: XLOG len (rec/tot): 210/ 210, tx: 0, lsn: 2/F171B940, prev 2/F171B908, desc: POLAR_WAL of slot test_logical_decode_slot_0 with restart 2/F171A748 and xid 0 confirmed to 2/F171B8E0
rmgr: XLOG len (rec/tot): 49/ 4441, tx: 0, lsn: 2/F171BA18, prev 2/F171B940, desc: FPI_FOR_HINT , blkref #0: rel 1663/17298/34448 blk 2543 FPW
rmgr: Heap2 len (rec/tot): 3138/ 3138, tx: 19944, lsn: 2/F171CB90, prev 2/F171BA18, desc: MULTI_INSERT+INIT 39 tuples, blkref #0: rel 1663/17298/34448 blk 2544
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D7D8, prev 2/F171CB90, desc: INSERT_LEAF off 182, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D818, prev 2/F171D7D8, desc: INSERT_LEAF off 183, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D858, prev 2/F171D818, desc: INSERT_LEAF off 184, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D898, prev 2/F171D858, desc: INSERT_LEAF off 185, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D8D8, prev 2/F171D898, desc: INSERT_LEAF off 186, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D918, prev 2/F171D8D8, desc: INSERT_LEAF off 187, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D958, prev 2/F171D918, desc: INSERT_LEAF off 188, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D998, prev 2/F171D958, desc: INSERT_LEAF off 189, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171D9D8, prev 2/F171D998, desc: INSERT_LEAF off 190, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DA18, prev 2/F171D9D8, desc: INSERT_LEAF off 191, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DA58, prev 2/F171DA18, desc: INSERT_LEAF off 192, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DA98, prev 2/F171DA58, desc: INSERT_LEAF off 193, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DAD8, prev 2/F171DA98, desc: INSERT_LEAF off 194, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DB18, prev 2/F171DAD8, desc: INSERT_LEAF off 195, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DB58, prev 2/F171DB18, desc: INSERT_LEAF off 196, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DB98, prev 2/F171DB58, desc: INSERT_LEAF off 197, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DBD8, prev 2/F171DB98, desc: INSERT_LEAF off 198, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DC18, prev 2/F171DBD8, desc: INSERT_LEAF off 199, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DC58, prev 2/F171DC18, desc: INSERT_LEAF off 200, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DC98, prev 2/F171DC58, desc: INSERT_LEAF off 201, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DCD8, prev 2/F171DC98, desc: INSERT_LEAF off 202, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DD18, prev 2/F171DCD8, desc: INSERT_LEAF off 203, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DD58, prev 2/F171DD18, desc: INSERT_LEAF off 204, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DD98, prev 2/F171DD58, desc: INSERT_LEAF off 205, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DDD8, prev 2/F171DD98, desc: INSERT_LEAF off 206, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DE18, prev 2/F171DDD8, desc: INSERT_LEAF off 207, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DE58, prev 2/F171DE18, desc: INSERT_LEAF off 208, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DE98, prev 2/F171DE58, desc: INSERT_LEAF off 209, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DED8, prev 2/F171DE98, desc: INSERT_LEAF off 210, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DF18, prev 2/F171DED8, desc: INSERT_LEAF off 211, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DF58, prev 2/F171DF18, desc: INSERT_LEAF off 212, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DF98, prev 2/F171DF58, desc: INSERT_LEAF off 213, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171DFD8, prev 2/F171DF98, desc: INSERT_LEAF off 214, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171E030, prev 2/F171DFD8, desc: INSERT_LEAF off 215, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171E070, prev 2/F171E030, desc: INSERT_LEAF off 216, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171E0B0, prev 2/F171E070, desc: INSERT_LEAF off 217, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171E0F0, prev 2/F171E0B0, desc: INSERT_LEAF off 218, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171E130, prev 2/F171E0F0, desc: INSERT_LEAF off 219, blkref #0: rel 1663/17298/34450 blk 565
rmgr: Btree len (rec/tot): 64/ 64, tx: 19944, lsn: 2/F171E170, prev 2/F171E130, desc: INSERT_LEAF off 220, blkref #0: rel 1663/17298/34450 blk 565
rmgr: MultiXact len (rec/tot): 54/ 54, tx: 19948, lsn: 2/F171E1B0, prev 2/F171E170, desc: CREATE_ID 208 offset 417 nmembers 2: 19936 (forupd) 19948 (nokeyupd)
rmgr: Heap len (rec/tot): 282/ 282, tx: 19948, lsn: 2/F171E1E8, prev 2/F171E1B0, desc: HOT_UPDATE off 23 xmax 208 IS_MULTI EXCL_LOCK KEYS_UPDATED ; new off 25 xmax 19936, blkref #0: rel 1663/17298/35747 blk 0
rmgr: Sequence len (rec/tot): 99/ 99, tx: 0, lsn: 2/F171E308, prev 2/F171E1E8, desc: LOG rel 1663/17298/35767, blkref #0: rel 1663/17298/35767 blk 0
rmgr: Heap len (rec/tot): 95/ 95, tx: 19950, lsn: 2/F171E370, prev 2/F171E308, desc: INSERT off 84, blkref #0: rel 1663/17298/35769 blk 1
rmgr: Btree len (rec/tot): 64/ 64, tx: 19950, lsn: 2/F171E3D0, prev 2/F171E370, desc: INSERT_LEAF off 204, blkref #0: rel 1663/17298/35779 blk 1
rmgr: Btree len (rec/tot): 64/ 64, tx: 19950, lsn: 2/F171E410, prev 2/F171E3D0, desc: INSERT_LEAF off 103, blkref #0: rel 1663/17298/35786 blk 1
rmgr: XLOG len (rec/tot): 210/ 210, tx: 0, lsn: 2/F171E450, prev 2/F171E410, desc: POLAR_WAL of slot test_logical_decode_slot_0 with restart 2/F171A748 and xid 0 confirmed to 2/F171B8E0
rmgr: XLOG len (rec/tot): 210/ 210, tx: 0, lsn: 2/F171E528, prev 2/F171E450, desc: POLAR_WAL of slot test_logical_decode_slot_0 with restart 2/F171B8A8 and xid 0 confirmed to 2/F171B8E0
rmgr: Transaction len (rec/tot): 154/ 154, tx: 19933, lsn: 2/F171E600, prev 2/F171E528, desc: COMMIT 2024-01-30 16:02:38.554115 UTC; subxacts: 19934 19935 19936 19937 19938 19939 19940 19941 19942 19943 19944 19945 19946 19947 19948 19949 19950; inval msgs: relcache 35747 relcache 35747
v1-0001-Testcase-Coredump-On-AssertTXNLsnOrder.patch (application/octet-stream, 2.5K) download
diff --git a/contrib/test_decoding/specs/catalog_change_snapshot.spec b/contrib/test_decoding/specs/catalog_change_snapshot.spec
index d8b9df97ed..d8a453479d 100644
--- a/contrib/test_decoding/specs/catalog_change_snapshot.spec
+++ b/contrib/test_decoding/specs/catalog_change_snapshot.spec
@@ -19,14 +19,17 @@ teardown
session "s0"
setup { SET synchronous_commit=on; }
+step "s0_create_temp" { CREATE TEMP TABLE temp_t (id int, data TEXT); }
step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding'); }
step "s0_begin" { BEGIN; }
+step "s0_txid_current" { SELECT 'xid' FROM txid_current(); }
step "s0_savepoint" { SAVEPOINT sp1; }
step "s0_savepoint_release" { RELEASE SAVEPOINT sp1; }
step "s0_truncate" { TRUNCATE tbl1; }
step "s0_insert" { INSERT INTO tbl1 VALUES (1); }
step "s0_insert2" { INSERT INTO user_cat VALUES (1); }
step "s0_insert_part" { INSERT INTO tbl1_part VALUES (1); }
+step "s0_insert_temp" { INSERT INTO temp_t VALUES(1); }
step "s0_create_part1" { CREATE TABLE tbl1_part_p1 PARTITION OF tbl1_part FOR VALUES FROM (0) TO (10); }
step "s0_create_part2" { CREATE TABLE tbl1_part_p2 PARTITION OF tbl1_part FOR VALUES FROM (10) TO (20); }
step "s0_commit" { COMMIT; }
@@ -75,3 +78,11 @@ permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert" "s1_checkpoint" "s1_
# that the corresponding substransaction is also marked as containing a catalog
# modifying change.
permutation "s0_init" "s0_begin" "s0_savepoint" "s0_create_part1" "s0_savepoint_release" "s1_checkpoint" "s0_create_part2" "s0_commit" "s0_begin" "s0_truncate" "s1_checkpoint" "s1_get_changes" "s0_insert_part" "s1_get_changes" "s0_commit" "s1_get_changes"
+
+# DML operations on temporary table temp_t consume XIDs, but do not generate WAL records.
+# The last decoding restarts from the first checkpoint and doesn't get any information of
+# last two substransactions that performed s0_insert_temp. While processing the commit record
+# for the corresponding top-level transaction which will be marked as containing catalog
+# change even before commit, we ensure that the corresponding substransaction is added into
+# ReorderBuffer as subxact.
+permutation "s0_create_temp" "s0_init" "s0_begin" "s0_txid_current" "s1_checkpoint" "s0_savepoint" "s0_insert_temp" "s0_savepoint_release" "s0_savepoint" "s0_insert_temp" "s0_savepoint_release" "s0_savepoint" "s0_insert_temp" "s0_savepoint_release" "s0_create_part1" "s1_get_changes" "s0_commit" "s1_get_changes"
v1-0002-Fix-Coredump-On-AssertTXNLsnOrder.patch (application/octet-stream, 4.3K) download
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 1c0058fc8f..29871a4117 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -336,7 +336,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
invals->nmsgs,
invals->msgs);
ReorderBufferXidSetCatalogChanges(ctx->reorder, xid,
- buf->origptr);
+ buf->origptr, true);
}
else if ((!ctx->fast_forward))
ReorderBufferImmediateInvalidation(ctx->reorder,
@@ -563,7 +563,7 @@ DecodeHeapOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
SnapBuildProcessChange(builder, xid, buf->origptr);
- ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr, true);
break;
case XLOG_HEAP_CONFIRM:
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 264c253a87..4726f6a507 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1060,7 +1060,8 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
* list of top-level txns. Now that we know it's not top-level,
* remove it from there.
*/
- dlist_delete(&subtxn->node);
+ if (subtxn->node.next != NULL)
+ dlist_delete(&subtxn->node);
}
}
@@ -1550,7 +1551,8 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
*/
- dlist_delete(&txn->node);
+ if (txn->node.next != NULL)
+ dlist_delete(&txn->node);
/* now remove reference from buffer */
hash_search(rb->by_txn,
@@ -3297,11 +3299,11 @@ ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
*/
void
ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn, bool is_top)
{
ReorderBufferTXN *txn;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, is_top);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 4f8c00f717..aec87c11ae 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -818,7 +818,7 @@ SnapBuildProcessNewCid(SnapBuild *builder, TransactionId xid,
* we only log new_cid's if a catalog tuple was modified, so mark the
* transaction as containing catalog modifications
*/
- ReorderBufferXidSetCatalogChanges(builder->reorder, xid, lsn);
+ ReorderBufferXidSetCatalogChanges(builder->reorder, xid, lsn, true);
ReorderBufferAddNewTupleCids(builder->reorder, xlrec->top_xid, lsn,
xlrec->target_node, xlrec->target_tid,
@@ -2131,9 +2131,9 @@ SnapBuildXidSetCatalogChanges(SnapBuild *builder, TransactionId xid, int subxcnt
if (bsearch(&xid, InitialRunningXacts, NInitialRunningXacts,
sizeof(TransactionId), xidComparator) != NULL)
{
- ReorderBufferXidSetCatalogChanges(builder->reorder, xid, lsn);
+ ReorderBufferXidSetCatalogChanges(builder->reorder, xid, lsn, true);
for (int i = 0; i < subxcnt; i++)
- ReorderBufferXidSetCatalogChanges(builder->reorder, subxacts[i], lsn);
+ ReorderBufferXidSetCatalogChanges(builder->reorder, subxacts[i], lsn, false);
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index ba257d81b5..55b9bede87 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -661,7 +661,7 @@ void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
-void ReorderBufferXidSetCatalogChanges(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
+void ReorderBufferXidSetCatalogChanges(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn, bool is_top);
bool ReorderBufferXidHasCatalogChanges(ReorderBuffer *, TransactionId xid);
bool ReorderBufferXidHasBaseSnapshot(ReorderBuffer *, TransactionId xid);
view thread (32+ messages) latest in thread
Message-ID: <6d0e80d6.c1fc.18deeb8120a.Coremail.ocean_li_996@163.com>
Permalink: ../6d0e80d6.c1fc.18deeb8120a.Coremail.ocean_li_996@163.com/
Also on: postgresql.org/message-id/6d0e80d6.c1fc.18deeb8120a.Coremail.ocean_li_996@163.com
copy link · copy postgr.es · copy git am
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: ocean_li_996@163.com, pgsql-bugs@lists.postgresql.org
Subject: Re:BUG #18369: logical decoding core on AssertTXNLsnOrder()
In-Reply-To: <6d0e80d6.c1fc.18deeb8120a.Coremail.ocean_li_996@163.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox