Subject: [PATCH] pgarchives: Bugfix: allow message's parentid to be null

From: Célestin Matte <[email-masked]>
Date: Mon, 25 Oct 2021 16:52:55 +0200

Hello,

Upon execution of load_messages.py, when creating a new thread, I get a crash:
(Sorry about the format, I get this out of strace)

Traceback (most recent call last):\n  File \"/srv/pgarchives/local/loader/load_message.py\", lin\
e 174, in <module>\n    ap.store(conn, listid, opt.overwrite, opt.overwrite)\n  File \"/srv/pgarchives/local/loader/li\
b/storage.py\", line 232, in store\n    'rawtxt': bytearray(self.rawtxt),\npsycopg2.errors.NotNullViolation: null valu\
e in column \"parentid\" violates not-null constraint\nDETAIL:  Failing row contains (2, 2, cmatte <cmatte@[my_domain]\
.tld>, helloworld@[my_pglister_domain], , Test archive after redeployment 2, 2021-10-25 14:13:32+00, 1635171212.3\
49997.1068459.nullmailer@[my_domain], hello world\n\n\n\n, null, f, null, \\x46726f6d2068656c6c6f776f726c642d6f776\
e65722b617263686976654062..., '2':5A 'after':3A 'archive':2A 'hello':6 'redeployment':4A 'test...).\n\n

I think the way to fix this is simply to allow message.parentid to be null.

Cheers,
-- 
Célestin Matte

From ba3852df7a69a1a25dd366fd3657fce3740cf23f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?C=C3=A9lestin=20Matte?= <[email-masked]>
Date: Mon, 25 Oct 2021 16:42:49 +0200
Subject: [PATCH] Bugfix: allow message's parentid to be null

Otherwise, load_message.py will crash upon creation of a new thread
---
 django/archives/mailarchives/models.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/django/archives/mailarchives/models.py b/django/archives/mailarchives/models.py
index 44c4469..879d280 100644
--- a/django/archives/mailarchives/models.py
+++ b/django/archives/mailarchives/models.py
@@ -25,7 +25,7 @@ class Message(models.Model):
     messageid = models.TextField(null=False)
     bodytxt = models.TextField(null=False)
     # rawtxt is a bytea field, which django doesn't support (easily)
-    parentid = models.IntegerField(null=False, blank=False)
+    parentid = models.IntegerField(null=True, blank=False)
     has_attachment = models.BooleanField(null=False, default=False)
     hiddenstatus = models.IntegerField(null=True)
     # fti is a tsvector field, which django doesn't support (easily)
-- 
2.33.1

---

From: Célestin Matte <[email-masked]>
Date: Tue, 26 Oct 2021 20:31:46 +0200

Attached: associated migration

-- 
Célestin Matte

From 9049a67567c568ee3a35217ee620d40cb567d8ae Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?C=C3=A9lestin=20Matte?= <[email-masked]>
Date: Tue, 26 Oct 2021 20:29:58 +0200
Subject: [PATCH 2/2] Add migration associated with ba3852df (allow parentid to
 be null)

---
 .../migrations/0005_auto_20211026_1827.py      | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
 create mode 100644 django/archives/mailarchives/migrations/0005_auto_20211026_1827.py

diff --git a/django/archives/mailarchives/migrations/0005_auto_20211026_1827.py b/django/archives/mailarchives/migrations/0005_auto_20211026_1827.py
new file mode 100644
index 0000000..95e81ed
--- /dev/null
+++ b/django/archives/mailarchives/migrations/0005_auto_20211026_1827.py
@@ -0,0 +1,18 @@
+# Generated by Django 2.2.24 on 2021-10-26 18:27
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('mailarchives', '0004_resend_rate_limit'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='message',
+            name='parentid',
+            field=models.IntegerField(null=True),
+        ),
+    ]
-- 
2.33.1

---

From: Magnus Hagander <[email-masked]>
Date: Wed, 27 Oct 2021 11:19:46 +0200

On Tue, Oct 26, 2021 at 8:31 PM Célestin Matte <[email-masked]>
wrote:

> > Traceback (most recent call last):\n  File
> \"/srv/pgarchives/local/loader/load_message.py\", lin\
> > e 174, in <module>\n    ap.store(conn, listid, opt.overwrite,
> opt.overwrite)\n  File \"/srv/pgarchives/local/loader/li\
> > b/storage.py\", line 232, in store\n    'rawtxt':
> bytearray(self.rawtxt),\npsycopg2.errors.NotNullViolation: null valu\
> > e in column \"parentid\" violates not-null constraint\nDETAIL:  Failing
> row contains (2, 2, cmatte <cmatte@[my_domain]\
> > .tld>, helloworld@[my_pglister_domain], , Test archive after
> redeployment 2, 2021-10-25 14:13:32+00, 1635171212.3\
> > 49997.1068459.nullmailer@[my_domain], hello world\n\n\n\n, null, f,
> null, \\x46726f6d2068656c6c6f776f726c642d6f776\
Yeah, this looks to be another disreptency between the Django model and the
schema.sql file. The table when created from schema.sql will allow nulls.
So I can confirm that the field *should* allow nulls, and the only reason
it hasn't been an actual problem is that we never ever do any writes to the
system through the django model.

That said, I think the correct solutoin is still, as I mentioned in one of
the other threads, to completely move everything into the initial
migration, instead of doing a piecemeal hack that's going to result in a
lot of small migrations for no real reason. We should  just accept that the
existing one is broken and replace it.

If you're willing to look into working on that I'd be happy to review on
that one, but if it seems a bit too much I can work on that one myself as
well, but it might take a bit before I have time to dig into it properly.

//Magnus

---

From: Célestin Matte <[email-masked]>
Date: Wed, 27 Oct 2021 12:04:02 +0200

> 
> That said, I think the correct solutoin is still, as I mentioned in one of the other threads, to completely move everything into the initial migration, instead of doing a piecemeal hack that's going to result in a lot of small migrations for no real reason. We should  just accept that the existing one is broken and replace it.

Can everything be moved to django's model though? The schema.sql file defines several other components (indexes, text search configurations, text search dictionaries, functions, triggers) and I'm not sure django is able to handle all of that by itself. Is it? Or do you only want to move tables?

Cheers,
-- 
Célestin Matte

---

From: Magnus Hagander <[email-masked]>
Date: Wed, 27 Oct 2021 15:50:08 +0200

On Wed, Oct 27, 2021 at 12:04 PM Célestin Matte <[email-masked]>
wrote:

> >
> > That said, I think the correct solutoin is still, as I mentioned in one
> of the other threads, to completely move everything into the initial
> Can everything be moved to django's model though? The schema.sql file
> defines several other components (indexes, text search configurations, text
> search dictionaries, functions, triggers) and I'm not sure django is able
> to handle all of that by itself. Is it? Or do you only want to move tables?
>

The models can't handle them, but migrations can.

I'm thinking similar to pglister, where we use django models to create the
parts that *can* be created, and then RunSQL() migrations to do the rest.
That way the pure SQL parts can still run as part of the same flow, and can
be mixed in with the django generated ones. The end result in the database
should be the same, but avoiding to have to maintain a separate .sql file
for it to keep in sync.

I suggest moving everything, for that reason -- to ensure the order of
application.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ <http://www.hagander.net/>
 Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>