Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tPnDz-00FLOZ-3B for pgsql-general@arkaria.postgresql.org; Mon, 23 Dec 2024 18:32:15 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tPnDy-00ERPt-Dr for pgsql-general@arkaria.postgresql.org; Mon, 23 Dec 2024 18:32:14 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tPnDy-00ERPl-0d for pgsql-general@lists.postgresql.org; Mon, 23 Dec 2024 18:32:13 +0000 Received: from mail-oo1-xc35.google.com ([2607:f8b0:4864:20::c35]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1tPnDu-001GMi-Ox for pgsql-general@postgresql.org; Mon, 23 Dec 2024 18:32:13 +0000 Received: by mail-oo1-xc35.google.com with SMTP id 006d021491bc7-5f2e31139d9so2006012eaf.0 for ; Mon, 23 Dec 2024 10:32:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734978729; x=1735583529; darn=postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=bGauBUAmEn4GpnzGTgSxDgLkMVRXeg2xBLSC7CPvav8=; b=dkKkVJUUl2qiNVGHBlyws2bRradLaBpYXrTT9LHsDizDgxvIHD4OHpt9U4Itp7EXJe tJHq7NdAGZNZi9BNJ07gVSn7/kLdNC1IZ87H2W0pDiXwwA4dFJexFlHh3f9BIrxjkcqx N0PMWV8vM/07h50GMZ5bqg3xnrVNI2bqk1NmXlA6EWHxnIGZQm9CFG5Ls8l4hPaHPdHZ IJprgSYXoZCb1pO/zpsS73MzXRIkpufhV4jGZWrH4GkYdN7tKXSg/kZby07I7wWz1hqp 9XqSV2+41whmK+Ybn490NF3lm/DHxObZirbowSwJRPekeAx0fUfjrXW0MOrT4yqQQHZ4 NdIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734978729; x=1735583529; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bGauBUAmEn4GpnzGTgSxDgLkMVRXeg2xBLSC7CPvav8=; b=gD2dulZox5iLoyb5WNWVxyqGqzr7WrHTv5WDQHvnHveU5hYnJH1fvL9atBYWZpOeND ngqw18jzirFwTU3eXJ+m1Nb9cifd0uil3rev/uUzCCRrCM6OwljrYfMMxrrnwwn+ePcw POctjUgiNf1nerC6AErOyErNb6OtKYZV+C2SnGXuLpSHFDGmXa0FBS/5n/M4MIiE06/G US33flDTDN/xYVXPqAOxzY94TkSeMSdMCyMYOta15XIppaWCkFc+FgOFDb/UvC1COWs5 u6Xbnmrr0BytX2qCwWYLC8X2ZDJx+PPXzJ8hgLbU/AC4oEhv3vK/ynIktWja9QGVore4 o/gw== X-Gm-Message-State: AOJu0Yxi90hLw+OMwUO0Qp1ZkR0sZ7GVsoPsLd14/JHmmZH/xzQX41dL MTiYrXQbrAJGFLo4Y7oB/Nkwo73m8bou9UDdLeYcwHfGsdg8Xl4U8ytSfrkUldYT8CATVswnhzT o9R9zmFZzZVjbPHO1cTeHYNMs+1RmWP4I16+51A== X-Gm-Gg: ASbGncufXmWzpyw/UJq7vU++WNy6ah8WTllerm/mLREMUBexs3kBiyrL8Bzbl3SpLHY OgiLfYjSwF0uxhgDGfrXXNiNRlMcUjYf+LXxu8onmyWj5toCaRzn98hGBAgfaiAr9nyZfM7bX X-Google-Smtp-Source: AGHT+IHE9zBHup/YRW0nGeiAMODG7U6QVqghdJ1tHhxDXFNielQpZdaMb0hXE/cfY2x0yWMpsHRzhOqVMhkh7piLC6s= X-Received: by 2002:a05:6820:4b95:b0:5f2:d799:ab91 with SMTP id 006d021491bc7-5f62e6f3a4dmr7505607eaf.2.1734978729393; Mon, 23 Dec 2024 10:32:09 -0800 (PST) MIME-Version: 1.0 References: <06e1f1ee-74b2-43a2-9a63-da20ae455ae2@aklaver.com> In-Reply-To: From: Divyansh Gupta JNsThMAudy Date: Tue, 24 Dec 2024 00:01:57 +0530 Message-ID: Subject: Re: Need help in database design To: Ron Johnson Cc: pgsql-general Content-Type: multipart/alternative; boundary="000000000000dbc4cc0629f436fd" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000dbc4cc0629f436fd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Currently I haven't created those columns , I have created addons_json column which is a JSONB column yet in a discussion weather I should create or consider only one JSONB column. On Tue, 24 Dec 2024, 12:00=E2=80=AFam Divyansh Gupta JNsThMAudy, < ag1567827@gmail.com> wrote: > Range partition can help when you applies filter for a specific range but > in my case I need to apply filter on userid always, however I have date > columns but there is less variation in timestamp which I have that's why > didn't go for range partition. > > On Mon, 23 Dec 2024, 11:57=E2=80=AFpm Ron Johnson, > wrote: > >> >> 1. I bet you'd get better performance using RANGE partitioning. >> 2. Twenty million rows per userid is a *LOT*. No subdivisions (like >> date range)? >> >> On Mon, Dec 23, 2024 at 1:23=E2=80=AFPM Divyansh Gupta JNsThMAudy < >> ag1567827@gmail.com> wrote: >> >>> Adrian, Please check this out; >>> >>> PARTITION BY HASH (userid); CREATE TABLE dbo.googledocs_tbl_clone_part_= 0 >>> PARTITION OF dbo.googledocs_tbl_clone FOR VALUES WITH (modulus 84, >>> remainder 0); ... CREATE TABLE dbo.googledocs_tbl_clone_part_83 >>> PARTITION OF dbo.googledocs_tbl_clone FOR VALUES WITH (modulus 84, >>> remainder 83); >>> >>> On Mon, Dec 23, 2024 at 11:48=E2=80=AFPM Divyansh Gupta JNsThMAudy < >>> ag1567827@gmail.com> wrote: >>> >>>> Adrian, the partition is on userid using hash partition with 84 >>>> partitions >>>> >>>> Ron, there could be more than 20 Million records possible for a single >>>> userid in that case if I create index on userid only not on other colu= mn >>>> the query is taking more than 30 seconds to return the results. >>>> >>>> On Mon, 23 Dec 2024, 11:40=E2=80=AFpm Ron Johnson, >>>> wrote: >>>> >>>>> If your queries all reference userid, then you only need indices on >>>>> gdid and userid. >>>>> >>>>> On Mon, Dec 23, 2024 at 12:49=E2=80=AFPM Divyansh Gupta JNsThMAudy < >>>>> ag1567827@gmail.com> wrote: >>>>> >>>>>> I have one confusion with this design if I opt to create 50 columns = I >>>>>> need to create 50 index which will work with userid index in Bitmap = on the >>>>>> other hand if I create a JSONB column I need to create a single inde= x ? >>>>>> >>>>>> On Mon, 23 Dec 2024, 11:10=E2=80=AFpm Ron Johnson, >>>>>> wrote: >>>>>> >>>>>>> Given what you just wrote, I'd stick with 50 separate t* columns. >>>>>>> Simplifies queries, simplifies updates, and eliminates JSONB conver= sions. >>>>>>> >>>>>>> On Mon, Dec 23, 2024 at 12:29=E2=80=AFPM Divyansh Gupta JNsThMAudy = < >>>>>>> ag1567827@gmail.com> wrote: >>>>>>> >>>>>>>> Values can be updated based on customer actions >>>>>>>> >>>>>>>> All rows won't have all 50 key value pairs always if I make those >>>>>>>> keys into columns the rows might have null value on the other hand= if it is >>>>>>>> JSONB then the key value pair will not be there >>>>>>>> >>>>>>>> Yes in UI customers can search for the key value pairs >>>>>>>> >>>>>>>> During data population the key value pair will be empty array in >>>>>>>> case of JSONB column or NULL in case of table columns, later when = customer >>>>>>>> performs some actions that time the key value pairs will populate = and >>>>>>>> update, based on what action customer performs. >>>>>>>> >>>>>>>> On Mon, 23 Dec 2024, 10:51=E2=80=AFpm Divyansh Gupta JNsThMAudy, < >>>>>>>> ag1567827@gmail.com> wrote: >>>>>>>> >>>>>>>>> Let's make it more understandable, here is the table schema with >>>>>>>>> 50 columns in it >>>>>>>>> >>>>>>>>> CREATE TABLE dbo.googledocs_tbl ( >>>>>>>>> gdid int8 GENERATED BY DEFAULT AS IDENTITY( INCREMENT BY 1 >>>>>>>>> MINVALUE 1 MAXVALUE 9223372036854775807 START 1 CACHE 1 NO CYCLE)= NOT NULL, >>>>>>>>> userid int8 NOT NULL, >>>>>>>>> t1 int4 NULL, >>>>>>>>> t2 int4 NULL, >>>>>>>>> t3 int4 NULL, >>>>>>>>> t4 int4 NULL, >>>>>>>>> t5 int4 NULL, >>>>>>>>> t6 int4 NULL, >>>>>>>>> t7 int4 NULL, >>>>>>>>> t8 int4 NULL, >>>>>>>>> t9 int4 NULL, >>>>>>>>> t10 int4 NULL, >>>>>>>>> t11 int4 NULL, >>>>>>>>> t12 int4 NULL, >>>>>>>>> t13 int4 NULL, >>>>>>>>> t14 int4 NULL, >>>>>>>>> t15 int4 NULL, >>>>>>>>> t16 int4 NULL, >>>>>>>>> t17 int4 NULL, >>>>>>>>> t18 int4 NULL, >>>>>>>>> t19 int4 NULL, >>>>>>>>> t20 int4 NULL, >>>>>>>>> t21 int4 NULL, >>>>>>>>> t22 int4 NULL, >>>>>>>>> t23 int4 NULL, >>>>>>>>> t24 int4 NULL, >>>>>>>>> t25 int4 NULL, >>>>>>>>> t26 int4 NULL, >>>>>>>>> t27 int4 NULL, >>>>>>>>> t28 int4 NULL, >>>>>>>>> t29 int4 NULL, >>>>>>>>> t30 int4 NULL, >>>>>>>>> t31 int4 NULL, >>>>>>>>> t32 int4 NULL, >>>>>>>>> t33 int4 NULL, >>>>>>>>> t34 int4 NULL, >>>>>>>>> t35 int4 NULL, >>>>>>>>> t36 int4 NULL, >>>>>>>>> t37 int4 NULL, >>>>>>>>> t38 int4 NULL, >>>>>>>>> t39 int4 NULL, >>>>>>>>> t40 int4 NULL, >>>>>>>>> t41 int4 NULL, >>>>>>>>> t42 int4 NULL, >>>>>>>>> t43 int4 NULL, >>>>>>>>> t44 int4 NULL, >>>>>>>>> t45 int4 NULL, >>>>>>>>> t46 int4 NULL, >>>>>>>>> t47 int4 NULL, >>>>>>>>> t48 int4 NULL, >>>>>>>>> t49 int4 NULL, >>>>>>>>> t50 int4 NULL, >>>>>>>>> CONSTRAINT googledocs_tbl_pkey PRIMARY KEY (gdid), >>>>>>>>> ); >>>>>>>>> >>>>>>>>> Every time when i query I will query it along with userid >>>>>>>>> Ex : where userid =3D 12345678 and t1 in (1,2,3) and t2 in (0,1,2= ) >>>>>>>>> more key filters if customer applies >>>>>>>>> >>>>>>>>> On the other hand if I create a single jsonb column the schema >>>>>>>>> will look like : >>>>>>>>> >>>>>>>>> CREATE TABLE dbo.googledocs_tbl ( >>>>>>>>> gdid int8 GENERATED BY DEFAULT AS IDENTITY( INCREMENT BY 1 >>>>>>>>> MINVALUE 1 MAXVALUE 9223372036854775807 START 1 CACHE 1 NO CYCLE)= NOT NULL, >>>>>>>>> userid int8 NOT NULL, >>>>>>>>> addons_json jsonb default '{}'::jsonb >>>>>>>>> CONSTRAINT googledocs_tbl_pkey PRIMARY KEY (gdid), >>>>>>>>> ); >>>>>>>>> >>>>>>>>> and the query would be like >>>>>>>>> where userid =3D 12345678 and ((addons_json @> {t1:1}) or >>>>>>>>> (addons_json @> {t1:2}) or (addons_json @> {t1:3}) >>>>>>>>> more key filters if customer applies >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Dec 23, 2024 at 10:38=E2=80=AFPM David G. Johnston < >>>>>>>>> david.g.johnston@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Dec 23, 2024, 10:01 Divyansh Gupta JNsThMAudy < >>>>>>>>>> ag1567827@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> So here my question is considering one JSONB column is perfect >>>>>>>>>>> or considering 50 columns will be more optimised. >>>>>>>>>>> >>>>>>>>>> The relational database engine is designed around the >>>>>>>>>> column-based approach. Especially if the columns are generally = unchanging, >>>>>>>>>> combined with using fixed-width data types. >>>>>>>>>> >>>>>>>>>> David J. >>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Death to , and butter sauce. >>>>>>> Don't boil me, I'm still alive. >>>>>>> lobster! >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> Death to , and butter sauce. >>>>> Don't boil me, I'm still alive. >>>>> lobster! >>>>> >>>> >> >> -- >> Death to , and butter sauce. >> Don't boil me, I'm still alive. >> lobster! >> > --000000000000dbc4cc0629f436fd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Currently I haven't created those columns , I have creat= ed addons_json column which is a JSONB column yet in a discussion weather I= should create or consider only one JSONB column.


On Tue, 24 Dec 2024, 12:00=E2=80=AFam Divyansh Gupta JNsThM= Audy, <ag1567827@gmail.com>= ; wrote:

Range parti= tion can help when you applies filter for a specific range but in my case I= need to apply filter on userid always, however I have date columns but the= re is less variation in timestamp which I have that's why didn't go= for range partition.


On Mon= , 23 Dec 2024, 11:57=E2=80=AFpm Ron Johnson, <ronljohnsonjr@gmail.c= om> wrote:
=

1. I bet you'd get better performance u= sing RANGE partitioning.
2. Twenty million rows per userid=C2=A0i= s a LOT.=C2=A0 No subdivisions (like date range)?

On Mon, Dec 23, 202= 4 at 1:23=E2=80=AFPM Divyansh Gupta JNsThMAudy <ag1567827@gm= ail.com> wrote:
Adrian, Please check this out;

PARTITION BY HASH (userid); = CREATE TABLE dbo.googledocs_tbl_clone_part_0 PARTITION OF dbo.googledocs_tbl_clone FOR VALUES WITH (modulus 84, remainder 0); ... = CREATE TABLE dbo.googledocs_tbl_clone_part_83 PARTITION OF dbo.googledocs_tbl_clone FOR VALUES WITH (modulus 84, remainder 83);

On Mon, Dec 23, 2024 at 11:48=E2=80=AFPM D= ivyansh Gupta JNsThMAudy <ag1567827@gmail.com> wrote:=

= Adrian, the partition is on userid using hash partition with 84 partitions<= /p>

Ron, there could be more than 20 Million records possible fo= r a single userid in that case if I create index on userid only not on othe= r column the query is taking more than 30 seconds to return the results.

On Mon= , 23 Dec 2024, 11:40=E2=80=AFpm Ron Johnson, <ronljohnso= njr@gmail.com> wrote:
If your queries all reference userid, th= en you only need indices on gdid and userid.

On Mon, Dec 23, 2024 at 12:49= =E2=80=AFPM Divyansh Gupta JNsThMAudy <ag1567827@= gmail.com> wrote:

I have one confusion with this design if I opt to= create 50 columns I need to create 50 index which will work with userid in= dex in Bitmap on the other hand if I create a JSONB column I need to create= a single index ?


On Mon= , 23 Dec 2024, 11:10=E2=80=AFpm Ron Johnson, <ronljohnsonjr@gmail.com> wrote:
Given what you just wrote, I&= #39;d stick with 50 separate t* columns.=C2=A0 Simplifies queries, simplifi= es updates, and eliminates JSONB conversions.

On Mon, Dec 23, 2024 at 12:29= =E2=80=AFPM Divyansh Gupta JNsThMAudy <ag1567827@gmail.com> wrote:

Values can be updated based on customer = actions

All rows won't have all 50 key value pairs always if I m= ake those keys into columns the rows might have null value on the other han= d if it is JSONB then the key value pair will not be there

Yes in UI customers can search for the key value pairs

During data population the key value pair will be empty arra= y in case of JSONB column or NULL in case of table columns, later when cust= omer performs some actions that time the key value pairs will populate and = update, based on what action customer performs.


On Mon= , 23 Dec 2024, 10:51=E2=80=AFpm Divyansh Gupta JNsThMAudy, <ag1567827@gmail.com> wrote:
Let's make it= more understandable, here is the table schema with 50 columns in it=C2=A0<= br>
CREATE TABLE dbo.googledocs_tbl (
gdid int8 GENERATED BY DEFAULT= AS IDENTITY( INCREMENT BY 1 MINVALUE 1 MAXVALUE 9223372036854775807 START = 1 CACHE 1 NO CYCLE) NOT NULL,
userid int8 NOT NULL,
t1 int4 NULL, t2 int4 NULL,
t3 int4 NULL,
t4 int4 NULL,
t5 int4 NULL,
= t6 int4 NULL,
t7 int4 NULL,
t8 int4 NULL,
t9 int4 NULL,
t10= int4 NULL,
t11 int4 NULL,
t12 int4 NULL,
t13 int4 NULL,
t1= 4 int4 NULL,
t15 int4 NULL,
t16 int4 NULL,
t17 int4 NULL,
t= 18 int4 NULL,
t19 int4 NULL,
t20 int4 NULL,
t21 int4 NULL,
= t22 int4 NULL,
t23 int4 NULL,
t24 int4 NULL,
t25 int4 NULL,
= t26 int4 NULL,
t27 int4 NULL,
t28 int4 NULL,
t29 int4 NULL, t30 int4 NULL,
t31 int4 NULL,
t32 int4 NULL,
t33 int4 NULL, t34 int4 NULL,
t35 int4 NULL,
t36 int4 NULL,
t37 int4 NULL,<= br> t38 int4 NULL,
t39 int4 NULL,
t40 int4 NULL,
t41 int4 NULL,=
t42 int4 NULL,
t43 int4 NULL,
t44 int4 NULL,
t45 int4 NULL= ,
t46 int4 NULL,
t47 int4 NULL,
t48 int4 NULL,
t49 int4 NUL= L,
t50 int4 NULL,
CONSTRAINT googledocs_tbl_pkey PRIMARY KEY (gdid)= ,
);

Every time when i query I will query it along with userid=C2= =A0
Ex : where userid =3D 12345678 and t1 in (1,2,3) and t2 in (0,1,2)more key filters if customer applies=C2=A0

On the other hand if I = create a single jsonb column the schema will look like :

CREATE TABL= E dbo.googledocs_tbl (
gdid int8 GENERATED BY DEFAULT AS IDENTITY( INCR= EMENT BY 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 CACHE 1 NO CYCLE= ) NOT NULL,
userid int8 NOT NULL,
addons_json jsonb default '{}= '::jsonb
CONSTRAINT googledocs_tbl_pkey PRIMARY KEY (gdid),
);
and the query would be like=C2=A0
where userid =3D 12345678 and ((= addons_json=C2=A0@> {t1:1}) or=C2=A0 (addons_json=C2=A0@> {t1:2}) or=C2=A0 (addons_json=C2=A0@> {t1:3})
more key filters if customer a= pplies=C2=A0



On Mon, Dec 23, 2024 at 10:38=E2=80=AFPM Davi= d G. Johnston <david= .g.johnston@gmail.com> wrote:


On Mon, Dec 23, 2024, 10:= 01 Divyansh Gupta JNsThMAudy <ag1567827@gmail.com> wrote:


S= o here my question is considering one JSONB column is perfect or considerin= g 50 columns will be more optimised.

The relational database engine is designed around the column-base= d approach.=C2=A0 Especially if the columns are generally unchanging, combi= ned with using fixed-width data types.

David J.



--
Death to <Redacted>, and butter sauce.Don't boil me, I'm still alive.
<Redacted> lobs= ter!


--
Death to <Redacted>, and butter sauce.Don't boil me, I'm still alive.
<Redacted> lobs= ter!


--
Death to <Redacted>, and butter sauce.Don't boil me, I'm still alive.
<Redacted> lobs= ter!
--000000000000dbc4cc0629f436fd--