Content-Type: text/plain;
	charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\))
Subject: Re: how to make duplicate finding query faster?
From: Scott Ribe <scott_ribe@elevated-dev.com>
In-Reply-To: 
 <CALg-PKBPQPBMbNAxf-pF0OLbYs0exC7Vf2Sn3hNPUAJ7Q=mPtA@mail.gmail.com>
Date: Wed, 30 Dec 2020 06:13:07 -0700
Cc: pgsql-admin@postgresql.org,
 krishna@thewebconz.com,
 pgsql-admin@lists.postgresql.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <FEEE2DC4-B506-4B34-80FE-07FCC0ADC61E@elevated-dev.com>
References: 
 <CALg-PKBPQPBMbNAxf-pF0OLbYs0exC7Vf2Sn3hNPUAJ7Q=mPtA@mail.gmail.com>
To: Sachin Kumar <sachinkumaras@gmail.com>
Precedence: bulk

> On Dec 30, 2020, at 12:36 AM, Sachin Kumar <sachinkumaras@gmail.com> =
wrote:
>=20
> Hi All,
>=20
> I am uploading data into PostgreSQL using the CSV file and checking if =
there is any duplicates value in DB it should return a duplicate error.  =
I am using below mention query.
>=20
> if Card_Bank.objects.filter( Q(ACCOUNT_NUMBER=3Dcard_number) =
).exists():=20
>         flag=3D2
>       else:
>         flag=3D1
> it is taking too much time i am using 600k cards in CSV.
>=20
> Kindly help me in making the query faster.
>=20
> I am using Python, Django & PostgreSQL.
> --=20
>=20
> Best Regards,=20
> Sachin Kumar

Are you checking one-by-one because your goal is not to fail the whole =
upload that contains the duplicates, but rather to skip only the =
duplicates?

If that's the case, I think you'd be better off copying the CSV straight =
into a temp table, using a join to delete duplicates from it, then =
insert the remainder into the target table, and finally drop the temp =
table.=