From: Holger Jakobs <holger@jakobs.com>
To: pgsql-admin@lists.postgresql.org
Subject: Re: how to make duplicate finding query faster?
Date: Wed, 30 Dec 2020 09:23:42 +0100
Message-ID: <5bf642e4-eb22-8eba-e96d-5b33b5010b03@jakobs.com> (raw)
In-Reply-To: <CALg-PKBPQPBMbNAxf-pF0OLbYs0exC7Vf2Sn3hNPUAJ7Q=mPtA@mail.gmail.com>
References: <CALg-PKBPQPBMbNAxf-pF0OLbYs0exC7Vf2Sn3hNPUAJ7Q=mPtA@mail.gmail.com>
Am 30.12.20 um 08:36 schrieb Sachin Kumar:
> Hi All,
>
> I am uploading data into PostgreSQL using the CSV file and checking if
> there is any duplicates value in DB it should return a duplicate
> error. I am using below mention query.
>
> if Card_Bank.objects.filter( Q(ACCOUNT_NUMBER=card_number) ).exists():
> flag=2
> else:
> flag=1
> it is taking too much time i am using 600k cards in CSV.
>
> Kindly help me in making the query faster.
>
> I am using Python, Django & PostgreSQL.
> --
>
> Best Regards,
> Sachin Kumar
I think it would be easier to not check the duplicates before, but let
the DB complain about duplicates.
That would about slash the roundtrips to the DB in half. Instead of
check + insert there would be only an insert, which might fail every now
and then.
Regards,
Holger
--
Holger Jakobs, Bergisch Gladbach, Tel. +49-178-9759012
Attachments:
OpenPGP_signature (application/pgp-signature, 202B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: holger@jakobs.com, pgsql-admin@lists.postgresql.org
Subject: Re: how to make duplicate finding query faster?
In-Reply-To: <5bf642e4-eb22-8eba-e96d-5b33b5010b03@jakobs.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox