From: Scott Ribe <scott_ribe@elevated-dev.com>
To: Sachin Kumar <sachinkumaras@gmail.com>
Cc: pgsql-admin@postgresql.org
Cc: krishna@thewebconz.com
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: how to make duplicate finding query faster?
Date: Wed, 30 Dec 2020 06:13:07 -0700
Message-ID: <FEEE2DC4-B506-4B34-80FE-07FCC0ADC61E@elevated-dev.com> (raw)
In-Reply-To: <CALg-PKBPQPBMbNAxf-pF0OLbYs0exC7Vf2Sn3hNPUAJ7Q=mPtA@mail.gmail.com>
References: <CALg-PKBPQPBMbNAxf-pF0OLbYs0exC7Vf2Sn3hNPUAJ7Q=mPtA@mail.gmail.com>
> On Dec 30, 2020, at 12:36 AM, Sachin Kumar <sachinkumaras@gmail.com> wrote:
>
> Hi All,
>
> I am uploading data into PostgreSQL using the CSV file and checking if there is any duplicates value in DB it should return a duplicate error. I am using below mention query.
>
> if Card_Bank.objects.filter( Q(ACCOUNT_NUMBER=card_number) ).exists():
> flag=2
> else:
> flag=1
> it is taking too much time i am using 600k cards in CSV.
>
> Kindly help me in making the query faster.
>
> I am using Python, Django & PostgreSQL.
> --
>
> Best Regards,
> Sachin Kumar
Are you checking one-by-one because your goal is not to fail the whole upload that contains the duplicates, but rather to skip only the duplicates?
If that's the case, I think you'd be better off copying the CSV straight into a temp table, using a join to delete duplicates from it, then insert the remainder into the target table, and finally drop the temp table.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: scott_ribe@elevated-dev.com, sachinkumaras@gmail.com, krishna@thewebconz.com, pgsql-admin@lists.postgresql.org
Subject: Re: how to make duplicate finding query faster?
In-Reply-To: <FEEE2DC4-B506-4B34-80FE-07FCC0ADC61E@elevated-dev.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox