From: Adrian Klaver <adrian.klaver@aklaver.com>
To: arun chirappurath <arunsnmimt@gmail.com>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Sample data generator for performance testing
Date: Wed, 3 Jan 2024 10:31:20 -0800
Message-ID: <49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com> (raw)
In-Reply-To: <CAA23SdsPFiVDfBZWO-oKgq5SzZEAs9fNydgdbw=2BELMmXfxAA@mail.gmail.com>
References: <CAA23Sdv+0PnmpkK1CtHMzVBuHmw09m-aBefEDL2PTRv5rFsB+w@mail.gmail.com>
<42a39fc7-e5e1-4ee7-ab9e-a2da4bf5209f@aklaver.com>
<CAA23SdvtLY1exoY6dj03YAcL2PRXxSpnVPhzjpZKkMqBB13QNw@mail.gmail.com>
<25f33c90-38e6-4f86-940f-05ed6baf27ab@aklaver.com>
<CAA23SdsPFiVDfBZWO-oKgq5SzZEAs9fNydgdbw=2BELMmXfxAA@mail.gmail.com>
On 1/3/24 9:50 AM, arun chirappurath wrote:
>
>
> On Wed, 3 Jan, 2024, 23:03 Adrian Klaver, <adrian.klaver@aklaver.com>
> wrote:
>
> On 1/3/24 09:24, arun chirappurath wrote:
> > Hi Adrian,
> >
> > Thanks for your mail.
> >
> > Is this for all tables in the database or a subset? Yes
>
> Yes all tables or yes just some tables?
> All tables.except some which has user details.
>
>
> >
> > Does it need to deal with foreign key relationships? No
> >
> > What are the sizes of the existing data and what size sample
> data do you
> > want to produce?1Gb and 1Gb test data.
>
> If the source data is 1GB and the test data is 1GB then there is no
> sampling, you are using the data population in its entirety.
>
> Yes.would like to double the load and test.
>
Does that mean you want to take the 1GB of your existing data and double
it to 2GB while maintaining
the data distribution from the original data?
>
> Also do we have any standard methods for sampling and generating test data
Something like?:
https://www.postgresql.org/docs/current/sql-select.html
"|TABLESAMPLE /|sampling_method|/ ( /|argument|/ [, ...] ) [ REPEATABLE
( /|seed|/ ) ]|
A |TABLESAMPLE| clause after a /|table_name|/ indicates that the
specified /|sampling_method|/ should be used to retrieve a subset of
the rows in that table. This sampling precedes the application of
any other filters such as |WHERE| clauses. The standard PostgreSQL
distribution includes two sampling methods, |BERNOULLI| and
|SYSTEM|, and other sampling methods can be installed in the
database via extensions
...
"
Read the rest of the documentation for TABLESAMPLE to get the details.
>
>
> >
> > On Wed, 3 Jan, 2024, 22:40 Adrian Klaver,
> <adrian.klaver@aklaver.com
> > <mailto:adrian.klaver@aklaver.com>> wrote:
> >
> > On 1/2/24 23:23, arun chirappurath wrote:
> > > Hi All,
> > >
> > > Do we have any open source tools which can be used to create
> > sample data
> > > at scale from our postgres databases?
> > > Which considers data distribution and randomness
> >
> >
> >
> > >
> > > Regards,
> > > Arun
> >
> > --
> > Adrian Klaver
> > adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>
> >
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: adrian.klaver@aklaver.com, arunsnmimt@gmail.com
Subject: Re: Sample data generator for performance testing
In-Reply-To: <49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com>
References: <CAA23Sdv+0PnmpkK1CtHMzVBuHmw09m-aBefEDL2PTRv5rFsB+w@mail.gmail.com> <42a39fc7-e5e1-4ee7-ab9e-a2da4bf5209f@aklaver.com> <CAA23SdvtLY1exoY6dj03YAcL2PRXxSpnVPhzjpZKkMqBB13QNw@mail.gmail.com> <25f33c90-38e6-4f86-940f-05ed6baf27ab@aklaver.com> <CAA23SdsPFiVDfBZWO-oKgq5SzZEAs9fNydgdbw=2BELMmXfxAA@mail.gmail.com> <49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox