pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Adrian Klaver <adrian.klaver@aklaver.com>
To: arun chirappurath <arunsnmimt@gmail.com>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Sample data generator for performance testing
Date: Wed, 3 Jan 2024 10:31:20 -0800
Message-ID: <49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com> (raw)
In-Reply-To: <CAA23SdsPFiVDfBZWO-oKgq5SzZEAs9fNydgdbw=2BELMmXfxAA@mail.gmail.com>
References: <CAA23Sdv+0PnmpkK1CtHMzVBuHmw09m-aBefEDL2PTRv5rFsB+w@mail.gmail.com>
	<42a39fc7-e5e1-4ee7-ab9e-a2da4bf5209f@aklaver.com>
	<CAA23SdvtLY1exoY6dj03YAcL2PRXxSpnVPhzjpZKkMqBB13QNw@mail.gmail.com>
	<25f33c90-38e6-4f86-940f-05ed6baf27ab@aklaver.com>
	<CAA23SdsPFiVDfBZWO-oKgq5SzZEAs9fNydgdbw=2BELMmXfxAA@mail.gmail.com>


On 1/3/24 9:50 AM, arun chirappurath wrote:
>
>
> On Wed, 3 Jan, 2024, 23:03 Adrian Klaver, <adrian.klaver@aklaver.com> 
> wrote:
>
>     On 1/3/24 09:24, arun chirappurath wrote:
>     > Hi Adrian,
>     >
>     > Thanks for your mail.
>     >
>     > Is this for all tables in the database or a subset? Yes
>
>     Yes all tables or yes just some tables?
>     All tables.except some which has user details. 
>
>
>     >
>     > Does it need to deal with foreign key relationships? No
>     >
>     > What are the sizes of the existing data and what size sample
>     data do you
>     > want to produce?1Gb and 1Gb test data.
>
>     If the source data is 1GB and the test data is 1GB then there is no
>     sampling, you are using the data population in its entirety.
>
>     Yes.would like to double the load and test.
>

Does that mean you want to take the 1GB of your existing data and double 
it to 2GB while maintaining

the data distribution from the original data?


>
> Also do we have any standard methods for sampling and generating test data


Something like?:


https://www.postgresql.org/docs/current/sql-select.html


"|TABLESAMPLE /|sampling_method|/ ( /|argument|/ [, ...] ) [ REPEATABLE 
( /|seed|/ ) ]|

    A |TABLESAMPLE| clause after a /|table_name|/ indicates that the
    specified /|sampling_method|/ should be used to retrieve a subset of
    the rows in that table. This sampling precedes the application of
    any other filters such as |WHERE| clauses. The standard PostgreSQL
    distribution includes two sampling methods, |BERNOULLI| and
    |SYSTEM|, and other sampling methods can be installed in the
    database via extensions

...
    "

Read the rest of the documentation for TABLESAMPLE to get the details.



>
>
>     >
>     > On Wed, 3 Jan, 2024, 22:40 Adrian Klaver,
>     <adrian.klaver@aklaver.com
>     > <mailto:adrian.klaver@aklaver.com>> wrote:
>     >
>     >     On 1/2/24 23:23, arun chirappurath wrote:
>     >      > Hi All,
>     >      >
>     >      > Do we have any open source tools which can be used to create
>     >     sample data
>     >      > at scale from our postgres databases?
>     >      > Which considers data distribution and randomness
>     >
>     >
>     >
>     >      >
>     >      > Regards,
>     >      > Arun
>     >
>     >     --
>     >     Adrian Klaver
>     > adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>
>     >
>
>     -- 
>     Adrian Klaver
>     adrian.klaver@aklaver.com
>

view thread (8+ messages)

Message-ID: <49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com>
Permalink:  ../49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com/
Also on:    postgresql.org/message-id/49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: adrian.klaver@aklaver.com, arunsnmimt@gmail.com
  Subject: Re: Sample data generator for performance testing
  In-Reply-To: <49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com>
  References: <CAA23Sdv+0PnmpkK1CtHMzVBuHmw09m-aBefEDL2PTRv5rFsB+w@mail.gmail.com> <42a39fc7-e5e1-4ee7-ab9e-a2da4bf5209f@aklaver.com> <CAA23SdvtLY1exoY6dj03YAcL2PRXxSpnVPhzjpZKkMqBB13QNw@mail.gmail.com> <25f33c90-38e6-4f86-940f-05ed6baf27ab@aklaver.com> <CAA23SdsPFiVDfBZWO-oKgq5SzZEAs9fNydgdbw=2BELMmXfxAA@mail.gmail.com> <49380122-9643-43f0-bbef-bd34ab6d434d@aklaver.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox