From: Arthur Zakirov <a.zakirov@postgrespro.ru>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Pavel Stehule <pavel.stehule@gmail.com>
Cc: Ildus Kurbangaliev <i.kurbangaliev@postgrespro.ru>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: [PROPOSAL] Shared Ispell dictionaries
Date: Mon, 19 Mar 2018 22:18:32 +0300
Message-ID: <20180319191830.GA16075@arthur.localdomain> (raw)
In-Reply-To: <90c04bf7-fcab-8b0b-b461-43b46bf79970@2ndquadrant.com>
References: <CAFj8pRB=FWzq=FfoSPeoJfXG4a7P0Abp+0XKmet3E34tiuTUFA@mail.gmail.com>
<CAFj8pRDm9nhZCf16XDer-eeciqy0wipR0Y4+ZdwGMAWuv-k6ag@mail.gmail.com>
<20180307131819.GA13301@zakirov.localdomain>
<3c7910ba-2a1b-5b73-71c6-d037ba1e89bd@2ndquadrant.com>
<CAKNkYnzWuSV-JoMBPDrWPjomUC4n1QtYd8Yugo9hpJrbtHcALg@mail.gmail.com>
<d4d52484-e269-9d7b-cef5-c1e95a03d9c6@2ndquadrant.com>
<20180319013438.s7jf3kwwwy6m7hqk@alap3.anarazel.de>
<9c27384d-4538-15be-0e94-0a67a52e7b0b@2ndquadrant.com>
<20180319180717.wrqxltiftelf2boc@alap3.anarazel.de>
<90c04bf7-fcab-8b0b-b461-43b46bf79970@2ndquadrant.com>
On Mon, Mar 19, 2018 at 07:40:54PM +0100, Tomas Vondra wrote:
>
>
> On 03/19/2018 07:07 PM, Andres Freund wrote:
> > You've to manually configure a setting that can only be set at server
> > start. You can't set it as big as necessary because it might use up
> > memory better used for other things. It needs the full space for
> > dictionaries even if the majority of it never will be needed. All of
> > those aren't needed in an mmap world.
> >
>
> Which is not quite true, because that's not what the patch does.
>
> Each dictionary is loaded into a separate dsm segment when needed, which
> is then stored in a dhash table. So most of what you wrote is not really
> true - the patch does not pre-allocate the space, and the setting might
> be set even after server start (it's not defined like that currently,
> but that should be trivial to change).
Oh, it's true. I had plans to fix it but somehow I forgot to allow to change
max_shared_dictionaries_size GUC via pg_reload_conf(). I'll fix it and
will send new version of the patch.
> > To me it seems we'll end up needing a heck of a lot more code that
> > the OS already implements if we do it ourselves.
> >
>
> Like what? Which features do you expect to need much more code?
>
> The automated reloading will need a fairly small amount of code - the
> main issue is deciding when to reload, and as I mentioned before that's
> more complicated than you seem to believe. In fact, it may not even be
> possible - there's no way to decide if all files are already updated.
> Currently we kinda ignore that, on the assumption that dictionaries
> change only rarely. We may do the same thing and reload the dict if at
> least one file changes. In any case, the amount of code is trivial.
>
> In fact, it may be more complicated in the mmap case - how do you update
> a dictionary that is already mapped to multiple processes?
>
> The eviction is harder - I'll give you that. But then again, I'm not
> sure the mmap approach is really what we want here - it seems better to
> evict the whole dictionary, than some random pages from many of them.
Agree. mmap approach requires same code plus code to handle cache files,
which will be mapped into memory. In mmap approach we need to solve same
issues we face and more. Also we need somehow automatically reload
dictionaries in both cases.
--
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: a.zakirov@postgrespro.ru, tomas.vondra@2ndquadrant.com, andres@anarazel.de, pavel.stehule@gmail.com, i.kurbangaliev@postgrespro.ru
Subject: Re: [PROPOSAL] Shared Ispell dictionaries
In-Reply-To: <20180319191830.GA16075@arthur.localdomain>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox