Subject: Re: Loading table with indexed jsonb field is stalling
To: Will Hartung <willhartung@gmail.com>
Cc: Ron <ronljohnsonjr@gmail.com>, pgsql-general@lists.postgresql.org
References: 
 <CAKMEDdxE95SC76wadMzrH454HpvCV4tOdEmuZRO=pwArEw6YVg@mail.gmail.com>
 <20505.1558202555@sss.pgh.pa.us>
 <90835D2B-1FBA-4F26-8157-7C831C24B266@gmail.com>
 <cdc049f0-1980-f5a8-726a-629345b41ef0@aklaver.com>
 <3A35A039-D8F5-4B40-9B41-42E70BEF2C2E@gmail.com>
 <806d4915-f05d-8e75-0a42-e11b48596ab5@gmail.com>
 <12E9FEDD-37E9-4D51-9BA7-94FF5039BB1B@gmail.com>
 <563e995f-d7b1-7e2f-c1f1-b00db7a18589@gmail.com>
 <3EF519CA-17FE-4B68-8B7A-78200E0A408F@gmail.com>
 <7c0a001e-d3d5-c521-1a27-77d05f547fef@aklaver.com>
 <375E86FF-5B4D-4896-B4ED-FCA8C1D3A60B@gmail.com>
From: Adrian Klaver <adrian.klaver@aklaver.com>
Message-ID: <703a6c76-854d-3d1c-8be6-30864c873d05@aklaver.com>
Date: Mon, 20 May 2019 17:14:01 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.6.1
MIME-Version: 1.0
In-Reply-To: <375E86FF-5B4D-4896-B4ED-FCA8C1D3A60B@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 5/20/19 4:49 PM, Will Hartung wrote:
> 
>> On May 20, 2019, at 4:39 PM, Adrian Klaver <adrian.klaver@aklaver.com 
>> <mailto:adrian.klaver@aklaver.com>> wrote:
>>
>> So the issue is building the index(s) not loading the data, correct?
> 
> Correct.
> 
> Since I have been exploring this, I have not been able to successfully 
> create a gin jsonb_path_ops index on a jsonb field either as a new index 
> on a table with existing data, or as a pre-existing index on an empty 
> table and loading the data in.
> 
> The create index does not finish on the existing data, and loading data 
> does not finish when the index pre-exists.

Well looks like you are down to Tom's suggestion of creating a test 
case. Given that it seems confined to the jsonb field and corresponding 
index, I would think that is all that is needed for the test case. Start 
with some smaller subset, say 10,000 rows and work up till you start 
seeing an issue.

> 
> We currently have a 68M row table loaded in an Aurora instance on AWS (I 
> do not know the instance/machine size), and the index has been building 
> for 48 hours now.
> 
> The pg_restore is currently working on that index:
> 
> 
> CREATE INDEX eis_ent_traits
>    ON public.eis_entity
>    USING gin
>    (traits jsonb_path_ops);
> 
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com