Subject: Re: [HACKERS] Custom compression methods
To: Ildus Kurbangaliev <i.kurbangaliev@postgrespro.ru>
Cc: pgsql-hackers@postgresql.org, Ildar Musin <i.musin@postgrespro.ru>
References: <20170907194236.4cefce96@wp.localdomain>
 <20170912175505.4afa11fd@wp.localdomain>
 <20171102152836.60c041e4@wp.localdomain>
 <20171114162356.52e3d388@wp.localdomain>
 <62e46a47-08a8-6e06-5e4f-6f52e0a13202@2ndquadrant.com>
 <20171121174717.69ecd8f4@wp.localdomain>
 <bfb3d34e-ba74-02cb-ca2a-43344866d9fd@2ndquadrant.com>
 <20171123123849.5a686d7b@wp.localdomain>
 <57daf28d-ed76-c364-a9ca-65d0ff71a36f@2ndquadrant.com>
 <20171124123800.034c9208@wp.localdomain>
 <e3378c06-d805-a3f5-6ea7-af2a2417361e@2ndquadrant.com>
 <b969fa5a-bc2a-c220-c992-e6e0de34df14@2ndquadrant.com>
 <20171127185218.5cddad41@wp.localdomain>
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Message-ID: <0158242e-af45-c0f0-4fbb-4d54ca219cfd@2ndquadrant.com>
Date: Mon, 27 Nov 2017 18:20:12 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.3.0
MIME-Version: 1.0
In-Reply-To: <20171127185218.5cddad41@wp.localdomain>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi,

On 11/27/2017 04:52 PM, Ildus Kurbangaliev wrote:
> ...
>
> Hi. This looks like a serious bug, but I couldn't reproduce it yet.
> Did you upgrade some old database or this bug happened after 
> insertion of all data to new database? I tried using your 'archie' 
> tool to download mailing lists and insert them to database, but
> couldn't catch any errors.
> 

I can trigger it pretty reliably with these steps:

    git checkout f65d21b258085bdc8ef2cc282ab1ff12da9c595c
    patch -p1 < ~/custom_compression_methods_v6.patch
    ./configure --enable-debug --enable-cassert \
     CFLAGS="-fno-omit-frame-pointer -O0 -DRANDOMIZE_ALLOCATED_MEMORY" \
     --prefix=/home/postgres/pg-compress
    make -s clean && make -s -j4 install
    cd contrib/
    make -s clean && make -s -j4 install

    export PATH=/home/postgres/pg-compress/bin:$PATH
    pg_ctl -D /mnt/raid/pg-compress init
    pg_ctl -D /mnt/raid/pg-compress -l compress.log start
    createdb archie
    cd ~/archie/sql/
    psql archie < create.sql

    ~/archie/bin/load.py --workers 4 --db archie */* > load.log 2>&1


I guess the trick might be -DRANDOMIZE_ALLOCATED_MEMORY (I first tried
without it, and it seemed working fine). If that's the case, I bet there
is a palloc that should have been palloc0, or something like that.

If you still can't reproduce that, I may give you access to this machine
so that you can debug it there.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services