Subject: Re: make installcheck-world in a clean environment
To: Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>
Cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
References: <bfee872b-ab16-ec7c-8f8a-b3949add1c1b@gmail.com>
 <CA+TgmoafXiYMj6ROd09PxLFQ_dpurP1FJOL4ZwDonCkPeSubWg@mail.gmail.com>
 <aff02bd6-20c9-0b6e-4e3c-a3428ce6568b@gmail.com>
 <CA+TgmoZV08_6SXmag0hFkOurgWPjks6Jwyh+DeFXsCxLWLVjfw@mail.gmail.com>
 <24834.1525712835@sss.pgh.pa.us>
From: Alexander Lakhin <exclusion@gmail.com>
Message-ID: <ec02a440-0346-4bdc-5118-0f7970d33c16@gmail.com>
Date: Tue, 8 May 2018 08:44:22 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.7.0
MIME-Version: 1.0
In-Reply-To: <24834.1525712835@sss.pgh.pa.us>
Content-Type: multipart/alternative;
 boundary="------------AC4C62FB2A91E58D36B4B061"
Content-Language: en-US
Precedence: bulk

This is a multi-part message in MIME format.
--------------AC4C62FB2A91E58D36B4B061
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

07.05.2018 20:07, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> After thinking about this some more, I think the question here is
>> definitional.  A first attempt at defining 'make installcheck' is to
>> say that it runs the tests from the build tree against the running
>> server.  Certainly, we intend to use the SQL files that are present in
>> the build tree, not the ones that were present in the build tree where
>> the running server was built.  But what about client programs that we
>> use to connect to the server?  You're suggesting that we use the
>> pre-installed ones, but that is actually pretty problematic because
>> the ones we see as installed might correspond neither to the contents
>> of the build tree nor to the running server.
If the contents of the source tree doesn't correspond to the running 
server, then I'm afraid, we can't installcheck it for sure.
I think it's supposed that we use for installcheck exactly the same 
source files that was used to build the server.
Regarding clients program, if we will not use/check it while 
installchecking, then by what means they can be tested when installed?
I think, the pgsql-packagers would like to check whether the whole 
installation of PostgreSQL is working, not only the server.
For me, very realistic and most useful scenario of installcheck is:
install postgresql-x.rpm
install postgresql-x.src.rpm
./configure --prefix=$target_path --enable-tap-tests, etc...
make installcheck(-world)
>>    Conceivably we could end
>> up having a mix of assets from three different places: (1) the running
>> server, (2) the build tree, (3) whatever is in our path at the moment.
>> That seems very confusing.  So now I think it's probably right to
>> define 'make installcheck' as using the assets from the build tree to
>> test the running server.  Under that definition, we're missing some
>> dependencies, but USE_INSTALLED_ASSETS isn't a thing we need.
I see another scenario, that was discussed a month ago - remote (or 
server-only) installcheck.
( 
https://www.postgresql.org/message-id/flat/CAM0nTJ6iorX_tmg5MX0mQU3z3w%3D9wk%2BpGK8zrvn7DNWqnyv%2BsQ%40mail.gmail.com 
)
It can be useful too and if it will be supported, then 
USE_INSTALLED_ASSETS usage should be extended to psql, etc.
> Nah, I disagree with this.  To me, the purpose of "make installcheck"
> is to verify the correctness of an installation, which I take to include
> the client programs as well as the server.  I think that "make
> installcheck" ought to use the installed version of any file that we
> actually install, and go to the build tree only for things we don't
> install (e.g. SQL test scripts).
Yes, that's the proposed patches intended for.  I didn't encounter any 
problems with it during internal testing with Linux and mingw-builds.
> If the user has screwed up his PATH or other environmental aspects so that
> what he's testing isn't a single installation, that's his error, not
> something that "make installcheck" ought to work around.  Indeed, maybe
> such aspects of his setup are intentional, and second-guessing them would
> completely defeat his purpose.  In any case, if you want to test the
> build-tree assets, that's what "make check" is for.
Even modified configs could lead to test failures (for example, 
lc_messages can break server log checking), so the installcheck should 
be performed against some clean and determinated installation anyway.

Best regards,
------
Alexander Lakhin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company


--------------AC4C62FB2A91E58D36B4B061
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">07.05.2018 20:07, Tom Lane wrote:<br>
    </div>
    <blockquote type="cite" cite="mid:24834.1525712835@sss.pgh.pa.us">
      <pre wrap="">Robert Haas <a class="moz-txt-link-rfc2396E" href="mailto:robertmhaas@gmail.com">&lt;robertmhaas@gmail.com&gt;</a> writes:
</pre>
      <blockquote type="cite">
        <pre wrap="">After thinking about this some more, I think the question here is
definitional.  A first attempt at defining 'make installcheck' is to
say that it runs the tests from the build tree against the running
server.  Certainly, we intend to use the SQL files that are present in
the build tree, not the ones that were present in the build tree where
the running server was built.  But what about client programs that we
use to connect to the server?  You're suggesting that we use the
pre-installed ones, but that is actually pretty problematic because
the ones we see as installed might correspond neither to the contents
of the build tree nor to the running server.</pre>
      </blockquote>
    </blockquote>
    If the contents of the source tree doesn't correspond to the running
    server, then I'm afraid, we can't installcheck it for sure.<br>
    I think it's supposed that we use for installcheck exactly the same
    source files that was used to build the server.<br>
    Regarding clients program, if we will not use/check it while
    installchecking, then by what means they can be tested when
    installed?<br>
    I think, the pgsql-packagers would like to check whether the whole
    installation of PostgreSQL is working, not only the server.<br>
    For me, very realistic and most useful scenario of installcheck is:<br>
    install postgresql-x.rpm<br>
    install postgresql-x.src.rpm<br>
    ./configure --prefix=$target_path --enable-tap-tests, etc...<br>
    make installcheck(-world)<br>
    <blockquote type="cite" cite="mid:24834.1525712835@sss.pgh.pa.us">
      <blockquote type="cite">
        <pre wrap="">  Conceivably we could end
up having a mix of assets from three different places: (1) the running
server, (2) the build tree, (3) whatever is in our path at the moment.
That seems very confusing.  So now I think it's probably right to
define 'make installcheck' as using the assets from the build tree to
test the running server.  Under that definition, we're missing some
dependencies, but USE_INSTALLED_ASSETS isn't a thing we need.</pre>
      </blockquote>
    </blockquote>
    I see another scenario, that was discussed a month ago - remote (or
    server-only) installcheck.<br>
    (
<a class="moz-txt-link-freetext" href="https://www.postgresql.org/message-id/flat/CAM0nTJ6iorX_tmg5MX0mQU3z3w%3D9wk%2BpGK8zrvn7DNWqnyv%2BsQ%40mail.gmail.com">https://www.postgresql.org/message-id/flat/CAM0nTJ6iorX_tmg5MX0mQU3z3w%3D9wk%2BpGK8zrvn7DNWqnyv%2BsQ%40mail.gmail.com</a>
    )<br>
    It can be useful too and if it will be supported, then
    USE_INSTALLED_ASSETS usage should be extended to psql, etc.
    <blockquote type="cite" cite="mid:24834.1525712835@sss.pgh.pa.us">
      <pre wrap="">Nah, I disagree with this.  To me, the purpose of "make installcheck"
is to verify the correctness of an installation, which I take to include
the client programs as well as the server.  I think that "make
installcheck" ought to use the installed version of any file that we
actually install, and go to the build tree only for things we don't
install (e.g. SQL test scripts).</pre>
    </blockquote>
    Yes, that's the proposed patches intended for.  I didn't encounter
    any problems with it during internal testing with Linux and
    mingw-builds.<br>
    <blockquote type="cite" cite="mid:24834.1525712835@sss.pgh.pa.us">
      <pre wrap="">If the user has screwed up his PATH or other environmental aspects so that
what he's testing isn't a single installation, that's his error, not
something that "make installcheck" ought to work around.  Indeed, maybe
such aspects of his setup are intentional, and second-guessing them would
completely defeat his purpose.  In any case, if you want to test the
build-tree assets, that's what "make check" is for.</pre>
    </blockquote>
    Even modified configs could lead to test failures (for example,
    lc_messages can break server log checking), so the installcheck
    should be performed against some clean and determinated installation
    anyway.<br>
    <br>
    <font color="#000000"><span style="background: #ffffff">Best
        regards, <br>
        ------ <br>
        Alexander Lakhin <br>
        Postgres Professional: <a class="moz-txt-link-freetext"
          href="http://www.postgrespro.com">http://www.postgrespro.com</a>
        <br>
        The Russian Postgres Company<br>
        <br>
      </span></font><br>
  </body>
</html>

--------------AC4C62FB2A91E58D36B4B061--