Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1iGGgJ-0001sN-AQ
	for pgsql-performance@arkaria.postgresql.org; Fri, 04 Oct 2019 06:03:11 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1iGGgH-0002Tg-7n
	for pgsql-performance@arkaria.postgresql.org; Fri, 04 Oct 2019 06:03:09 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
	by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <pryzby@telsasoft.com>)
	id 1iGGgG-0002TQ-RC
	for pgsql-performance@lists.postgresql.org; Fri, 04 Oct 2019 06:03:08 +0000
Received: from mail-io1-xd43.google.com ([2607:f8b0:4864:20::d43])
	by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <pryzby@telsasoft.com>)
	id 1iGGgD-0002th-Ii
	for pgsql-performance@lists.postgresql.org; Fri, 04 Oct 2019 06:03:08 +0000
Received: by mail-io1-xd43.google.com with SMTP id q10so11107085iop.2
        for <pgsql-performance@lists.postgresql.org>;
 Thu, 03 Oct 2019 23:03:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=telsasoft-com.20150623.gappssmtp.com; s=20150623;
        h=date:from:to:cc:subject:message-id:references:mime-version
         :content-disposition:in-reply-to:user-agent;
        bh=Wd0eJBARo0aIaJJ2m0p8PwFv/xjwH1g9mJL0ittaOdY=;
        b=MImxjRz5G+U5MvIZJ7IrQf0NG5bPCyxGX6sgdwZyl2qXHUKThezZ9g0gHGXgUkuWBM
         02D+sf6IOWnR7VXKANdUzJGgXBcChmcq+9aNaRTTKKwqTB2DFYcOdXomDOwwHmC6a3so
         cxoDiUdxvNKgsxzTkHBEgkeuBjTaq/6kySkVPyhHfXCfMi0j7J4hNfkBnjXZCxXSYbZw
         mjbMuszKq+aCYGfGc5SQIes/TnJLOkWM70+72YNYuttelxMAeKeFD3cU7uzh09/UYDje
         jReBgvA5/eQSfTylQOrhteF75aFQ9qqy4gqbjXMysWXbzicjybXmC+25S5uIdjEv0pZE
         dDCg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:date:from:to:cc:subject:message-id:references
         :mime-version:content-disposition:in-reply-to:user-agent;
        bh=Wd0eJBARo0aIaJJ2m0p8PwFv/xjwH1g9mJL0ittaOdY=;
        b=XXagj8oIc0tFhBvUm4eD0L1Lkn5B96Sfidoc17jCQnF0MAIAXxZmpUi2uZOVDPo1ZK
         8hFtG/nKjcLqkoEZBqKIw6zY7sQfwypijrhrSRhAJcoHNJt8VtRuGipTDrRwKdGtfznw
         za9dnbPxJAJsHtOWFc1wphEpgKz/fPettkW3VmBcHovtPBbIpyLVg5g4u1zU7cy7Uepy
         wQHRpHa1P9lh+ieuVoi7drizUlj6jYlQcAgqsG5ax19AKWLGpEljHWkqL2NtV31woMyN
         +8OWotezRVmLu4uXaKwxPYtWwTexyXtIDMDB7EHhH+MfLblbFZLTthENcrs42Yyew36d
         f2kw==
X-Gm-Message-State: APjAAAXhtAulx7QT2HbKwb2Z0t/o6ph0z9jH2EZMlIplHndc9qQQfT7V
	QLVzk8kO9e1n6kHWjqBAsa2KSykLy6c=
X-Google-Smtp-Source: 
 APXvYqw/t7klWsN/ICMn32vzk4ox7kk4jFUmPIBlHu2cvdkUxP5XzJ2U287FF1jMlCMUSTESx13Ufg==
X-Received: by 2002:a92:4799:: with SMTP id e25mr14504751ilk.72.1570168982991;
        Thu, 03 Oct 2019 23:03:02 -0700 (PDT)
Received: from pryzbyj (charmander.telsasoft.com. [50.244.222.1])
        by smtp.gmail.com with ESMTPSA id
 17sm1802795ioo.21.2019.10.03.23.03.01
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Thu, 03 Oct 2019 23:03:01 -0700 (PDT)
Received: by pryzbyj (Postfix, from userid 1000)
	id 92C7480091D; Fri,  4 Oct 2019 01:03:00 -0500 (CDT)
Date: Fri, 4 Oct 2019 01:03:00 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: Ogden Brash <info@litika.com>
Cc: pgsql-performance@lists.postgresql.org
Subject: Re: Some observations on very slow pg_restore operations
Message-ID: <20191004060300.GA11241@telsasoft.com>
References: 
 <CAFCR_K35nhd3FxEuDsFsu9rQDHEEYxeRm0J-KFDF+jVMODSJgQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CAFCR_K35nhd3FxEuDsFsu9rQDHEEYxeRm0J-KFDF+jVMODSJgQ@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
List-Id: <pgsql-performance.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-performance@lists.postgresql.org>
List-Owner: <mailto:pgsql-performance-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-performance>
Precedence: bulk

What is the state of KSM/THP?  Did you try disabling them ?  I've seen these
can cause high iowait (although that was a virtual environment).  It would be
interesting to see vmstat output.
tail /sys/kernel/mm/ksm/run /sys/kernel/mm/transparent_hugepage/khugepaged/defrag /sys/kernel/mm/transparent_hugepage/enabled /sys/kernel/mm/transparent_hugepage/defrag                                                           
https://www.postgresql.org/message-id/20170718180152.GE17566%40telsasoft.com

I realize the postgres params don't seem related to the difference between
local/remote/NFS, but did you see all these ?
https://www.postgresql.org/docs/current/populate.html

Justin