Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kbBoD-0006Z2-F9 for pgsql-bugs@arkaria.postgresql.org; Sat, 07 Nov 2020 00:10:21 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kbBoA-0005vu-RI for pgsql-bugs@arkaria.postgresql.org; Sat, 07 Nov 2020 00:10:18 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kbBoA-0005vn-LS for pgsql-bugs@lists.postgresql.org; Sat, 07 Nov 2020 00:10:18 +0000 Received: from mail-ej1-x644.google.com ([2a00:1450:4864:20::644]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1kbBo3-0001sE-3L for pgsql-bugs@lists.postgresql.org; Sat, 07 Nov 2020 00:10:17 +0000 Received: by mail-ej1-x644.google.com with SMTP id s25so4273292ejy.6 for ; Fri, 06 Nov 2020 16:10:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0j8ZceJeu/gp/D3UQYAcZvX7USetv5Va4FhE0VGOUsw=; b=TQ8diDuTQsNlEoNUcKlhlCv01C+yC9HDt94O2v2dSCj61/pxjmAzLBmP+DHrQe3fBX r7VjddfZz9ZMD3k0REvrqhEJLN5PbuAK5Q0lX1JryhoiOVc0aHvpRS398BSzCjkmA/s3 GRjZUYMz0M4NPHEs94SyhjdlHvFijEtTKn6qt1/BIEmJ2J8FMLsDR4dcj87qvIgCWbpW 38QODEFlaiwfhcHVtCiOCQn+MZFeFChgQCqGSHTcFqZB+jKOvJ3jr1NyDS7iokuVR38A 1NASbuTdQTIOGWVrgz3Cu6z0oxaNHr78l6jTYXEcIUoiTnDCoU1kK86tmMhS2p2W5Z6+ BbKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0j8ZceJeu/gp/D3UQYAcZvX7USetv5Va4FhE0VGOUsw=; b=duL/r7MKY+FqUK1kdyEX2L78Qkbw/rHZ8ifrGloL+mAdHeWNh8nJu8nU6MzMIHvrQ6 /u7aRB7Pr1ayQCVQCWeX50qD3W3u5S8ZQRc8FLsIc0TDaRUcgi/yylwh9IIYWgDxN4JR FhWl0ermtuiCz2TgJ/4CLx8mdLjja9BZPJphnCDflvVIP7eD1isSkVTI1vGbnGsJyc38 zWMF7zZkcHCdV5UiPQMXMwJflSO59PTO1W+zwKV7Zkt1W8LoM4N0XEi/NaRBexSbebLm K5vzRBf6oLgevpRzr03kiI/3CzmymPvwBly0H9tnyWfGBa69/obXioNeXtZ6JilvUl9J aXcA== X-Gm-Message-State: AOAM531HbvLU27auRCpwtydY7gn8A/Vlo1+w74QK2ULNz4vTZKFu8tf2 7HCAVcpVqMwgzEuNF8xqYcTNeG/zD6FXPUKXDir9+dstEkhyxDmalWa5cOyl0aEVFYYpdoMQEi6 qJXmv9zj7r2gN5bYAshUr5jVXRLnvgs2yCPCbYcNQ4ZleanKIKbs0Up8HhZf/lQu5/hkNSxrRbv uO9SmOWDLT3I674SrUjoFIWe9DMaJF+YMPVgvlYeoOkyeUyDBW9tDh1y4= X-Google-Smtp-Source: ABdhPJzXFuHbO4qMuXUu1c1rVffhpVk0tBppfU/mBcxNOIiy0rNwyqsntihQBPS6q4Oxk6sDnIROmA== X-Received: by 2002:a17:906:2683:: with SMTP id t3mr4756654ejc.414.1604707809305; Fri, 06 Nov 2020 16:10:09 -0800 (PST) Received: from localhost (ip-86-49-245-76.net.upcbroadband.cz. [86.49.245.76]) by smtp.gmail.com with ESMTPSA id z14sm2020202ejl.110.2020.11.06.16.10.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 16:10:09 -0800 (PST) Date: Sat, 7 Nov 2020 01:10:04 +0100 From: Tomas Vondra To: "Burgess, Freddie" Cc: "pgsql-bugs@lists.postgresql.org" Subject: Re: pg_dump error attempting to upgrade from PostgreSQL 10 to PostgreSQL 12 Message-ID: <20201107011004.50b6de0f@enterprisedb.com> In-Reply-To: References: X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-CLOUD-SEC-AV-Info: enterprisedb,google_mail,monitor X-CLOUD-SEC-AV-Sent: true X-Gm-Spam: 0 X-Gm-Phishy: 0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Thu, 5 Nov 2020 21:19:17 +0000 "Burgess, Freddie" wrote: > Simple steps: > > BACKUP: pg_dump -U postgres -d > sherlock.dmp <- From the > pg10 instance RESTORE: psql -U postgres -d -1 -f > sherlock.dmp <- On the pg12 instance > > Postgres Log: > > free(): invalid pointer > free(): invalid pointer > 2020-11-05 14:07:33.784 EST [26] LOG: background worker "parallel > worker" (PID 150) was terminated by signal 6: Aborted 2020-11-05 > 14:07:33.784 EST [26] LOG: terminating any other active server > processes 2020-11-05 14:07:33.784 EST [32] WARNING: terminating > connection because of crash of another server process 2020-11-05 > 14:07:33.784 EST [32] DETAIL: The postmaster has commanded this > server process to roll back the current transaction and exit, because > another server process exited abnormally and possibly corrupted > shared memory. 2020-11-05 14:07:33.784 EST [32] HINT: In a moment > you should be able to reconnect to the database and repeat your > command. 2020-11-05 14:07:33.784 EST [61] WARNING: terminating > connection because of crash of another server process 2020-11-05 > 14:07:33.784 EST [61] DETAIL: The postmaster has commanded this > server process to roll back the current transaction and exit, because > another server process exited abnormally and possibly corrupted > shared memory. 2020-11-05 14:07:33.784 EST [61] HINT: In a moment > you should be able to reconnect to the database and repeat your > command. 2020-11-05 14:07:34.699 EST [26] LOG: all server processes > terminated; reinitializing 2020-11-05 14:07:42.266 EST [154] LOG: > database system was interrupted; last known up at 2020-11-05 14:06:02 > EST 2020-11-05 14:08:05.855 EST [154] LOG: database system was not > properly shut down; automatic recovery in progress 2020-11-05 > 14:08:05.859 EST [154] LOG: redo starts at 7E/93B22C8 2020-11-05 > 14:08:15.931 EST [154] LOG: invalid record length at 7F/74ECBE30: > wanted 24, got 0 2020-11-05 14:08:15.931 EST [154] LOG: redo done at > 7F/74ECBDF8 2020-11-05 14:08:41.673 EST [26] LOG: database system is > ready to accept connections > > PostgreSQL is installed on a docker container, running on a EC2 > instance with 256 GB of memory > It'd be interesting to know what is doing the crashing parallel worker. Considering it's a background worker, the easiest way is probably enabling core dumps and inspecting them with gdb. Make sure you have debug symbols installed and send us the backtrace. Some basic instructions are in: https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD#Getting_a_trace_from_a_randomly_crashing_backend The error message is most likely a random glibc free() error, not sure where it's coming from or whether it has something to do with docker. Maybe try preparing a reproducer, i.e. a small database triggering the issue, which we might use to reproduce the issue on our machines. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company