Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1t7i9p-0007zk-9v for pgsql-general@arkaria.postgresql.org; Sun, 03 Nov 2024 21:29:12 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1t7i8n-000rEl-9p for pgsql-general@arkaria.postgresql.org; Sun, 03 Nov 2024 21:28:09 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1t7i8m-000rEd-Ho for pgsql-general@lists.postgresql.org; Sun, 03 Nov 2024 21:28:09 +0000 Received: from mail-ej1-x643.google.com ([2a00:1450:4864:20::643]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1t7i8g-00018L-Py for pgsql-general@lists.postgresql.org; Sun, 03 Nov 2024 21:28:08 +0000 Received: by mail-ej1-x643.google.com with SMTP id a640c23a62f3a-a9a5f555cfbso279479966b.1 for ; Sun, 03 Nov 2024 13:28:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730669281; x=1731274081; darn=lists.postgresql.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=N43znsr7nkEIXE0kHPY3HPWdMO5aInKMapWRvR/HtQY=; b=kTFKEtyvDNx9Wu577qYx4Vpg3zd+qRgRG7LDzVPT/2yxQ9mBpkEI4rlZt90XxDMtlA UdmjZ4n3gD6e3m3ZviG5tOU216BZTovPl7fs9rzCemt5OniX5pkW8b5T55NsFP3XNb1s KLNwbQLgbR4tCERpIw0SlBVI5TZ1fdaXtaFbhs4VbcmL8XIu1t8tXqE1LRowmaJD5Lpa olvlnnrjg25SHNFOWnxgHVGc+y2RhYOiwuRUg+rHrxhyP0dvqERGOoC9n0WSvg8nwETR NoeWasN5OB2WGx/LrmAIolq6491nQxUhqmnLLFOTbLgE8qIenjndISdK+00VWz77TzOj BvlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730669281; x=1731274081; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N43znsr7nkEIXE0kHPY3HPWdMO5aInKMapWRvR/HtQY=; b=vUohzM5gHBkf2OfdEGgpnll3Fr2q6jcCZW8jy1x1b31oez0g+10zSWUJ13dJdF3DS1 6oASxhEhHqu5fQa3Jlb65zk7FcG5DBa1YBxFPHQe7za+gq08AVIRU2I0QMiNR1LYWb5O 1utqt+/M4H9s9sX0gqqtYoVrq54cNU+i0Z6mSQ8hhRxoBaDOwUgx5JtvYHQFwU9hD69u lj2c1kceXZFc2g6i7tevQNUYxgtJCCdgcr59+uUuu2ST82eRlNYKWQV2Y4JvwiTYOfPX BgrcDPlxBnqN7HQoT/u+B42LiKM7XkDggj8gX1c8xzP7sEud8NtrbHP1YC/+myWCoG8P +I0w== X-Gm-Message-State: AOJu0YwugPUlttw+UOjpg6wRPaFrLDXvxLurlu9YuDY3or8bY9UqmMVu GVvido7FGkLbwLlKtXrCxM6eolxNOLC2K0xJcTh2XzlZeXmxZQgnDkGGYXgskCxmk0RJHVpdPKk rcFOg+enpOEJEj3A+6CEcnBgUNF6jyWC74ig= X-Google-Smtp-Source: AGHT+IErnL8I+zH5QOsub2nT1tpFN8jVPltXNAeHWlcH55QpVX+zBuYSdT7uW5cLdzRMaQFxasVYRdDRKUgAMDxz3t8= X-Received: by 2002:a05:6402:4308:b0:5c9:34b4:69a8 with SMTP id 4fb4d7f45d1cf-5ceb923ee52mr12025207a12.6.1730669280823; Sun, 03 Nov 2024 13:28:00 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Vijaykumar Jain Date: Mon, 4 Nov 2024 02:57:49 +0530 Message-ID: Subject: Re: Random memory related errors on live postgres 14.13 instance on Ubuntu 22.04 LTS To: Ian J Cottee Cc: pgsql-general@lists.postgresql.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Sat, 2 Nov 2024 at 12:50, Ian J Cottee wrote: > As the previous errors (thankfully) are not showing now I can't really do= any more debugging but I'll report back on the results of the memtest. just for the sake of this thread, i wanted to make a mention of Summary of Errors reported early on... ( Stuck Spinlock Detected: This error indicates that a thread is unable to acquire a lock, which can happen due to contention or a deadlock situation. It often points to performance issues or bugs in the PostgreSQL codebase. Free(): Corrupted Unsorted Chunks: This message typically arises from memory management issues, particularly when the memory allocator detects corruption in its internal structures. This could be indicative of bugs in the application code or PostgreSQL itself. Double Free or Corruption (!prev): This error occurs when the program attempts to free memory that has already been freed or is corrupted. It can lead to crashes and is often linked to improper memory handling in the application. Corrupted Size vs. Prev_Size: Similar to the previous errors, this suggests that there is a mismatch in expected memory sizes, which can be caused by buffer overflows or improper memory allocation. Corrupted Double-Linked List: This indicates that the internal data structures used for managing memory are corrupted, often resulting from improper memory handling. Stack Smashing Detected: This error occurs when a program writes more data to a buffer than it can hold, corrupting adjacent memory. It=E2=80=99s= a common sign of buffer overflow vulnerabilities. Segmentation Fault: A segmentation fault indicates that the program tried to access an invalid memory location, leading to a crash. This is often a result of programming errors such as dereferencing null pointers or accessing out-of-bounds array elements. yes, i used perplexity to understand these errors and asked for what scenarios do we get those errors, it may not be all correct, or may even be wrong, which i'll have dive more in later. but if these are not hardware errors, then there can be serious bugs which can be exploited for vulnerabilities ... around buffer overflows. if you have a standard installation from ubuntu package binaries .... i think there might be some lower level C code in functions? or extensions also in play which can lead to above errors.i hope there was a core dumped somewhere and a stacktrace might have some info on what led to those errors could help understand if there is unsafe code somewhere and what queries resulted in that. I understand this forum is not to discuss AI responses ... but I also did not want to ignore them due to lack of knowledge... hence sharing. as always, i can be corrected or wronged