Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tZXoy-00Afs7-73 for pgsql-general@arkaria.postgresql.org; Sun, 19 Jan 2025 16:06:44 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tZXov-008EC3-PW for pgsql-general@arkaria.postgresql.org; Sun, 19 Jan 2025 16:06:42 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tZXov-008EBu-Ey for pgsql-general@lists.postgresql.org; Sun, 19 Jan 2025 16:06:41 +0000 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1tZXos-000ONT-1b for pgsql-general@lists.postgresql.org; Sun, 19 Jan 2025 16:06:40 +0000 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-4368a293339so41888655e9.3 for ; Sun, 19 Jan 2025 08:06:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737302798; x=1737907598; darn=lists.postgresql.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=FFqD06+iXTYJqQSZ20CN8cwn0HPmUnsjWdppxnWntKE=; b=FM2Ufb/ipf5EDoxb1wuRHL41BNvDxB5Hg2HKQLe41fi6OZ7h7yqXQkBYP1iV6+KByY mzQRy/jStLeOoHQt067i/nXRSkzQ7QDQ7bXTegnGb3D2CtdUNpwyV933u4SKCaz0N79c 8hJJePWjb0V6A0P5hhYlVOWK/fYJ8W0jQOVHjvwTGHDhC6zZtQOPsN7eyxxmroTh5gyu 6ijU+4Rb51dwK9cqYGjmZhaCj2OJIT7VtdA4JEzYJsw6BOXG4pHYsHqnNwi5sqnTuWAX 0iyc0TP2m3dySMgpmkYhf28uu2m1Hpb/z4MqhV4hkkXsEJNWBsUPDKffQ0oi4Rt7A8Rl 21Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737302798; x=1737907598; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FFqD06+iXTYJqQSZ20CN8cwn0HPmUnsjWdppxnWntKE=; b=TSTwgoRTksddhkfbBNf+oXgOz1khsirtn9shxSZnSrSZIAJkp1nY2N4QSwoaZBtyCl KmJ7vIUQX7KyA8ZmKhvHRDxfu+4fHrXb750FFMMlnGjF/mlSZ3xKMNfCp0/f1zdRGoic NMugtJK1HRZTQ4jwx5jICv/CmjAhrqeVnmFvbF5pIyP4XEORH0uPC8k21h4kdSM7GnbO Y/xvX6qRr9N8qvTa4t+b3e1pJQgLEglVIBWdcE2hvT1SNPWCGBrG391XOuCFkTteHM32 GM8QCDVhPcGiYcHvItsSKOeg82UzX5DMpjPWTBtOJ3ApS/Jh27NZN6g6g6R3ta9T8fvE 5ugQ== X-Gm-Message-State: AOJu0YxmpoQT3Nw3JFXmHraKpcfCiWWCWm9X3fUuq2PnHaerNtMY/quD Bb2ffHBmzwZc1cxbbgbdiSk9QUOmk8D3xyuHXBGWeXdQNgW7PBsMihFVlTx8 X-Gm-Gg: ASbGnctesorK55/psk3PhE0zS96PJQEvaRQpjZsktBsKcAa2q8wkv/ZUD2yW2vKn41Z X1tl4p9gaf4Mj/2TQEMgvHN+s7Zyxol9ar77ZjyyqR36HW+J2zcVfIORX11dUlu+NJoM64+/Gul 24+O9UIv7oMmNMaAYUHsnjWFu4N3QtSXzqQnD0ywqUmNSU0MfX/+c+9twlgshiEFEbp5JCu6qKZ 3rwnSMUPuuluziV9u16loIcuL/yvBltQF0xsYxKsR/QPYQS175rZeUbM86BNtVNjXXYlDs= X-Google-Smtp-Source: AGHT+IEA5Ei+l969H6p8GfI+5PO4INeXiTUZEDqzJneX+Riwkexai0ZM9KAYURMUst+1Bnb2Ew6nGQ== X-Received: by 2002:a05:600c:4743:b0:434:a802:43d with SMTP id 5b1f17b1804b1-438914315afmr82291355e9.27.1737302797408; Sun, 19 Jan 2025 08:06:37 -0800 (PST) Received: from linux.localnet ([2a02:85f:e05c:e701:f8b1:c915:3e80:7a7f]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-437c74c4f85sm168172705e9.18.2025.01.19.08.06.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jan 2025 08:06:37 -0800 (PST) From: auxsvr@gmail.com To: pgsql-general@lists.postgresql.org Subject: Re: Design of a reliable task processing queue Date: Sun, 19 Jan 2025 18:06:12 +0200 Message-ID: <2686461.fDdHjke4Dd@linux> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="utf-8" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Saturday, 18 January 2025 12:44:07 EET Alex Burkhart wrote: > Hey team, > > I'm looking for help to organize locks and transaction for a reliable task > queue. > > REQUIREMENTS > > 1. Pending actions are persisted to a database. There's a trace once they > are done. > 2. Application workers pick actions one by one. At any given time, each > action can be assigned to at most one worker (transaction). > 3. If multiple actions have same "lock_id", only one of them is processed > at the time. That has to be action with smallest id. Why reinvent the wheel and not use production-ready code from projects such as que (Ruby), pgqueuer (Python)? -- Regards, Peter