MIME-Version: 1.0
References: <CANzqJaCaLwc6rOK7O8PkCgVDi_8fg6duzdUrMy0rtn_tQL144g@mail.gmail.com>
In-Reply-To: <CANzqJaCaLwc6rOK7O8PkCgVDi_8fg6duzdUrMy0rtn_tQL144g@mail.gmail.com>
From: Florents Tselai <florents.tselai@gmail.com>
Date: Thu, 12 Sep 2024 22:43:56 +0300
Message-ID: <CA+v5N41tqJDYxd3JVAz3NKxgPBp28RUnAgQ3h0M1bPNMLmtm+Q@mail.gmail.com>
Subject: Re: Better way to process records in bash?
To: Ron Johnson <ronljohnsonjr@gmail.com>
Cc: pgsql-general <pgsql-general@postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000faffd00621f15518"
Archived-At: <https://www.postgresql.org/message-id/CA%2Bv5N41tqJDYxd3JVAz3NKxgPBp28RUnAgQ3h0M1bPNMLmtm%2BQ%40mail.gmail.com>
Precedence: bulk

--000000000000faffd00621f15518
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Sep 12, 2024 at 6:08=E2=80=AFPM Ron Johnson <ronljohnsonjr@gmail.co=
m> wrote:

> (This might be a bash question instead of a PG question, or it might be a=
n
> A/B question.)
>
> I need to process table records in a bash script.  Currently, I read them
> using a while loop and redirection.  The table isn't that big (30ish
> thousand rows), and performance is adequate, but am always looking for
> "better".
>
> Here's the current code:
> declare f1 f3 f8
> while IFS=3D'|' read f1 f3 f8; do
>     something f8 f3 f1
> done < <(psql -XAt -c "select f1, f3, f8 from some.table_name;")
>
> --
> Death to America, and butter sauce.
> Iraq lobster!
>

Another approach I've found more enjoyable in some cases,
is using select to_jsonb(...) on the resultset, and then pipe this to jq
<https://jqlang.github.io/jq/>.
This will preserve (more than) decent performance,
something you'll have to give up should you move to python for example.

You'll still have to use bash at some point,
but jq will be more expressive before that happens.
So instead of psql -> bash, you'll have to psql --> jq --> (bash).

--000000000000faffd00621f15518
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div><div dir=3D"ltr" class=3D"gmail_sign=
ature"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr">=
<p style=3D"font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:=
0pt"><br></p></div></div></div></div></div></div><br></div><br><div class=
=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Sep 12, 2024=
 at 6:08=E2=80=AFPM Ron Johnson &lt;<a href=3D"mailto:ronljohnsonjr@gmail.c=
om">ronljohnsonjr@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gm=
ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-l=
eft-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div d=
ir=3D"ltr">(This might be a bash question instead of a PG question, or it m=
ight be an A/B question.)<div><br></div><div>I need to process table record=
s in a bash script.=C2=A0 Currently, I read them using a while loop and red=
irection.=C2=A0 The table isn&#39;t that big (30ish thousand rows), and per=
formance is adequate, but am always looking for &quot;better&quot;.</div><d=
iv><br></div><div>Here&#39;s the current code:</div><div><font face=3D"mono=
space">declare f1 f3 f8<br>while IFS=3D&#39;|&#39; read f1 f3 f8; do<br>=C2=
=A0 =C2=A0 something f8 f3 f1<br>done &lt; &lt;(psql -XAt -c &quot;select f=
1, f3, f8 from some.table_name;&quot;)<br clear=3D"all"></font><div><br></d=
iv><span class=3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" cl=
ass=3D"gmail_signature"><div dir=3D"ltr">Death to America, and butter sauce=
.<div>Iraq lobster!</div></div></div></div></div></blockquote><div><br></di=
v><div>Another approach I&#39;ve found more enjoyable in some cases,</div><=
div>is using select to_jsonb(...) on the resultset, and then pipe this to <=
a href=3D"https://jqlang.github.io/jq/">jq</a>.</div><div>This will preserv=
e (more than) decent performance,</div><div>something you&#39;ll have to gi=
ve up should you move to python for example.</div><div><br></div><div>You&#=
39;ll still have to use bash at some point,</div><div>but jq will be more e=
xpressive before that happens.</div><div>So instead of psql -&gt; bash, you=
&#39;ll have to psql --&gt; jq --&gt; (bash).</div><div><br></div></div></d=
iv>

--000000000000faffd00621f15518--