+ foreach_ptr(TargetEntry, te, defineClause)
+ (void) coerce_to_boolean(pstate, (Node *) te->expr, "DEFINE");

Isn't this incorrect? I think it should update te->expr, as currently
it is possible to construct queries where this produces unexpected
results.

Good catch, Zsolt. You're right — the return value of
coerce_to_boolean() must be assigned back to te->expr, otherwise
any implicit cast (e.g., a user-defined type with an assignment
cast to boolean) is silently discarded.

The fix is straightforward:

    foreach_ptr(TargetEntry, te, defineClause)
        te->expr = (Expr *) coerce_to_boolean(pstate, (Node *) te->expr, "DEFINE");

I've confirmed that your example query produces incorrect results
without the fix (the truthyint value is evaluated as-is without
the cast) and correct results with it.

Patch 13 is attached with the fix and a regression test case based
on your example.

By the way, thank you for joining the TDE hooks discussion when
I was just getting started as a contributor — that meant a lot.
Great work on pg_tde at Percona, and rooting for the extensible
SMGR effort as well. Hope it's all going well!

Tatsuo, please review when you get a chance.

Regards,
Henson