ci(postgres): teach the PG-compat tooling the audit 6-9 divergence classes

The whole-repo MariaDB<->PostgreSQL audits surfaced classes the checker and
review guide did not yet cover. Add them:

Static checker (.github/helper/postgres_compat.py) - new mechanical breaks:
- .rlike() / raw RLIKE: frappe rewrites REGEXP->~* on Postgres but NOT RLIKE.
- Cast(x, "char") / raw CAST AS CHAR: bare CHAR is character(1) on Postgres
  and truncates multi-digit values; use "varchar".
(Both flag zero production code; the only repo hit is in patches/, which the
hook already excludes.)

Greptile config + POSTGRES_COMPATIBILITY.md - new semantic/hard classes:
- aggregate (Sum/Count) selected next to bare columns with no GROUP BY at all.
- .like()/LIKE on a non-text column (bigint ILIKE) -> Cast_ to varchar.
- get_all(fields=["CapitalCase"]) identifier-case (extends the get_value case).
- bool into a Check column via qb.update().set() (extends set_value/db_set).
- int/int division: float a literal (col/1440 -> col/1440.0).
- Concat over a nullable column leaking a bare prefix on Postgres.
- clarify REGEXP/.regexp() is translated but RLIKE/.rlike() is not.
This commit is contained in:
Mihir Kandoi
2026-06-23 12:22:22 +05:30
parent ca908b69cf
commit fc9544435e
3 changed files with 53 additions and 13 deletions

View File

@@ -31,9 +31,12 @@ Flag a changed query that uses any of these:
- **Loose `GROUP BY`** — selecting/ordering a column that is neither in `GROUP BY` nor wrapped
in an aggregate. MariaDB tolerates it; PostgreSQL errors (`must appear in the GROUP BY
clause or be used in an aggregate function`). Fix: add it to `GROUP BY` **if it is
functionally dependent on the group key**, otherwise wrap it in `Max()`/`Min()`. **See §3 —
the row-count trap — before suggesting "add it to GROUP BY".**
clause or be used in an aggregate function`). This **also covers an aggregate (`Sum`/`Count`/…)
selected alongside bare columns with NO `.groupby()` at all** — MariaDB silently collapses
every row into one arbitrary-valued row (often a *wrong-output* bug there too), PostgreSQL
errors. Fix: add the bare column to `GROUP BY` **if it is functionally dependent on the group
key**, otherwise wrap it in `Max()`/`Min()`. **See §3 — the row-count trap — before suggesting
"add it to GROUP BY".**
- **MySQL-only functions** — `TIMESTAMP(date,time)`, `TIMEDIFF`, `STR_TO_DATE`, `DATE_FORMAT`,
`DATE_ADD/SUB`, `GROUP_CONCAT`, `PERIOD_DIFF`, SQL `IF(cond,a,b)`. Use the portable
`frappe.query_builder.functions` equivalents (`CombineDatetime`, `DateDiff`, `Case`,
@@ -47,12 +50,23 @@ Flag a changed query that uses any of these:
unquoted (or double-quoted) alias.
- **`varchar | varchar`** (bitwise OR misused as a coalesce) — errors on PostgreSQL. Use
`Coalesce(...)`.
- **Capital-cased identifiers** used as column/field names in `get_value(dt, dn, "Status")` and
similar — PostgreSQL folds unquoted identifiers to lower case; a stored column named
`status` won't match `"Status"`. Use the exact stored case.
- **Capital-cased identifiers** used as column/field names in `get_value(dt, dn, "Status")`,
`get_all(dt, fields=["Account"])`, and similar — PostgreSQL quotes the identifier and matches
it case-sensitively; a stored column named `status`/`account` won't match `"Status"`/`"Account"`
(`column "Account" does not exist`). Use the exact stored (lower-case) fieldname.
- **Boolean passed where an integer column is expected** — `frappe.db.set_value(dt, dn,
check_field, True)` emits `SET col = true`, which PostgreSQL rejects on a `smallint`
(`DatatypeMismatch`). Pass `1`/`0`.
check_field, True)`, `doc.db_set(field, False)`, or `frappe.qb.update(dt).set(check_field, True)`
emit `SET col = true`, which PostgreSQL rejects on a `smallint`/`Check` column
(`column is of type smallint but expression is of type boolean`). Pass `1`/`0`.
- **`.like()`/`.ilike()` (or raw `LIKE`) on a NON-text column** — `idx`, `docstatus`, a date, etc.
frappe maps `.like()` → `ILIKE`, and PostgreSQL has no `bigint ILIKE text` operator (`operator
does not exist: bigint ~~* unknown`). Cast the column to text first — **`Cast_(col, "varchar")`**,
not `Cast(col, "char")` (see below). MariaDB coerces the int implicitly, so the cast is a no-op there.
- **`CAST(… AS CHAR)` / `Cast(x, "char")`** — on PostgreSQL bare `CHAR` is `character(1)`, so
`CAST(12 AS CHAR)` → `'1'` (silently truncates multi-digit values); MariaDB gives the full string.
Use `VARCHAR` / `Cast_(x, "varchar")`.
- **`.rlike()` / raw `RLIKE`** — frappe rewrites `REGEXP` → `~*` on PostgreSQL but does **not**
translate `RLIKE` (no such PostgreSQL operator). Use `.regexp()` (or `.like()` for a simple prefix).
---
@@ -75,8 +89,11 @@ These don't error, so a one-engine CI stays green. Flag them:
- **`ORDER BY … LIMIT 1` with no unique tiebreaker** — when rows tie on the ordered column the
two engines may pick different rows. Add a `creation`/`name` tiebreaker **only if it does not
change MariaDB's current pick** (see §4).
- **Integer division** — `COUNT(...) / COUNT(...) * 100` truncates to `0` on PostgreSQL
(integer/integer) but is decimal on MariaDB. Multiply by `100.0` first.
- **Integer division** — `int / int` truncates on PostgreSQL but is decimal on MariaDB, e.g.
`COUNT(...) / COUNT(...) * 100` → `0`, or `manufacturing_time_in_mins / 1440` flooring a
lead-time to whole days. Force float: multiply by `100.0`, or make a literal a float
(`/ 1440` → `/ 1440.0`), or cast an operand. (Only SQL-level `/` on integer **columns/literals**
— Python `/` is already float.)
- **`DISTINCT` list ordering** — `frappe.get_all(distinct=True, order_by=…)` /
`SELECT DISTINCT … ORDER BY`: frappe's `db_query` **silently drops `ORDER BY` for distinct
queries on PostgreSQL**, so the result is unordered there. Sort in Python instead — and use
@@ -119,9 +136,12 @@ scope for a portability fix.
These are auto-handled by the framework and are **not** breaks:
- **`.like()` / `["like", …]`** already renders as `ILIKE` on PostgreSQL — not a
case-sensitivity bug.
case-sensitivity bug. *(Exception: `.like()` on a **non-text** column — `idx`, `docstatus` —
is a hard break, `bigint ILIKE`; see §1.)*
- **Raw `ifnull(...)`** inside `frappe.db.sql()` is rewritten to `coalesce(...)` on all engines.
- **Backticks**, **`LOCATE`**, **`REGEXP`** in raw SQL are auto-translated on PostgreSQL.
- **Backticks**, **`LOCATE`**, **`REGEXP`** / **`.regexp()`** in raw SQL are auto-translated on
PostgreSQL (`REGEXP` → `~*`). **But `RLIKE` / `.rlike()` is NOT translated** — that one is a
hard break (see §1).
- **An `ORDER BY … LIMIT 1` tie where the two engines already agree**, or where adding a
tiebreaker would *change* MariaDB's current pick — leave it; "fixing" it would either change
MariaDB or has no observable effect.

View File

@@ -62,6 +62,10 @@ SQL_PATTERNS: list[tuple[re.Pattern, str]] = [
"single-quoted column alias breaks on Postgres -> use a bare or double-quoted alias"),
(re.compile(r"\bif\s*\(", re.I),
"SQL IF() is MySQL-only -> use CASE WHEN ... THEN ... ELSE ... END (frappe.qb.Case())"),
(re.compile(r"\brlike\b", re.I),
"RLIKE is MySQL-only -> frappe rewrites REGEXP->~* on Postgres but NOT RLIKE; use REGEXP / .regexp() / ~"),
(re.compile(r"\bcast\s*\([^)]*\bas\s+char\b", re.I),
"CAST(... AS CHAR) is character(1) on Postgres and truncates -> CAST AS VARCHAR (frappe Cast_(x, 'varchar'))"),
]
# UPDATE ... JOIN: both keywords in the same SQL string.
@@ -71,6 +75,10 @@ MYSQL_RESULT_KEYS = {"Column_name", "Key_name", "Seq_in_index", "Non_unique", "I
SET_BOOL_FUNCS = {"set_value", "db_set"}
# query-builder cast helpers: pypika Cast / frappe Cast_. A "char" target type is character(1)
# on Postgres (truncates); "varchar" is the full-length cast.
CAST_FUNCS = {"Cast", "Cast_"}
# frappe.get_all / get_list: frappe's db_query SILENTLY drops ORDER BY for `distinct` queries on
# Postgres (the ORDER BY column must appear in the SELECT-DISTINCT list), so `distinct=True` together
# with a literal `order_by` is a no-op on PG and the result comes back unordered.
@@ -178,6 +186,18 @@ class Visitor(ast.NodeVisitor):
if has_distinct and has_literal_order:
self._flag(node, f"{name}(distinct=True, order_by=...) -> frappe drops ORDER BY for distinct queries on Postgres; sort in python instead, e.g. sorted(..., key=str.casefold)")
# query-builder .rlike(...): pypika emits the MySQL-only RLIKE operator, which frappe does
# NOT translate for Postgres (it rewrites only REGEXP -> ~*).
if name == "rlike":
self._flag(node, ".rlike() emits MySQL-only RLIKE (not translated on Postgres) -> use .regexp() (rewritten to ~*) or .like()")
# Cast(col, "char") / Cast_(col, "char"): on Postgres a bare CHAR is character(1) and truncates
# (e.g. CAST(12 AS CHAR) -> '1'); use "varchar" for a full-length string cast.
if name in CAST_FUNCS:
for arg in (*node.args, *(kw.value for kw in node.keywords)):
if isinstance(arg, ast.Constant) and isinstance(arg.value, str) and arg.value.strip().lower() == "char":
self._flag(node, f"{name}(..., 'char') is character(1) on Postgres and truncates -> use 'varchar'")
self.generic_visit(node)
def visit_Subscript(self, node: ast.Subscript) -> None: