Mirror frappe/erpnext#56655 for the Postgres CI. Run the
bootstrap_test_data module in the setup job while Postgres is still up, so
the BootStrapTestData records are baked into the PGDATA artifact every shard
hydrates from — the shards start on already-warmed data instead of each
building it.
Unlike the MariaDB step, no `su -m` wrapper: the Postgres CI is
GitHub-hosted ubuntu-latest running as the runner user directly, matching
its own "Run Tests" step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Greptile flagged that `gh release download` with `github.token` could be
rejected for fork pull requests (token scoped to the fork, asset in
frappe/erpnext). The release is public and published, so the asset is
downloadable anonymously from objects.githubusercontent.com — drop the token
and curl the public URL directly. Removes the cross-repo token dependency and
keeps fork PRs working. Cloudflare is still bypassed since GitHub serves the
asset, not frappe.io.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The "Stop DB and stage datadir" step swallowed a failed `pg_ctl -m fast -w
stop` with `|| true`, then moved and tarred the PGDATA regardless. A stop
that times out or errors would bake a still-running, crash-inconsistent
cluster into the artifact every test shard consumes — and with
full_page_writes off, crash recovery can't repair torn pages. Drop the
`|| true` so a failed stop fails the job, mirroring the MariaDB sister's
"don't bake a dirty datadir" guard.
Also drop the redundant `ALTER SYSTEM SET fsync/synchronous_commit/
full_page_writes = off` block from install.sh. Its comment claimed the
postgres workflow "runs a service-container DB and never calls start-db.sh",
but it does call start-db.sh, which already applies those flags via `-o` on
every postgres start (setup job and each shard). The block was a no-op and
its justification was factually wrong.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Patch Test job intermittently failed on the "Download erpnext v14 backup"
step with HTTP 403 Forbidden: frappe.io sits behind Cloudflare, and wget's
default User-Agent gets flagged by bot protection on cache misses. This caused
random failures across PRs that only a re-run would clear.
Pull the fixed baseline from the v14-baseline GitHub release using the built-in
token instead. Release assets are served from GitHub's CDN and authenticated
from the runner, so no rate-limit roulette.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Measured A/B on the self-hosted setup: warm-bench restore 85s vs full
bench init 82s — no gain (slightly slower). bench init is already fast
because the uv/pip caches are mounted warm, so the cache only replaces a
~13s init while adding a ~200MB untar and still running bench build.
Drop BENCH_CACHE_DIR so warm-bench stays inert (the helper functions
remain, matching develop's install.sh).
- Wire up warm-bench: set BENCH_CACHE_DIR on the setup job so the bench-base
cache actually activates (was inert with no dir set, so every run did a
full bench init). Lives on the node-local bench-staging hostPath; any
miss/failure still falls back to a full init.
- run_ci_step: capture the timeout exit code under `set -e` (the previous
`timeout ...` + `ec=$?` aborted at the timeout line on failure, skipping
::endgroup:: and the exit-code return).
- Raise the per-step timeout 600s -> 1800s so a contended reinstall isn't
killed before the 40-min job timeout.
- Propagate DB through the su re-exec in start-db.sh / hydrate.sh so a
DB=postgres invocation can't silently fall back to the mariadb branch.
- Simplify the coverage job `if` to the equivalent plain non-PR gate.
The fan-out moved fsync/synchronous_commit/full_page_writes=off into
start-db.sh startup flags, but the Postgres workflow runs a postgres:13.3
service container and calls install.sh directly — it never runs start-db.sh.
So those flags never reached the Postgres CI, regressing it to full durability
on a commit-heavy suite. Re-apply them via ALTER SYSTEM (reloadable) in the
DB == "postgres" path, where the service-container workflow executes. MariaDB
is unaffected (DB != postgres).
Remove the `|| 'develop'` fallback on FRAPPE_BRANCH so install.sh resolves the
frappe framework branch from the git context (GITHUB_BASE_REF/GITHUB_REF), the
same as the Postgres workflow. This makes the workflow backportable unchanged:
on version-15-hotfix / version-16-hotfix it now clones the matching frappe
branch instead of develop.
Adds the division-by-zero divergence class to the PG-compat review tooling:
on a divisor that the data can drive to 0 (e.g. Sum(a)/Sum(b)), MariaDB
returns NULL for division by zero while PostgreSQL raises `division by zero`
and aborts the query. The portable fix is to wrap the divisor in
NullIf(divisor, 0), which yields NULL on both engines (matching MariaDB).
- .greptile/config.json: add it to the "would ERROR on PostgreSQL" list.
- .github/POSTGRES_COMPATIBILITY.md: document it under §1 (hard breaks).
- .github/helper/postgres_compat.py: note it in the docstring as a
deliberately-not-statically-checked semantic divergence (data-dependent,
like integer-division intent), so it stays a reviewer/Greptile concern.
Tooling-only; no source query changes. The instance fix shipped in #56361.
ci(postgres): teach the guard about COALESCE(date, int) type mismatch
New class found by the whole-repo audit (the Asset Depreciations report fix in this PR): IfNull/Coalesce of a typed column with a different-typed literal -- e.g. IfNull(date_col, 0) -> COALESCE(date, integer), which PostgreSQL rejects (DatatypeMismatch). Added to the Greptile config and POSTGRES_COMPATIBILITY.md (not statically checkable without column types).
The [^)]* span stopped at the first inner ')', so CAST(ABS(col) AS CHAR) slipped through. Use a non-greedy .+? with re.S; still zero production false positives (verified). Addresses review feedback.
The whole-repo MariaDB<->PostgreSQL audits surfaced classes the checker and
review guide did not yet cover. Add them:
Static checker (.github/helper/postgres_compat.py) - new mechanical breaks:
- .rlike() / raw RLIKE: frappe rewrites REGEXP->~* on Postgres but NOT RLIKE.
- Cast(x, "char") / raw CAST AS CHAR: bare CHAR is character(1) on Postgres
and truncates multi-digit values; use "varchar".
(Both flag zero production code; the only repo hit is in patches/, which the
hook already excludes.)
Greptile config + POSTGRES_COMPATIBILITY.md - new semantic/hard classes:
- aggregate (Sum/Count) selected next to bare columns with no GROUP BY at all.
- .like()/LIKE on a non-text column (bigint ILIKE) -> Cast_ to varchar.
- get_all(fields=["CapitalCase"]) identifier-case (extends the get_value case).
- bool into a Check column via qb.update().set() (extends set_value/db_set).
- int/int division: float a literal (col/1440 -> col/1440.0).
- Concat over a nullable column leaking a bare prefix on Postgres.
- clarify REGEXP/.regexp() is translated but RLIKE/.rlike() is not.
frappe's db_query SILENTLY drops ORDER BY for distinct queries on Postgres (the ORDER BY
column must appear in the SELECT-DISTINCT list), so `get_all/get_list(distinct=True,
order_by="<col>")` is a no-op there and the result comes back unordered — the root cause of
the Sales Register, Purchase Register and Sales Analytics ordering fixes. Add an AST rule to
.github/helper/postgres_compat.py that flags this (literal order_by only; an empty order_by=""
suppression and a dynamic/variable order_by are not flagged). `# pg-ok` escape hatch as usual.
Grandfather the three pre-existing low-impact sites the rule surfaces (paging/iteration order
only, not data): job_card operation autocomplete, inventory_dimension config list, and a
work_order test loop.
The Patch Test starts the full bench (incl. workers) and then runs migrate.
Migrate enqueues orphan-link cleanup jobs (delete_dynamic_links) that the
workers pick up and process while migrate is altering tables, which
intermittently fails with MySQL 1412 'Table definition has changed, please
retry transaction'.
Start every bench process except the workers during migrate, so nothing
consumes the queue mid-migrate. Redis and the other services stay up; the
queued jobs just wait.
Address Greptile review:
- customContext.files scope was **/*.py only, so Query Report SQL in .js/.sql/report .json
files didn't get the guide attached as context (the global instructions still applied).
Widen to .py/.js/.sql/report **/*.json.
- The guide's HAVING-alias rule said "with no GROUP BY"; PostgreSQL rejects a SELECT-alias in
HAVING regardless of GROUP BY. Reworded to match (repeat the expression, or move a
non-aggregate predicate to WHERE).
The PostgreSQL server-test job is label-gated, so until it is required the Greptile
PR-review bot is the always-on guard against cross-engine breaks. Extend
.greptile/config.json with `instructions` (and a `customContext` reference to a new
guide) so every review flags new/changed queries that would error on PostgreSQL or
silently diverge from MariaDB, under the prime rule that MariaDB output must not change.
- .github/POSTGRES_COMPATIBILITY.md — the catalogue the bot (and contributors) follow:
hard breaks (loose GROUP BY, MySQL-only funcs, UPDATE..JOIN, HAVING-on-alias,
DISTINCT+ORDER BY, single-quoted alias, varchar bitwise OR, capital identifiers,
set_value(Check,bool)), silent divergences (text case-sensitivity, name-lookup case,
empty-string↔NULL, NULL ordering, ORDER BY..LIMIT 1 tiebreakers, integer division,
distinct-drops-ORDER-BY-on-PG + casefold sorting, function-rewrite parity, UnixTimestamp
TZ), the GROUP BY row-count trap (Max()-wrap vs add-to-GROUP-BY; FD-by-source-table),
the InFailedSqlTransaction/savepoint rule, and the false positives NOT to flag
(.like→ILIKE, ifnull/backtick/LOCATE/REGEXP auto-translation, MariaDB-changing tiebreakers).
- Existing disabledLabels and frappe/frappe context are preserved.
- semgrep: annotate the source-reading open() with # nosemgrep for the
frappe-security-file-traversal rule (dev-only lint tool; path comes from pre-commit,
not user input).
- bool-scan: only inspect the field *value* arg (db_set args[1]/dict args[0];
set_value args[3]/dict args[2]) so a positional update_modified=False
(e.g. db_set('f', 0, False)) no longer false-positives.
- # pg-ok: also honour the annotation on a multi-line call's closing paren line
(scan one line past the node's end).
The Postgres test job is label-gated, so it does not run on every PR. This adds an
always-on pre-commit hook that statically flags the *mechanical* breaks: MySQL-only
functions (timestamp(date,time), timediff, str_to_date, date_format/add/sub,
group_concat, period_diff, SQL IF()), SHOW INDEX/TABLES/COLUMNS, single-quoted
aliases, UPDATE..JOIN, interpolated/f-string SQL carrying MySQL-isms,
set_value/db_set(<Check>, bool), and MySQL SHOW INDEX result keys.
It deliberately does NOT flag the framework auto-translations (ifnull->coalesce,
backtick/locate/REGEXP, .like()->ILIKE) nor the *semantic* divergences (loose GROUP
BY, case-sensitive ==/IN, NULL ordering, tiebreakers) — those need the test suite,
which remains the backstop. AST + structure-gated regex keep false positives near
zero (docstrings and prose skipped); '# pg-ok' exempts intentional MariaDB-only
branches. Scoped to erpnext/ excluding patches/. Includes a unit test of the checker.
The Postgres CI site only listed erpnext in install_apps, so the payments app
(fetched and built by install.sh via 'bench get-app payments') was never
installed on the site — leaving 'tabPayment Gateway' absent. test_payment_request
(and other payment-gateway-dependent tests) then errored on Postgres with
'relation "tabPayment Gateway" does not exist', while MariaDB passed because its
site_config already lists ["payments", "erpnext"]. Match that ordering for parity.
Postgres fsyncs on every commit by default, which dominates a commit-heavy test suite.
Turn off synchronous_commit/fsync/full_page_writes on the throwaway CI database (reload-
time settings, no restart). MariaDB CI is unaffected (DB != postgres).
The MariaDB job is named 'Python Unit Tests', and 'Python Unit Tests (1..4)' are the
required status checks on develop. Naming the Postgres matrix job the same made its
checks report under those required contexts, effectively gating every (labelled) PR on
Postgres. Rename it to 'Postgres Unit Tests' so its contexts are distinct and the
workflow stays non-required until we deliberately add it to branch protection.
Bring the Server (Postgres) workflow in line with Server (MariaDB) internals while
keeping it opt-in for now: pull_request runs still require the 'postgres' label, but the
job now uses the full 4-container matrix (was 1), adds the nightly schedule /
workflow_dispatch / repository_dispatch triggers (which always run), and uploads
coverage. Builds ERPNext against frappe `develop` (PostgreSQL query-builder/ORM support
is merged there), so no fork override is needed.
The ERPNext server suite now passes on PostgreSQL and MariaDB from a single codebase;
flipping this to run on every PR / become a required check is a later, separate step.
