Two PCs exchanging changes through a folder must be able to name the SAME
contact in both logs. "the QSO with M0ABC at 14:32" is a guess and the two
machines can disagree about which row that is, so an edit or a deletion cannot
be addressed at all without an identity that travels with the record.
A real indexed column, not a key inside extras_json: the identity is resolved
once per incoming change, and scanning JSON for it would turn every sync into a
full read of a 120 000-QSO logbook. A column costs one migration; the JSON would
cost a scan every time. That was the decision to confirm, and it is confirmed.
It follows the award_refs pattern — read in selectCols, absent from columnList,
written only through its own methods. So an ordinary edit cannot clobber it,
which matters: another machine addresses the contact by that id, and losing it
makes the same QSO arrive again as a new one. Pinned by a test that saves an
edit with the field deliberately blanked.
Existing contacts are stamped in batches inside one transaction rather than one
commit each — on a remote MySQL the round trip dominates, and 120 000 commits is
the difference between a minute and an afternoon. And a dedupe-key map lets two
machines that already hold the same imported log recognise each other's contacts
instead of copying 120 000 of them across.
MySQL nearly lost its logbook to this. It cannot index a TEXT column without a
prefix length: the migration would fail with error 1170 and fail again on every
startup, with no way out from the interface. The translator only emits VARCHAR
for names listed by hand in varcharColumns. sync_uid is now listed — and a test
reads the migrations, works out which indexed columns are TEXT, and fails if the
list does not cover them, so the next one cannot reach a shared logbook.
Translating the schema correctly does nothing for a database that already
exists. This operator's had been created by an earlier attempt, so qso.state was
already TEXT and every launch died on the same statement — CREATE INDEX … ON
qso(state), error 1170 — with the connection failing at startup and no way
forward from the UI.
Columns that MUST be VARCHAR (the indexed and keyed ones) are now converted
before the migrations run. It reads information_schema, touches only the columns
in varcharColumns, and only when they are actually TEXT — a no-op on a healthy
database, and it preserves each column's NULL/NOT NULL.
A failure to repair IS reported: it is the difference between a logbook that
opens and one that does not. A failure to READ information_schema is not — a
server that will not answer it can still be running a perfectly good schema, and
refusing to open would turn a non-problem into an outage.
The quoting fix was necessary but not sufficient: "column 'op_name' can't have a
default value" persisted.
The translation worked line by line and rewrote only the FIRST column on each
line. Our own migrations put one column per line, so nothing showed there — but
sqlite_master stores a table's CREATE statement with every ALTER-added column
appended to the SAME line, and the baseline is built from exactly that. On a
logbook that had been through upgrades, most columns therefore kept their TEXT
type and MySQL rejected the schema.
Two consequences fixed together: every match is now rewritten, and each one is
decided from ITS OWN declaration — the text between the surrounding commas —
rather than from the whole line. Deciding per line would have made every column
sharing a line with a default-bearing one into VARCHAR(255), which breaches the
InnoDB row-size limit the TEXT rule exists to respect.
Tests cover the many-columns-on-one-line shape and pin that a neighbour's DEFAULT
does not leak.
Two errors from a shared MySQL logbook, both fatal to the schema and neither
naming its cause:
CREATE INDEX … ON qso(state) → 1170 BLOB/TEXT column used in key
specification without a key length
op_name TEXT … DEFAULT '' → 1101 TEXT column can't have a default value
One root cause. TEXT columns are rewritten to VARCHAR(255) when they are indexed
or carry a default — that is exactly what those two MySQL rules require — but the
pattern only matched a BARE column name. The SQLite baseline dump quotes its
identifiers, and mysqlDDL turns those quotes into backticks before the TEXT rules
run, so every column coming from the baseline kept its TEXT type and MySQL
rejected the schema.
The operator saw none of this: just "MySQL is enabled but the connection failed
at startup" and a silent fallback to local SQLite — which is the worst shape for
a shared logbook, because each operator then logs into their own copy.
The pattern now accepts a quoted or bare name and puts the quoting back. A test
covers both forms, plus the case that must NOT change: an unindexed column stays
TEXT, since VARCHAR everywhere would break MySQL's row-size limit. Reverting the
one-line change makes it fail.
An operator reported migrations being very slow and sent a log with three
interleaved runs — and no way to tell one database migrated three times from
three databases migrated once. Each line now carries its target: the SQLite file
name, mysql:<database>, or baseline:memory for the in-memory pass that derives
the schema for a FRESH MySQL database (that one is fast and is not a real
database — in the report it sat between two slow runs and looked like a third).
A pass also announces how many migrations it will apply, so a run that applies
nothing is visibly distinct from one that does.
A QSO with a non-Latin character (Cyrillic Я = \xD0\xAF, Polish ł, …) failed to
save on the shared MySQL backend with "Error 1366: Incorrect string value …
for column 'qso'.'address'". Cause: the database had been pre-created by a
hosting panel (o2switch/cPanel) as latin1, so OpsLog's CREATE DATABASE IF NOT
EXISTS … utf8mb4 was a no-op and every table inherited latin1 — which can't
store those letters, even over a utf8mb4 connection.
OpenMySQL now runs ensureMySQLUTF8MB4 after migrations: ALTER DATABASE to
utf8mb4 (for future tables) + CONVERT any table with a non-utf8mb4 text column
via an information_schema probe. Idempotent and cheap on a healthy DB (the probe
returns nothing → no ALTERs); best-effort so a charset issue never blocks
logbook access. SQLite is unaffected.
rouggy