Correcting an over-design of my own. I had contacts given an identity in bulk
the first time synchronisation was switched on, and a full dedupe-key map of the
logbook rebuilt on every pass, both sized for 123 000 contacts.
Neither is needed, because the change log starts EMPTY. Contacts logged before
synchronisation was switched on are never in anyone's log and so are never
exchanged; an identity is stamped only on a contact that is actually logged,
edited or deleted from then on. Seeding a second machine with the existing
history is a one-time copy of the database or an ADIF import — not something
synchronisation should be doing, and not something it can do from a file that
starts empty.
One case survives, and it is why the dedupe key stays. Two machines can already
hold the SAME old contact, the second seeded by that copy or import, with
different row ids and no identity on either. The day one of them edits it, it
stamps an identity and sends a change naming it; the other has never seen that
identity and would insert a duplicate. Matching on the contact itself —
callsign, minute, band, mode, the importer's own key — recognises it.
So that became a targeted lookup through idx_qso_callsign, run only when an
identity is unknown, instead of a whole-table map rebuilt each pass. Rare work,
priced as rare work.
Two PCs exchanging changes through a folder must be able to name the SAME
contact in both logs. "the QSO with M0ABC at 14:32" is a guess and the two
machines can disagree about which row that is, so an edit or a deletion cannot
be addressed at all without an identity that travels with the record.
A real indexed column, not a key inside extras_json: the identity is resolved
once per incoming change, and scanning JSON for it would turn every sync into a
full read of a 120 000-QSO logbook. A column costs one migration; the JSON would
cost a scan every time. That was the decision to confirm, and it is confirmed.
It follows the award_refs pattern — read in selectCols, absent from columnList,
written only through its own methods. So an ordinary edit cannot clobber it,
which matters: another machine addresses the contact by that id, and losing it
makes the same QSO arrive again as a new one. Pinned by a test that saves an
edit with the field deliberately blanked.
Existing contacts are stamped in batches inside one transaction rather than one
commit each — on a remote MySQL the round trip dominates, and 120 000 commits is
the difference between a minute and an afternoon. And a dedupe-key map lets two
machines that already hold the same imported log recognise each other's contacts
instead of copying 120 000 of them across.
MySQL nearly lost its logbook to this. It cannot index a TEXT column without a
prefix length: the migration would fail with error 1170 and fail again on every
startup, with no way out from the interface. The translator only emits VARCHAR
for names listed by hand in varcharColumns. sync_uid is now listed — and a test
reads the migrations, works out which indexed columns are TEXT, and fails if the
list does not cover them, so the next one cannot reach a shared logbook.
rouggy