feat(cluster): grids from the UDP decodes, plus spotter continent and LoTW

Backend groundwork; the columns and filters that consume it come next.

GRIDS. A CQ is the one WSJT-X message that carries a locator, and wsjtSender was
throwing that token away. It is now returned, validated as a real field+square,
and remembered per callsign. This is the ONLY grid source available: a DX-cluster
line carries the spotter's grid at best and never the DX's, and a per-callsign
QRZ lookup under an RBN firehose is not a trade worth making. So grids are known
for the stations this receiver decoded - which is exactly the FT8/FT4 watering
hole an operator is looking at while grid chasing.

RR73 is why the grid is validated rather than pattern-matched. R is inside A-R
and 73 inside 00-99, so a sign-off satisfies the Maidenhead shape exactly and
would have planted a grid that does not exist into the index, silently.

NEW GRID keys on "GRID|MODE" with the mode put through the same normMode as
everything else, so the "group digital modes" option decides whether a grid
worked on FT8 is still new on FT4 - one rule, no branch. Grids are truncated to
four characters: a log holds a mix of JN36 and JN36QU, and without that the same
square is new forever, once per subsquare.

On cost, which was the condition: one more DISTINCT scan when the status
snapshot is rebuilt, then map lookups per spot. The same shape as the county and
POTA sets it sits beside, and the snapshot exists precisely so a spot batch never
touches the logbook.

Spotter continent and the LoTW flag come from tables already in memory - the
DXCC prefix table and ARRL's user list - so they cost a lookup each. The spotter
continent answers a different question from the DX's: whether anyone near you is
hearing this at all.
This commit is contained in:
2026-08-10 16:52:36 +02:00
parent 0a8c1ac45f
commit 9b2115be8f
6 changed files with 204 additions and 41 deletions
+80
View File
@@ -595,6 +595,18 @@ type App struct {
// or when a setting that shapes the maps flips.
clusterStatusIdx *clusterStatusCache
clusterStatusMu sync.Mutex
// decodeGrids maps a callsign to the 4-character grid it announced in a CQ
// heard over the WSJT-X UDP link. It is the ONLY source of grids we have for
// a spot: a DX-cluster line carries the spotter's grid at best, never the
// DX's, and a per-callsign QRZ lookup under an RBN firehose is out of the
// question. So grids are known for the stations this station's own receiver
// decoded — which is exactly the FT8/FT4 watering hole an operator is looking
// at when grid chasing.
//
// In memory only, and bounded: it is a session-local view of who is on the
// air now, not a database.
decodeGrids map[string]string
decodeGridsMu sync.RWMutex
// Self-spot throttle: when and on what frequency we last announced ourselves.
// Held in memory only — a restart legitimately re-announces the station.
selfSpotMu sync.Mutex
@@ -11769,6 +11781,19 @@ func (a *App) consumeUDPEvents() {
}
switch {
case ev.DecodeCall != "":
// Remember the grid before anything else: a CQ is the one message that
// carries it, and the station may never send another.
if ev.DecodeGrid != "" {
a.decodeGridsMu.Lock()
if a.decodeGrids == nil {
a.decodeGrids = make(map[string]string, 512)
}
if len(a.decodeGrids) > 20000 {
a.decodeGrids = make(map[string]string, 512) // bound a long session
}
a.decodeGrids[strings.ToUpper(ev.DecodeCall)] = ev.DecodeGrid
a.decodeGridsMu.Unlock()
}
// A WSJT-X decode (heard station). Render it on the FlexRadio
// panadapter when the option is on; green + SNR comment, auto-expiring
// after the configured duration. De-duped per call in the Flex backend.
@@ -16368,6 +16393,10 @@ type SpotQuery struct {
Band string `json:"band"`
Mode string `json:"mode"`
POTARef string `json:"pota_ref,omitempty"` // park id if the spot is a POTA activation
// Spotter is the station that sent the spot. Only its continent is wanted, and
// resolving it here rather than in the frontend keeps the one DXCC prefix
// table as the single authority on what continent a callsign is in.
Spotter string `json:"spotter,omitempty"`
}
// SpotStatus is the per-tuple result. Status is one of:
@@ -16400,6 +16429,21 @@ type SpotStatus struct {
County string `json:"county,omitempty"`
State string `json:"state,omitempty"`
NewPOTA bool `json:"new_pota"`
// Grid is the 4-character square this station announced in a CQ on the UDP
// link, and NewGrid says that square has never been worked. Both are empty /
// false for any station this receiver has not decoded — a DX-cluster line
// carries the SPOTTER's grid at best, never the DX's.
Grid string `json:"grid,omitempty"`
NewGrid bool `json:"new_grid"`
// SpotterContinent is the continent of the station that SENT the spot, not of
// the DX. It answers a different question — "is anyone near me hearing this?"
// — which is what makes it worth filtering on: a JA spot on 20 m tells a
// European very little about their own path.
SpotterContinent string `json:"spotter_continent,omitempty"`
// LoTW is true when the DX callsign appears in ARRL's user-activity list, so
// an operator chasing confirmations can skip the stations that will never
// upload. Inert until that list has been downloaded.
LoTW bool `json:"lotw"`
// NewPfx flags a CQ WPX prefix never worked before, and Pfx is that prefix.
// Also orthogonal: a common entity on a worked band can still carry a prefix
// that has never been in the log, which is exactly what a WPX chaser is
@@ -16427,6 +16471,7 @@ type clusterStatusCache struct {
workedCounties map[string]struct{}
workedPOTA map[string]struct{}
workedPfx map[string]struct{}
workedGrids map[string]struct{} // "GRID|MODE", mode normalised like the rest
normMode func(string) string // nil unless digital-mode grouping is on
groupDigital bool // settings the maps were built under —
sameSlot bool // a change rebuilds the snapshot
@@ -16493,6 +16538,10 @@ func (a *App) clusterStatusMaps() *clusterStatusCache {
// lookup) and worked POTA parks.
c.workedCounties, _ = a.qso.WorkedCountyKeys(a.ctx, award.USCountyKey)
c.workedPOTA, _ = a.qso.WorkedPOTARefs(a.ctx)
// One more DISTINCT scan when the snapshot is rebuilt, then pure map lookups
// per spot — the same shape as the county and POTA sets beside it, which is
// why grids cost nothing under an RBN firehose.
c.workedGrids, _ = a.qso.WorkedGridKeys(a.ctx, c.normMode)
// Worked WPX prefixes, derived from the callsigns we already loaded — no
// extra query. Derived rather than read from the stored PFX column: that
// column is only filled when an import supplied it, and deriving keeps this
@@ -16581,6 +16630,37 @@ func (a *App) ClusterSpotStatuses(spots []SpotQuery) []SpotStatus {
out[i].NewPOTA = true
}
}
// The spotter's continent, and whether the DX uploads to LoTW. Both are
// in-memory lookups on tables already loaded, so they add nothing per spot.
if a.dxcc != nil && q.Spotter != "" {
if m, ok := a.dxcc.Lookup(q.Spotter); ok {
out[i].SpotterContinent = m.Continent
}
}
if a.lotwUsers != nil {
out[i].LoTW = a.lotwUsers.Lookup(q.Call).IsUser
}
// NEW GRID: the square this station announced in a CQ we decoded. The mode
// is part of the key, so the "group digital modes" option decides whether a
// grid worked on FT8 still counts as new on FT4 — one rule, no branch here.
{
// The length check has to be INSIDE the lock: the decode goroutine
// replaces this map wholesale when it grows too large, so reading len()
// unguarded is a race on the map header, not a cheap fast path.
a.decodeGridsMu.RLock()
g := a.decodeGrids[strings.ToUpper(q.Call)]
a.decodeGridsMu.RUnlock()
if g != "" {
out[i].Grid = g
cm := out[i].Mode
if normMode != nil && cm != "" {
cm = normMode(cm)
}
if _, done := idx.workedGrids[g+"|"+cm]; !done {
out[i].NewGrid = true
}
}
}
// NEW COUNTY: resolve the callsign's home county from the offline ULS
// store (US only; inert until downloaded) and flag if never worked.
if a.uls != nil {
+10
View File
@@ -2987,6 +2987,7 @@ export namespace main {
band: string;
mode: string;
pota_ref?: string;
spotter?: string;
static createFrom(source: any = {}) {
return new SpotQuery(source);
@@ -2998,6 +2999,7 @@ export namespace main {
this.band = source["band"];
this.mode = source["mode"];
this.pota_ref = source["pota_ref"];
this.spotter = source["spotter"];
}
}
export class SpotStatus {
@@ -3012,6 +3014,10 @@ export namespace main {
county?: string;
state?: string;
new_pota: boolean;
grid?: string;
new_grid: boolean;
spotter_continent?: string;
lotw: boolean;
new_pfx: boolean;
pfx?: string;
worked_slot: boolean;
@@ -3033,6 +3039,10 @@ export namespace main {
this.county = source["county"];
this.state = source["state"];
this.new_pota = source["new_pota"];
this.grid = source["grid"];
this.new_grid = source["new_grid"];
this.spotter_continent = source["spotter_continent"];
this.lotw = source["lotw"];
this.new_pfx = source["new_pfx"];
this.pfx = source["pfx"];
this.worked_slot = source["worked_slot"];
+11 -9
View File
@@ -47,18 +47,19 @@ func reusingListenConfig() net.ListenConfig {
// Event is what a Server emits to its consumer for every parsed packet.
// At most one of the fields is populated per event.
type Event struct {
ConfigID int64
Service ServiceType
Source string // remote addr that sent the packet, for diagnostics
ConfigID int64
Service ServiceType
Source string // remote addr that sent the packet, for diagnostics
DXCall string // ServiceWSJT (Status) or ServiceRemoteCall
DXGrid string // ServiceWSJT (Status)
Mode string // ServiceWSJT (Status/Decode)
FreqHz int64 // ServiceWSJT (Status)
LoggedADIF string // ServiceWSJT (LoggedADIF), ServiceADIF or ServiceN1MM
DXCall string // ServiceWSJT (Status) or ServiceRemoteCall
DXGrid string // ServiceWSJT (Status)
Mode string // ServiceWSJT (Status/Decode)
FreqHz int64 // ServiceWSJT (Status)
LoggedADIF string // ServiceWSJT (LoggedADIF), ServiceADIF or ServiceN1MM
// A WSJT-X Decode (heard station) to render on the panadapter.
DecodeCall string // transmitting (DE) callsign
DecodeGrid string // 4-char grid, CQ decodes only
DecodeFreqHz int64 // RF frequency (dial + audio offset)
DecodeSNR int // reported SNR (dB)
DecodeCQ bool // the decode was a CQ
@@ -93,7 +94,7 @@ type Server struct {
// 50.400 panadapter. WSJT-X requires --rig-name for a second instance, so the
// id is distinct whenever there is more than one.
dialHz map[string]int64
lastDX string // WSJT: last non-empty DX Call seen, to detect a clear
lastDX string // WSJT: last non-empty DX Call seen, to detect a clear
// badPkts counts datagrams this listener could not parse, so the diagnostic
// dump below stays bounded. A misconfigured port is not a one-off: the
@@ -300,6 +301,7 @@ func (s *Server) handle(pkt []byte, remote *net.UDPAddr) {
return
}
ev.DecodeCall = w.DecodeCall
ev.DecodeGrid = w.DecodeGrid
ev.DecodeFreqHz = dial + w.DeltaFreqHz
ev.DecodeSNR = w.SNR
ev.DecodeCQ = w.IsCQ
+39 -16
View File
@@ -39,18 +39,19 @@ const (
// WSJTEvent is the parsed, typed result of decoding a single packet.
// One of (DXCall, LoggedADIF, DecodeCall) is non-empty depending on the message.
type WSJTEvent struct {
DXCall string // current "DX Call" field in the WSJT app (Status)
DXGrid string // optional grid for that call (Status)
Mode string // FT8 / FT4 / …
FreqHz int64 // current dial freq when available (Status)
LoggedADIF string // full ADIF text when message is LoggedADIF
ProgramID string // "WSJT-X" / "JTDX" / "MSHV" — for diagnostics / dedup
DXCall string // current "DX Call" field in the WSJT app (Status)
DXGrid string // optional grid for that call (Status)
Mode string // FT8 / FT4 / …
FreqHz int64 // current dial freq when available (Status)
LoggedADIF string // full ADIF text when message is LoggedADIF
ProgramID string // "WSJT-X" / "JTDX" / "MSHV" — for diagnostics / dedup
// Decode (type 2): the transmitting station heard on the band. FreqHz is NOT
// set here (Decode carries only the audio offset); the caller adds the last
// known dial frequency (from Status) to DeltaFreqHz to get the RF frequency.
IsDecode bool
DecodeCall string // the sender (DE) callsign extracted from the message text
DecodeGrid string // 4-char grid, CQ decodes only — the exchange carries none
DeltaFreqHz int64 // audio offset within the passband (Hz)
SNR int // reported signal-to-noise (dB)
IsCQ bool // the decode was a CQ call
@@ -239,13 +240,14 @@ func ParseWSJT(pkt []byte) (WSJTEvent, bool, error) {
if err != nil {
return WSJTEvent{}, false, err
}
call, isCQ := wsjtSender(msg)
call, isCQ, grid := wsjtSender(msg)
if call == "" {
return WSJTEvent{}, false, nil // free-text / telemetry / unparseable → ignore
}
ev.IsDecode = true
ev.DecodeCall = call
ev.IsCQ = isCQ
ev.DecodeGrid = grid
ev.DeltaFreqHz = int64(df)
ev.SNR = int(snr)
ev.Mode = strings.ToUpper(strings.TrimSpace(mode))
@@ -263,17 +265,20 @@ func ParseWSJT(pkt []byte) (WSJTEvent, bool, error) {
return WSJTEvent{}, false, nil
}
// wsjtSender extracts the transmitting (DE) callsign from a WSJT-X message and
// whether it was a CQ. Grammar:
// wsjtSender extracts the transmitting (DE) callsign from a WSJT-X message,
// whether it was a CQ, and the grid when the message carries one. Grammar:
//
// CQ [modifier] <de_call> [grid] → de_call, isCQ=true
// CQ [modifier] <de_call> [grid] → de_call, isCQ=true, grid
// <to_call> <de_call> [report|…] → de_call, isCQ=false
//
// Only a CQ carries a grid: the standard exchange puts a signal report in that
// third slot, never a locator.
//
// Returns "" for free-text / telemetry / hashed-call messages we can't resolve.
func wsjtSender(message string) (call string, isCQ bool) {
func wsjtSender(message string) (call string, isCQ bool, grid string) {
f := strings.Fields(strings.ToUpper(strings.TrimSpace(message)))
if len(f) == 0 {
return "", false
return "", false, ""
}
if f[0] == "CQ" {
// Skip an optional modifier after CQ (DX / a region like NA / a zone like
@@ -283,15 +288,33 @@ func wsjtSender(message string) (call string, isCQ bool) {
idx = 2
}
if idx < len(f) && looksLikeCall(f[idx]) {
return f[idx], true
if idx+1 < len(f) && isGridField(f[idx+1]) {
grid = f[idx+1]
}
return f[idx], true, grid
}
return "", true
return "", true, ""
}
// Standard exchange: the DE (sender) call is the second token.
if len(f) >= 2 && looksLikeCall(f[1]) {
return f[1], false
return f[1], false, ""
}
return "", false
return "", false, ""
}
// isGridField reports a 4-character Maidenhead field+square (JN36).
//
// RR73 is the reason this is not a bare pattern match: it is a sign-off, not a
// locator, yet R falls inside AR and 73 inside 0099, so it satisfies the
// Maidenhead shape exactly. WSJT-X never puts it in the slot after a CQ call,
// but a station sending "CQ RR73" style free text would silently plant a
// nonexistent grid in the log's grid index, and nothing downstream could tell.
func isGridField(s string) bool {
if len(s) != 4 || s == "RR73" {
return false
}
return s[0] >= 'A' && s[0] <= 'R' && s[1] >= 'A' && s[1] <= 'R' &&
s[2] >= '0' && s[2] <= '9' && s[3] >= '0' && s[3] <= '9'
}
// looksLikeCall is a loose callsign test: 312 chars of AZ/09//, with at least
+24 -16
View File
@@ -7,26 +7,34 @@ func TestWSJTSender(t *testing.T) {
msg string
wantCall string
wantCQ bool
wantGrid string
}{
{"CQ K1ABC FN42", "K1ABC", true},
{"CQ DX W2XYZ EM12", "W2XYZ", true}, // modifier "DX" skipped
{"CQ NA VE3ABC FN03", "VE3ABC", true}, // region modifier skipped
{"CQ 020 JA1XYZ PM95", "JA1XYZ", true},// zone modifier skipped
{"W2XYZ K1ABC -10", "K1ABC", false}, // exchange → sender is 2nd call
{"W2XYZ K1ABC R-10", "K1ABC", false},
{"W2XYZ K1ABC RR73", "K1ABC", false},
{"F4BPO K1ABC/P 73", "K1ABC/P", false},// portable call kept
{"CQ F/DL1ABC JO31", "F/DL1ABC", true},// compound prefix
{"CQ K1ABC FN42", "K1ABC", true, "FN42"},
{"CQ DX W2XYZ EM12", "W2XYZ", true, "EM12"}, // modifier "DX" skipped
{"CQ NA VE3ABC FN03", "VE3ABC", true, "FN03"}, // region modifier skipped
{"CQ 020 JA1XYZ PM95", "JA1XYZ", true, "PM95"}, // zone modifier skipped
{"W2XYZ K1ABC -10", "K1ABC", false, ""}, // exchange → sender is 2nd call
{"W2XYZ K1ABC R-10", "K1ABC", false, ""},
{"W2XYZ K1ABC RR73", "K1ABC", false, ""},
{"F4BPO K1ABC/P 73", "K1ABC/P", false, ""}, // portable call kept
{"CQ F/DL1ABC JO31", "F/DL1ABC", true, "JO31"}, // compound prefix, grid still valid
{"CQ K1ABC", "K1ABC", true, ""}, // CQ without a grid
{"CQ K1ABC RR73", "K1ABC", true, ""}, // RR73 is a sign-off, NOT grid RR73
{"CQ K1ABC FN4", "K1ABC", true, ""}, // 3 chars is not a field+square
{"CQ K1ABC FN42AB", "K1ABC", true, ""}, // 6-char: WSJT-X never sends it here
{"CQ K1ABC 73", "K1ABC", true, ""}, // bare sign-off
{"CQ K1ABC SS42", "K1ABC", true, ""}, // S is past R — no such field
// Non-callsign / free text → no sender.
{"TNX 73 GL", "", false},
{"K1ABC RR73", "", false}, // only one call + a token → 2nd token not a call
{"", "", false},
{"CQ CQ CQ", "", true}, // CQ but no resolvable call
{"TNX 73 GL", "", false, ""},
{"K1ABC RR73", "", false, ""}, // only one call + a token → 2nd token not a call
{"", "", false, ""},
{"CQ CQ CQ", "", true, ""}, // CQ but no resolvable call
}
for _, c := range cases {
call, cq := wsjtSender(c.msg)
if call != c.wantCall || cq != c.wantCQ {
t.Errorf("wsjtSender(%q) = (%q,%v), want (%q,%v)", c.msg, call, cq, c.wantCall, c.wantCQ)
call, cq, grid := wsjtSender(c.msg)
if call != c.wantCall || cq != c.wantCQ || grid != c.wantGrid {
t.Errorf("wsjtSender(%q) = (%q,%v,%q), want (%q,%v,%q)",
c.msg, call, cq, grid, c.wantCall, c.wantCQ, c.wantGrid)
}
}
}
+40
View File
@@ -3002,3 +3002,43 @@ func IsFilterable(column string) bool { return filterableColumns[column] }
// column of their own and live in extras_json.
func IsBulkEditableExtra(field string) bool { _, ok := bulkEditableExtras[field]; return ok }
func IsFilterableExtra(field string) bool { _, ok := filterableExtras[field]; return ok }
// WorkedGridKeys returns the set of "GRID|MODE" keys already in the log, where
// GRID is the 4-character field+square and MODE has been through normMode.
//
// The mode is part of the key, and normMode is what makes the DX-cluster
// "group digital modes" option apply to grids for free: with grouping on, FT8
// and FT4 both normalise to the same token, so a grid worked on FT8 is not new
// on FT4; with it off they stay separate keys and it is.
//
// Truncated to four characters on the way in. A log holds a mix of JN36 and
// JN36QU depending on where each QSO came from, and grid chasing is a
// field+square game — without the truncation the same square counts as new
// forever, once per subsquare.
func (r *Repo) WorkedGridKeys(ctx context.Context, normMode func(string) string) (map[string]struct{}, error) {
rows, err := r.db.QueryContext(ctx,
`SELECT DISTINCT COALESCE(gridsquare,''), COALESCE(mode,'') FROM qso
WHERE gridsquare IS NOT NULL AND gridsquare != ''`)
if err != nil {
return nil, err
}
defer rows.Close()
out := make(map[string]struct{}, 4096)
for rows.Next() {
var grid, mode string
if err := rows.Scan(&grid, &mode); err != nil {
return nil, err
}
grid = strings.ToUpper(strings.TrimSpace(grid))
if len(grid) < 4 {
continue
}
grid = grid[:4]
mode = strings.ToUpper(strings.TrimSpace(mode))
if normMode != nil && mode != "" {
mode = normMode(mode)
}
out[grid+"|"+mode] = struct{}{}
}
return out, rows.Err()
}