fix: ADIF import silently dropped a file with no header
Reported with screenshots: a dozen records lifted out of wsjtx_log.adi into a new .adi imported as "0 imported, 0 ignored, 0 total". The parser only emitted records after <EOH>, and a hand-assembled file has no header — so the whole file was read as header text and thrown away. The count says the file was empty; there is nothing on screen to suggest a missing tag, which is why this was reported as "import does nothing". An <eor> is proof that a record ended, header or no header. Fields are now collected always and DISCARDED at <eoh> — that tag is exactly the statement "everything before this was the header", so real headers stay out of the data just as reliably, and a headerless file keeps its first record instead of losing it to the same rule. An existing test asserted the old behaviour. It encoded a defensible reading of the spec, but the file it rejects is one operators really make and the rejection is silent, so the test is rewritten with the evidence that changed it rather than deleted.
This commit is contained in:
+4
-2
@@ -21,7 +21,8 @@
|
|||||||
"Yaesu console meters are right: power now reads the meter that follows the RF (in watts, on a curve measured against the rig own display, not the power setting) and SWR shows the ratio itself. Both were reading the wrong index — the SWR bar sat near 80 on a perfect match. The bars also hold their peak for a moment before falling back, so they stay readable through the gaps between words in a CQ, in CW as in SSB.",
|
"Yaesu console meters are right: power now reads the meter that follows the RF (in watts, on a curve measured against the rig own display, not the power setting) and SWR shows the ratio itself. Both were reading the wrong index — the SWR bar sat near 80 on a perfect match. The bars also hold their peak for a moment before falling back, so they stay readable through the gaps between words in a CQ, in CW as in SSB.",
|
||||||
"Voice keyer: a level control for the messages sent to the radio (Settings → Audio). They went out exactly as recorded, so a quietly recorded microphone drove the rig quietly with nothing in OpsLog to fix it. Play previews at the same level, and the hint names the rig setting to check when it transmits almost nothing (on an FTDX10, SSB MOD SOURCE = REAR).",
|
"Voice keyer: a level control for the messages sent to the radio (Settings → Audio). They went out exactly as recorded, so a quietly recorded microphone drove the rig quietly with nothing in OpsLog to fix it. Play previews at the same level, and the hint names the rig setting to check when it transmits almost nothing (on an FTDX10, SSB MOD SOURCE = REAR).",
|
||||||
"The voice keyer PTT list now names the Yaesu and Xiegu backends too — with a native backend selected it showed no CAT name, which read as \"CAT PTT is not available for my radio\".",
|
"The voice keyer PTT list now names the Yaesu and Xiegu backends too — with a native backend selected it showed no CAT name, which read as \"CAT PTT is not available for my radio\".",
|
||||||
"WinKeyer 2: keying is fixed. The dit/dah ratio was sent on the command that sets KEY COMPENSATION, so every element got 50 ms of extra key-down — at 25 wpm a dit is 48 ms — and the keyer sent one element then stalled. The compensation is also cleared, since a keyer already set that way keeps it."
|
"WinKeyer 2: keying is fixed. The dit/dah ratio was sent on the command that sets KEY COMPENSATION, so every element got 50 ms of extra key-down — at 25 wpm a dit is 48 ms — and the keyer sent one element then stalled. The compensation is also cleared, since a keyer already set that way keeps it.",
|
||||||
|
"ADIF import accepts a file with no header. Records pasted straight out of another log — a few lines lifted from wsjtx_log.adi into a new .adi — imported as \"0 imported, 0 ignored, 0 total\", which reads as an empty file rather than a missing tag."
|
||||||
],
|
],
|
||||||
"fr": [
|
"fr": [
|
||||||
"Les zones CQ et ITU corrigées à la main dans votre profil restent corrigées. Elles proviennent de cty.dat, qui donne les zones de l'entité entière — dans un pays qui en couvre plusieurs, la valeur automatique est fausse et revenait à chaque redémarrage. Elles ne se remplissent désormais que si le champ est vide, et se recalculent quand l'indicatif ou le locator change.",
|
"Les zones CQ et ITU corrigées à la main dans votre profil restent corrigées. Elles proviennent de cty.dat, qui donne les zones de l'entité entière — dans un pays qui en couvre plusieurs, la valeur automatique est fausse et revenait à chaque redémarrage. Elles ne se remplissent désormais que si le champ est vide, et se recalculent quand l'indicatif ou le locator change.",
|
||||||
@@ -42,7 +43,8 @@
|
|||||||
"Les mesures de la console Yaesu sont justes : la puissance lit l'instrument qui suit la HF (en watts, sur une courbe relevée face à l'affichage de la radio, et non le réglage de puissance) et le ROS affiche le rapport lui-même. Les deux lisaient le mauvais index — la barre de ROS restait vers 80 sur une antenne parfaite. Les barres retiennent aussi leur crête un instant avant de redescendre : elles restent lisibles pendant les silences entre les mots d’un CQ, en CW comme en phonie.",
|
"Les mesures de la console Yaesu sont justes : la puissance lit l'instrument qui suit la HF (en watts, sur une courbe relevée face à l'affichage de la radio, et non le réglage de puissance) et le ROS affiche le rapport lui-même. Les deux lisaient le mauvais index — la barre de ROS restait vers 80 sur une antenne parfaite. Les barres retiennent aussi leur crête un instant avant de redescendre : elles restent lisibles pendant les silences entre les mots d’un CQ, en CW comme en phonie.",
|
||||||
"Voice keyer : un réglage de niveau pour les messages envoyés à la radio (Réglages → Audio). Ils partaient exactement tels qu'enregistrés : un micro capté faiblement attaquait donc la radio faiblement, sans rien dans OpsLog pour y remédier. Le bouton Lire fait entendre ce même niveau, et l'aide indique le réglage radio à vérifier quand elle n'émet presque rien (sur un FTDX10, SSB MOD SOURCE = REAR).",
|
"Voice keyer : un réglage de niveau pour les messages envoyés à la radio (Réglages → Audio). Ils partaient exactement tels qu'enregistrés : un micro capté faiblement attaquait donc la radio faiblement, sans rien dans OpsLog pour y remédier. Le bouton Lire fait entendre ce même niveau, et l'aide indique le réglage radio à vérifier quand elle n'émet presque rien (sur un FTDX10, SSB MOD SOURCE = REAR).",
|
||||||
"La liste PTT du voice keyer nomme aussi les backends Yaesu et Xiegu — avec un backend natif sélectionné, aucun nom de CAT n'apparaissait, ce qui se lisait « il n'y a pas de PTT CAT pour ma radio ».",
|
"La liste PTT du voice keyer nomme aussi les backends Yaesu et Xiegu — avec un backend natif sélectionné, aucun nom de CAT n'apparaissait, ce qui se lisait « il n'y a pas de PTT CAT pour ma radio ».",
|
||||||
"WinKeyer 2 : la manipulation est réparée. Le rapport point/trait était envoyé sur la commande qui règle la COMPENSATION DE MANIPULATION : chaque élément recevait donc 50 ms de fermeture supplémentaire — à 25 mots/minute un point dure 48 ms — et le keyer envoyait un élément puis se bloquait. La compensation est aussi remise à zéro, car un keyer déjà réglé ainsi la conserve."
|
"WinKeyer 2 : la manipulation est réparée. Le rapport point/trait était envoyé sur la commande qui règle la COMPENSATION DE MANIPULATION : chaque élément recevait donc 50 ms de fermeture supplémentaire — à 25 mots/minute un point dure 48 ms — et le keyer envoyait un élément puis se bloquait. La compensation est aussi remise à zéro, car un keyer déjà réglé ainsi la conserve.",
|
||||||
|
"L'import ADIF accepte un fichier sans en-tête. Des enregistrements collés depuis un autre log — quelques lignes reprises de wsjtx_log.adi dans un nouveau .adi — donnaient « 0 importé, 0 ignoré, 0 au total », ce qui se lit comme un fichier vide plutôt que comme une balise manquante."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
+23
-7
@@ -22,9 +22,11 @@ import (
|
|||||||
// Record is a single ADIF record. Keys are lowercased field names.
|
// Record is a single ADIF record. Keys are lowercased field names.
|
||||||
type Record map[string]string
|
type Record map[string]string
|
||||||
|
|
||||||
// Parse reads an ADI stream and invokes fn for each record (after <EOH>).
|
// Parse reads an ADI stream and invokes fn for each record.
|
||||||
// Returning a non-nil error from fn stops parsing and is propagated.
|
// Returning a non-nil error from fn stops parsing and is propagated.
|
||||||
// The header (text before <EOH>) is silently discarded.
|
//
|
||||||
|
// The header (text before <EOH>) is silently discarded. A file with NO header is
|
||||||
|
// read as records from the first tag — hand-assembled files often have none.
|
||||||
func Parse(r io.Reader, fn func(Record) error) error {
|
func Parse(r io.Reader, fn func(Record) error) error {
|
||||||
return parseWith(r, nil, fn)
|
return parseWith(r, nil, fn)
|
||||||
}
|
}
|
||||||
@@ -44,7 +46,6 @@ func parseWith(r io.Reader, decodeValue func([]byte) string, fn func(Record) err
|
|||||||
br := bufio.NewReaderSize(r, 64*1024)
|
br := bufio.NewReaderSize(r, 64*1024)
|
||||||
|
|
||||||
rec := Record{}
|
rec := Record{}
|
||||||
headerDone := false
|
|
||||||
|
|
||||||
for {
|
for {
|
||||||
// Seek next '<'. Bytes before it are either header text or
|
// Seek next '<'. Bytes before it are either header text or
|
||||||
@@ -65,11 +66,19 @@ func parseWith(r io.Reader, decodeValue func([]byte) string, fn func(Record) err
|
|||||||
name, length := parseSpec(spec)
|
name, length := parseSpec(spec)
|
||||||
switch name {
|
switch name {
|
||||||
case "eoh":
|
case "eoh":
|
||||||
headerDone = true
|
rec = Record{} // everything collected so far was the header
|
||||||
rec = Record{}
|
|
||||||
continue
|
continue
|
||||||
case "eor":
|
case "eor":
|
||||||
if headerDone && len(rec) > 0 {
|
// An <eor> proves we are in the record section, whether or not an <eoh>
|
||||||
|
// came first. Files pasted together by hand — a few lines lifted out of
|
||||||
|
// wsjtx_log.adi into a new file — have no header at all, and requiring
|
||||||
|
// one made every record invisible: the import reported 0 imported,
|
||||||
|
// 0 ignored, 0 total, which reads as "the file is empty" rather than
|
||||||
|
// "there was no header".
|
||||||
|
//
|
||||||
|
// A real header cannot contain <eor>, so accepting it costs nothing for
|
||||||
|
// well-formed files: theirs ends at <eoh> long before any record.
|
||||||
|
if len(rec) > 0 {
|
||||||
if err := fn(rec); err != nil {
|
if err := fn(rec); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -95,7 +104,14 @@ func parseWith(r io.Reader, decodeValue func([]byte) string, fn func(Record) err
|
|||||||
if decodeValue == nil && !utf8.Valid(val) {
|
if decodeValue == nil && !utf8.Valid(val) {
|
||||||
val = extendToRunes(br, val, length)
|
val = extendToRunes(br, val, length)
|
||||||
}
|
}
|
||||||
if headerDone && name != "" {
|
// Fields are collected ALWAYS, and thrown away at <eoh>.
|
||||||
|
//
|
||||||
|
// Collecting them only after the header meant a headerless file lost its
|
||||||
|
// FIRST record whole: its fields were discarded as header text, and the
|
||||||
|
// <eor> that followed found nothing to emit. Discarding at <eoh> instead
|
||||||
|
// keeps real headers out of the data just as well — that tag is exactly
|
||||||
|
// what says "everything before this was the header".
|
||||||
|
if name != "" {
|
||||||
if decodeValue != nil {
|
if decodeValue != nil {
|
||||||
rec[name] = decodeValue(val)
|
rec[name] = decodeValue(val)
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
@@ -51,19 +51,28 @@ func TestParseValueWithAngleBracket(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestParseNoHeader(t *testing.T) {
|
func TestParseNoHeader(t *testing.T) {
|
||||||
// Some loggers omit the header entirely — records before <EOH> are
|
// A file without a header is IMPORTED, not discarded.
|
||||||
// discarded by design. Verify nothing is emitted in that case.
|
//
|
||||||
|
// This test asserted the opposite until 2026-07-29, when an operator pasted a
|
||||||
|
// dozen lines out of wsjtx_log.adi into a new .adi and got "0 imported,
|
||||||
|
// 0 ignored, 0 total". Discarding records before <EOH> was a defensible
|
||||||
|
// reading of the spec, but the file it rejects is one people really make, and
|
||||||
|
// the rejection is silent: the count says the file was empty, not that a tag
|
||||||
|
// was missing. An <eor> is proof enough that a record ended.
|
||||||
src := `<CALL:5>F4XYZ<EOR>`
|
src := `<CALL:5>F4XYZ<EOR>`
|
||||||
var got int
|
var got []Record
|
||||||
err := Parse(strings.NewReader(src), func(r Record) error {
|
err := Parse(strings.NewReader(src), func(r Record) error {
|
||||||
got++
|
got = append(got, r)
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("parse: %v", err)
|
t.Fatalf("parse: %v", err)
|
||||||
}
|
}
|
||||||
if got != 0 {
|
if len(got) != 1 {
|
||||||
t.Errorf("expected 0 records without <EOH>, got %d", got)
|
t.Fatalf("expected the record to be imported without <EOH>, got %d", len(got))
|
||||||
|
}
|
||||||
|
if got[0]["call"] != "F4XYZ" {
|
||||||
|
t.Errorf("call = %q, want F4XYZ", got[0]["call"])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -82,3 +91,57 @@ func TestParseTypedField(t *testing.T) {
|
|||||||
t.Errorf("freq mismatch: %q", got["freq"])
|
t.Errorf("freq mismatch: %q", got["freq"])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A file with NO header must still import.
|
||||||
|
//
|
||||||
|
// Reported from the field: an operator lifted a dozen lines out of wsjtx_log.adi
|
||||||
|
// into a new .adi and got "0 imported, 0 ignored, 0 total" — which reads as an
|
||||||
|
// empty file rather than as a missing header. Records were only emitted after
|
||||||
|
// <EOH>, so a hand-assembled file was invisible in its entirety.
|
||||||
|
func TestParseHeaderlessFile(t *testing.T) {
|
||||||
|
// The operator's exact shape: records straight from a WSJT-X log, no header.
|
||||||
|
in := "<call:5>PY2VE <gridsquare:4>GG67 <mode:3>FT8 <rst_sent:3>-18 <rst_rcvd:3>-15 " +
|
||||||
|
"<qso_date:8>20260728 <time_on:6>194200 <band:3>15m <eor>\n" +
|
||||||
|
"<call:5>W8OBX <gridsquare:4>FM25 <mode:3>FT8 <rst_sent:3>-19 <rst_rcvd:3>-12 " +
|
||||||
|
"<qso_date:8>20260728 <time_on:6>194500 <band:3>15m <eor>\n"
|
||||||
|
|
||||||
|
var got []Record
|
||||||
|
if err := Parse(strings.NewReader(in), func(r Record) error {
|
||||||
|
got = append(got, r)
|
||||||
|
return nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("Parse: %v", err)
|
||||||
|
}
|
||||||
|
if len(got) != 2 {
|
||||||
|
t.Fatalf("parsed %d records from a headerless file, want 2", len(got))
|
||||||
|
}
|
||||||
|
if got[0]["call"] != "PY2VE" || got[1]["call"] != "W8OBX" {
|
||||||
|
t.Errorf("callsigns = %q, %q — want PY2VE, W8OBX", got[0]["call"], got[1]["call"])
|
||||||
|
}
|
||||||
|
if got[0]["gridsquare"] != "GG67" {
|
||||||
|
t.Errorf("gridsquare = %q, want GG67", got[0]["gridsquare"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// And a normal file must be unaffected: its header is still discarded, so header
|
||||||
|
// fields never leak into the first record.
|
||||||
|
func TestParseHeaderStillDiscarded(t *testing.T) {
|
||||||
|
in := "Generated by SomeLogger\n<adif_ver:5>3.1.4 <programid:6>OpsLog <eoh>\n" +
|
||||||
|
"<call:4>F4AA <mode:2>CW <eor>\n"
|
||||||
|
var got []Record
|
||||||
|
if err := Parse(strings.NewReader(in), func(r Record) error {
|
||||||
|
got = append(got, r)
|
||||||
|
return nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("Parse: %v", err)
|
||||||
|
}
|
||||||
|
if len(got) != 1 {
|
||||||
|
t.Fatalf("parsed %d records, want 1", len(got))
|
||||||
|
}
|
||||||
|
if _, leaked := got[0]["programid"]; leaked {
|
||||||
|
t.Error("a header field leaked into the record")
|
||||||
|
}
|
||||||
|
if got[0]["call"] != "F4AA" {
|
||||||
|
t.Errorf("call = %q, want F4AA", got[0]["call"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user