Recording Test: Flight Path Replay Alongside Recorded VideoLink to the section Recording Test: Flight Path Replay Alongside Recorded Video#

Status: Done, 2026-10-15. Roadmap Phase 1 item: "prove replay of flight path alongside recorded video using both in-band ST 0601 and out-of-band telemetry". Reference gateway 0.1.0 against specification 0.1.0-draft. Self-declared test of the reference implementation; it is not a conformance claim (Part 14 §14.6).

ResultLink to the section Result#

A UNIVIP client recorded a simulated drone's video together with its telemetry, and the flight path was replayed from the recording alone, twice: once from MISB ST 0601 metadata carried in the MPEG-TS beside the video, and once from the telemetry stream as received, placed on the video timeline through the recorded RTCP time mapping. Over about 26 s of a patrol (take-off and one full turn of a 150 m square at four times real speed, about 390 frames):

• The two paths agree to ST 0601 quantisation. The largest difference over all frames is 5.0 mm horizontally, 0.15 m in altitude and 0.0027° in heading, each within half a least significant bit of the ST 0601 item that carries it.
• Every frame's device time is within the 100 ms budget of Part 09 §9.5. The capture time burnt into each frame, read back from the recorded video, trails the ST 0601 Precision Time Stamp aligned to that frame by +6 to +21 ms (mean 8 to 12 ms) and is never early. That is no frame slip anywhere.
• The replayed path is the simulator's true path at an instant 6 to 13 ms after each frame's capture (the same bias), apart from the curve that a straight line between two telemetry samples cuts at a corner. Heading and altitude match within half an LSB, and the EARTH-locked gimbal replays at 90.000°/−45.000° through all four headings of the square.
• Either half of the recording suffices, as §9.6's "or" promises for position, heading, altitude, gimbal and footprint: a TS copy alone gives the identical in-band path and frame times; the telemetry plus the time mapping, with the KLV stripped out, gives the identical out-of-band path.
• Gaps are reported, not filled. A 2.6 s period in which the vehicle went silent appears as a run of samples without position and a run of Local Sets without position, and both paths break there. Telemetry samples deleted from a copy appear as sequence gaps with exact counts, and the path is not interpolated across the 3 s gap.
• Standard tools read the recording. ffprobe reports H.264 Constrained Baseline and a KLV data stream (KLVA, stream type 0x15) with one packet per video frame at the same PTS, and every frame decodes.

The test is gateway/tests/test_recording.py (11 checks, about 30 s). The tools are univip-recorder and univip-replay (gateway/univip_gateway/recording/).

MethodLink to the section Method#

Set-upLink to the section Set-up#

The reference gateway runs the simulator (scenario: patrol, time_scale: 4), so the vehicle flies at up to 32 m/s and turns at 360°/s on the device clock. The synthetic camera streams 640×360 at 15 fps over WebRTC, H.264 Constrained Baseline. The recorder is the client endpoint the gateway connects to. Its welcome asks for 10 Hz telemetry, which arrives on the session channel and on the univip-telemetry data channel and is deduplicated on (session_id, sequence). After the first telemetry arrives, the test takes a camera lease and points the gimbal EARTH-frame 90° (east), 45° down. Halfway along the third leg, the operator console makes the vehicle silent for 2.6 s (vehicle_link injection). The recording stops one second after the square is complete. A hook on the simulator's physics step logs the true state after every step (20 Hz) on the device clock.

What the recorder keeps (§9.6)Link to the section What the recorder keeps (§9.6)#

KLV carriage is the synchronous carriage of ISO/IEC 13818-1 as profiled by MISB ST 1402: stream type 0x15, PES stream id 0xFC, a metadata pointer descriptor and a metadata descriptor with format identifier KLVA, and one metadata access unit cell per PES. Asynchronous carriage (0x06 with a KLVA registration descriptor) is available with --klv async. A Local Set is about 210 bytes, 15 per second. At this test's low video bitrate (about 200 kbit/s) KLV is 20 % of the TS packets.

Each Local Set carries tag 2 set to the frame's device time. The other items are the telemetry interpolated to that time between the two samples around it, provided they are at most 2 s apart (§9.5, §7.11). A member is interpolated only when both samples carry it. Otherwise it is absent, so a frame in a gap gets a Local Set with time and identity but no position. Items written: 2, 3 (session_id), 4 (serial number), 5, 6, 7 (90/91 when pitch or roll exceeds the short items' range), 10 (model), 11 (payload_id), 13–17, 18–20 (gimbal relative to the platform body, see K3), 21, 23–25, 26–33 (82–89 when an offset exceeds ±0.075°), 65, 72 (start of the operation, when the vehicle.operation.started event was seen) and 75 (altitude_ellipsoid_m; Annex A.3 says 72, see K2).

The simulator sends no geospatial block, so the test enables --estimate-footprint. The recorder then computes frame centre, slant range and corners over flat ground at the home altitude (the §12.2 FLAT_EARTH assumption) and marks them client-estimated in the manifest. The replay estimates the out-of-band footprint the same way.

Time mapping: RTCP, not the timecodeLink to the section Time mapping: RTCP, not the timecode#

Each frame's device time comes from the RTCP sender reports, as §9.3 and §9.5 intend. The burnt-in timecode exists only on the gateway's synthetic camera and needs a decoded picture, so it serves here as the independent measure of the mapping's error. The gateway stamps each frame with its capture instant on the device clock and aiortc derives RTP time from that stamp, so RTP time is linear in capture time. A sender report pairs the RTP time of the last frame sent with the instant it was sent, after encoding (finding C1). The recorder takes the smallest RTP-to-device offset among the last eight reports (about 8 s), which keeps the least delayed one and still follows clock drift. It also logs the plain latest-report mapping for comparison.

These are twelve runs on a 4-core machine shared with other jobs. The 71 ms came from a run under heavy load. The min-offset mapping stayed at 11.5 ms in that same run. The timecode has 1 ms resolution and is truncated, so readings are biased +0.5 ms.

ReplayLink to the section Replay#

univip-replay uses only the recording. It demuxes the TS with FFmpeg (PyAV), decodes every Local Set (checksum verified), reads each frame's timecode, and rebuilds both paths. A frame's device time comes from timemap.jsonl when present, else from the tag 2 of its Local Set. When a remux has shifted every PTS by a constant, the replay detects the shift and reports it (K6). It writes summary.json, frames.csv, paths.geojson (both paths as MultiLineStrings broken at gaps) and replay.html, a self-contained page (no network fetches) that plays the H.264 copied into MP4 beside both paths drawn as SVG, the current positions, heading ticks and footprint.

TolerancesLink to the section Tolerances#

Measured against ground truthLink to the section Measured against ground truth#

In run 1, under load, one telemetry interval at a corner was long enough that the straight line between its two samples strayed 1.79 m from the curved path. Linear interpolation cannot do better between those samples, whatever the synchronisation. A device publishing at 10 Hz gives the client samples 30 to 50 ms from each frame (the "nearest sample" of §9.5).

What this proves and what it does notLink to the section What this proves and what it does not#

Proven, for the reference gateway and its simulator over loopback WebRTC:

• A client can record H.264 bit-exact with per-frame ST 0601 in a TS that FFmpeg reads, and replay position, heading, altitude, gimbal orientation and footprint from that TS alone, or from the telemetry plus the time mapping alone.
• The RTCP sender-report mapping meets the 100 ms budget with a fifth of the budget to spare even under load, if the client takes the least-delayed report. The plain latest-report mapping reached 71 ms.
• The BODY/EARTH conversion for tags 18–20 holds through all headings (K3).
• Gaps are carried through to the replay and drawn as gaps.

Not proven:

• Interoperability with a STANAG 4609 consumer other than FFmpeg (for example a commercial FMV player or the QGIS FMV plugin) was not tested. The ST 0601 encoding is checked against the standard's worked examples and the independent klvdata decoder only (next section).
• Packet loss, network jitter, a real camera and encoder, and clock drift between RTP and NTP clocks were not covered: the test runs on loopback, and the gateway derives both clocks from one system clock.
• A device that itself produces TS (requirement 9-6, SRT/RTSP ingest) was not covered: the client muxes here.
• Vehicle state, battery, link, health and events are not in the in-band path; Annex A.3 maps none of them (K5).
• Long recordings were not tested: PTS wrap after 26.5 h is unit-tested in the muxer only.
• The recorder takes encoded frames from aiortc through private attributes, since aiortc has no public API for that. Another WebRTC stack would need its own tap.

ST 0601 points not confirmedLink to the section ST 0601 points not confirmed#

MISB documents could not be consulted (the sites were unreachable). The codec follows the UAS Datalink Local Set as known: 16-byte key 06 0E 2B 34 02 0B 01 01 0E 01 03 01 01 00 00 00, BER-OID tags, BER lengths, the 16-bit running-sum checksum (tag 1, last), and linear fixed-point mappings with the most negative value reserved as the out-of-range indicator. It reproduces byte-exactly 41 item encodings from the ST 0601 worked examples and the ST 0902.5 Annex C "dynamic only" sample packet, as transcribed in the MIT-licensed klvdata test suite, including that packet's checksum C850. klvdata 0.0.3 also decodes 50 random full packets identically, and every 10th Local Set of each recording. Still unconfirmed:

1. Tag 65 edition. The recorder writes 17. The items used have not changed in recent editions as far as known.
2. Time scale of tag 2. Earlier editions define it as UTC microseconds since 1970 not counting leap seconds (POSIX), which is what is written. Later editions refer to MISP Time (ST 0603) and add Leap Seconds and Correction Offset items. If MISP Time counts leap seconds, the value differs by the accumulated leap seconds.
3. Rotation order and signs of tags 18–20 relative to tags 5–7. Implemented: platform heading, pitch, roll, then sensor azimuth, elevation, roll, each about Z, then the new Y, then the new X; elevation positive up.
4. Corner order of tags 26–33 and 82–89. Implemented as point 1 upper left, then clockwise, the same order as Part 12 §12.2.
5. Tags 90/91 beside 6/7. When pitch or roll exceeds ±20° or ±50°, tags 6/7 carry the out-of-range indicator and tags 90/91 the full value. Whether the standard wants both or only the full item is unconfirmed.
6. String encoding of tags 3, 4, 10, 11. Written as UTF-8, truncated to 127 bytes. Older editions say ISO 646.
7. Synchronous carriage details (ST 1402 rather than ISO/IEC 13818-1 itself). Used: a metadata pointer descriptor in the program loop with MPEG_carriage_flags 0, service id 0, and access unit cells marked complete and random access. FFmpeg 6.1 and 8.1 parse them.
8. Mandatory items. Tags 2 (first), 65 and 1 (last) are always written. The edition-specific minimum set is unconfirmed.

Errors found in Part 09 and Annex A.3Link to the section Errors found in Part 09 and Annex A.3#

• §9.4 / requirement 9-6: "a KLV private data PID (stream type 0x06, registration KLVA) synchronized by PTS" combines the asynchronous signalling with a synchronous intent (K1).
• Annex A.3 tag numbers: 72 is Event Start Time, not Sensor Ellipsoid Height (75). 73–74 are the RVT and VMTI Local Sets; Alternate Platform Latitude/Longitude are 67–68. 92 is Platform Angle of Attack (Full); there is no full-range heading item (K2). klvdata's independent tag table agrees.
• Annex A.3 tags 18–19: the subtraction and the "BODY" reading are only exact for a level platform. They are about 10° off at a 10° nose-down attitude with the camera to the side (K3).
• §9.6: the in-band alternative cannot replay vehicle state (K5).

Interoperability note for K1: FFmpeg 6.1.1, as shipped by Ubuntu 24.04, strips five bytes from every KLV PES as if it were a synchronous access-unit cell, so it corrupts the key of asynchronous KLV. FFmpeg 8.1, bundled with PyAV 17, handles both carriages. The recorder therefore defaults to synchronous carriage, and the TS tests check the KLV payload with PyAV.

Findings for specification 0.2Link to the section Findings for specification 0.2#

Same conventions as implementation-findings.md.

Supporting evidence for existing findings: C8 (timecode strip). The strip read back correctly on 100 % of frames decoded from the recorded TS (about 390 per run). This supports specifying it as the alignment test pattern.

ReproducingLink to the section Reproducing#

cd gateway
pip install -e '.[media,test]'        # test extra adds klvdata (MIT), used only as an oracle; ffmpeg/ffprobe on PATH
pytest -q -s tests/test_recording.py tests/test_st0601.py tests/test_mpegts.py

-s prints the measured numbers. To look at a recording, run univip-replay <dir> and open <dir>/replay/replay.html.