Part 09 — Media Streaming and Synchronized MetadataLink to the section Part 09 — Media Streaming and Synchronized Metadata#

Maturity: Draft · Profiles: S

9.1 Requirements summaryLink to the section 9.1 Requirements summary#

9.2 Stream descriptorLink to the section 9.2 Stream descriptor#

{
  "stream_id": "cam-1-main",
  "payload_id": "cam-1",
  "name": "EO main",
  "kind": "VIDEO",
  "codecs": ["H264", "H265"],
  "h264": { "profiles": ["CONSTRAINED_BASELINE", "MAIN", "HIGH"], "max_level": "4.1" },
  "resolutions": [{ "width": 1920, "height": 1080, "max_fps": 30 }, { "width": 1280, "height": 720, "max_fps": 30 }],
  "max_bitrate_kbps": 8000,
  "audio": false,
  "transports": ["WEBRTC", "RTSP"],
  "rtsp": { "url": "rtsps://gateway.example/univip/cam-1-main" },
  "srt": null,
  "in_band_metadata": ["MISB_0601"],
  "latency_class": "INTERACTIVE"
}

latency_class is INTERACTIVE (target glass-to-glass < 500 ms), LOW (< 2 s) or CONTRIBUTION (reliability over latency).

h264.max_level MUST cover every advertised resolution and frame rate (for example level 3.1 covers 1280×720 at 30 fps but not 1920×1080, which needs 4.0), and the level the device signals in SDP (profile-level-id) MUST cover the resolution it sends.

9.3 WebRTC profileLink to the section 9.3 WebRTC profile#

• Signaling over the session channel. The client always makes the offer: { "stream_id", "kind": "offer", "sdp", "constraints"? }. The device answers with { "stream_id", "kind": "answer", "sdp", "webrtc_session_id", "ref_msg_id" }, where ref_msg_id is the msg_id of the offer it answers (SHOULD; it tells concurrent offers apart) and webrtc_session_id identifies this view and carries at least 122 random bits; the client echoes it on every later message for the view, so one session can hold several views of the same stream. Trickled ICE candidates are { "stream_id", "webrtc_session_id"?, "kind": "candidate", "candidate", "sdpMid", "sdpMLineIndex" } (webrtc_session_id is unknown to the client until the answer arrives; candidates it sends before then belong to its pending offer for that stream); an empty candidate string signals end of candidates. Either side ends a view with { "stream_id", "webrtc_session_id", "kind": "close", "reason"? }. Every view negotiated over a session ends when that session ends. (The 0.1 device-offer mode WEBRTC_DEVICE_OFFER is withdrawn: nobody implemented it.)
• constraints (optional, in an offer): width, height, max_fps, max_bitrate_kbps that the client would like; the device chooses the nearest it supports and is not obliged to honor them.
• Signaling over HTTP. POST /media/streams/{stream_id}/webrtc with an SDP offer (application/sdp, ICE candidates included, no trickle) returns 201 with the SDP answer and a Location naming the session resource; DELETE on that resource ends the view. Location MUST be a relative reference on the same origin as the request, and a client MUST NOT send its credentials to any other origin it names. A device that hosts REST MUST support it, a client-hosted endpoint MUST offer it to its own clients, and only the client that created a view (same authenticated client_id; optionally also univip.admin) may delete it — a client-hosted endpoint, whose callers all reach the device under its one client_id, MUST enforce the same rule among its own callers — anyone else receives 404, as for an unknown view, so that views cannot be discovered. The resource identifier MUST carry at least 122 random bits. An offer larger than 64 KiB is refused (HTTP 413, or MALFORMED on the session channel); a device MAY apply tighter limits to the number of media sections, lines and candidates.
• Errors. A signaling failure is reported with an error envelope (ref_msg_id naming the offending message) or problem document: UNKNOWN_STREAM (also for a device with no media at all), CODEC_UNSUPPORTED (no common H.264 profile), BUSY (viewer limit, HTTP 409 stream-busy), MALFORMED.
• ICE. A device SHOULD implement full ICE and gather server-reflexive and relay candidates from the STUN and TURN servers it is configured with; this is REQUIRED where clients reach it over the internet. A device MAY implement ICE-lite with host candidates only where every client can reach those addresses (a managed network or VPN), and then declares media.webrtc.ice: "LITE". Clients MUST implement full ICE and SHOULD support TURN. Every device, including an ICE-lite one, MUST verify consent freshness (RFC 7675) and send media only to a candidate pair with fresh consent, so that a forged connectivity check cannot redirect video to a third party. When consent is lost the device MUST release the view (and its slot) within 30 s, so that a client that vanished without a DELETE or close does not hold it; clients SHOULD still end views explicitly. A device MAY restrict the interfaces whose addresses it offers as host candidates; they otherwise disclose its network addresses (Part 13 §13.7).
• DTLS-SRTP is REQUIRED (it is inherent to WebRTC). Baseline: DTLS 1.2 or later with ECDHE key exchange (AEAD cipher suites preferred), an ECDSA P-256 certificate generated per peer connection or regularly rotated, and the peer's certificate verified against an SHA-256-or-stronger a=fingerprint from the SDP (an offer carrying only weaker fingerprints is refused), and the SRTP protection profile SRTP_AES128_CM_HMAC_SHA1_80; SRTP_AEAD_AES_128_GCM SHOULD also be offered.
• Video. H.264 packetization mode 1, Constrained Baseline or Main REQUIRED on the mandatory stream; High SHOULD be decodable by clients. A device MAY also offer VP8 or AV1 on a WebRTC stream (listed in codecs), which lets browsers built without H.264 show video; H.264 remains mandatory. Clients MUST include profile-level-id in their offers; a device receiving an offer without it MAY assume 42e01f. H.265 is optional with the H265 codec name. A device MUST answer a PLI or FIR with a key frame within 1 s and SHOULD send a key frame at least every 2 s, so that a late-joining viewer does not wait for a picture.
• Timing. RTCP sender reports MUST be sent at least every 2 s and MUST carry NTP timestamps derived from the same clock that stamps telemetry. The NTP time SHOULD be the capture time of the frame with that RTP timestamp; a device that maps it to the send time instead (as common RTP stacks do, adding the encode delay) is acceptable while the end-to-end alignment in §9.5 stays within 100 ms.
• Telemetry data channel. The client creates univip-telemetry in its offer (in-band negotiation, ordered, reliable). It carries traffic from device to client only: the device ignores messages from the client and advertises its a=max-message-size. The device sends on it telemetry envelopes identical to those on the session channel, which reduces jitter between video and telemetry arrival. Under backpressure the device MAY drop telemetry envelopes rather than queue them (a newer sample replaces an older one; the gap is visible in sequence). A client MUST accept either source and deduplicate on (session_id, sequence).
• Simulcast and bandwidth adaptation are at the device's discretion. A device MAY limit the number of views per client_id (refusing further offers with BUSY) so that one client cannot take every slot.
• H.264 encoders and decoders may be subject to patent licences and, for some encoder libraries, copyleft terms; these are the implementer's concern and outside this specification (Informative).

9.4 RTSP and SRT alternativesLink to the section 9.4 RTSP and SRT alternatives#

• RTSP: RTSP 1.0 over TLS (rtsps://) REQUIRED when RTSP is offered; plain rtsp:// MAY be offered only on the local network and MUST be flagged insecure: true. Authentication per Part 13. RTP/RTCP with sender reports as in §9.3.
• SRT: caller or listener mode as declared; AES-128 or AES-256 encryption REQUIRED; payload MPEG-2 TS with H.264/H.265 and, when in_band_metadata lists MISB_0601, KLV carried as synchronous metadata (ISO/IEC 13818-1 metadata carriage as profiled by MISB ST 1402: stream type 0x15, metadata and metadata-STD descriptors with format identifier KLVA, PES stream id 0xFC), one Local Set per video access unit with that frame's PTS. Asynchronous carriage (stream type 0x06 with registration KLVA) has no PTS alignment and MUST NOT be used for frame-aligned metadata. (Some FFmpeg versions, for example 6.1, also corrupt asynchronous KLV when remuxing.)

9.5 Synchronization modelLink to the section 9.5 Synchronization model#

 device clock (GNSS/NTP) ──► telemetry.timestamp
                       └────► RTCP SR NTP time ◄──► RTP timestamp ◄──► decoded frame

A client computes the device time of each decoded frame from the RTCP mapping and presents the telemetry interpolated to that time (between neighbouring samples, never across a gap longer than 2 s and never extrapolated); it MAY use the nearest sample instead when samples are at most 100 ms apart. For TS-based streams the ST 0601 Precision Time Stamp (tag 2) of the Local Set at the frame's PTS is the frame's device time.

Because a sender report's NTP time may be the send time rather than the capture time (§9.3), a client SHOULD estimate the RTP-to-device-time offset as the smallest offset among recent sender reports (the reference recorder uses the last eight), which removes most of the encode and send delay; measured with the reference gateway, the latest report alone erred by up to 71 ms under load and the minimum over eight by at most 21 ms.

Conformance target: the error between the device time a client assigns to a frame and the frame's true capture time (read from the timecode pattern, Part 14 §14.4) MUST be ≤ 100 ms, and the telemetry the client presents with the frame MUST be interpolated to (or, per above, sampled near) that time. The test measures the whole chain, so it covers devices that map RTCP time to send time (§9.3).

9.6 Recording requirements (client)Link to the section 9.6 Recording requirements (client)#

For every recorded segment, a Profile S client MUST retain, with the same retention as the video:

• device_id, session_id;
• the full telemetry stream as received (or a lossless re-encoding of it) with original timestamp and sequence. A TS copy with in-band ST 0601 carries position, attitude, gimbal pointing and footprint, but not vehicle state, power, link or health, so it replaces the telemetry stream only for the members it carries;
• all events received during the segment;
• the capability document in force;
• the mapping from recording time to device time: implicit when every frame carries its device time (ST 0601 tag 2 at the frame's PTS); otherwise per frame (or piecewise linear) from presentation time, anchored to the first frame of the segment rather than to absolute PTS values, so that a remux which shifts every PTS does not break it.

This allows replay of position, heading, altitude, gimbal orientation, vehicle state and footprint alongside the video, and search of recordings by geographic area.

Clients SHOULD also export the ST 0601 KLV form on request for exchange with systems that consume STANAG 4609, with the mapping of Annex A.3.

9.7 AudioLink to the section 9.7 Audio#

Audio from the aircraft is optional (kind: "AUDIO" or audio: true in a video descriptor), Opus REQUIRED when offered over WebRTC. Audio to the aircraft (speaker payload) is a SPEAKER payload command in Part 10, not a media stream, in v1.