Media Source Extensions (Aaron)

1. What Changed since last year's FOMS

Many features added to SME since last year:

  • appendMode attribute to sourceMode object (to handle transport streams; ignore timestamps;
    • to offload discontinuities handling to framework
    • also good for advertising insertion.
  • All append operations are a-synchronous now
  • Append Window: support appending sections of chunks.
  • Better TrackList? management
  • New API to remove segments (to out-smart the browser around garbage collection)
  • Statistics functionality: videoPlaybackQuality object:
    • corruptedVideoFrames
    • creationTime
    • droppedVideoFrames
    • totalVideoFrames
    • totalFrameDelay

At risk for removal: "appendStream" and "totalFrameDelay".

videoPlaybackQuality is on video element, so could be used for non-MSE playback as well. Still part of MSE specification though.

2. Codec/container support in implementations:

  • Chrome supports:
    • WebM? with VP8? and Vorbis
    • ISOBFF with H264 and MP3?.
    • VP9? only on desktop, Opus coming soon?
    • TS support only in Chromecast
    • MP3? audio-only support behind a flag (skips ID3? tags and Shoutcast headers)
  • Internet Explorer 11 supports ISOBFF with H264/AAC
    • Only on Windows 8.1 (NOT Windows 7)
  • Firefox working on implementations
    • See bug #855130 on Bugzilla
  • Apple looking into MSE support
    • Some work done; desktop-only
    • Further down the road.
    • WebKit? gStreamer MSE work in progress too
    • All Webkit bugs tagged "mse"

3. Is API compatible between browsers?

  • MSE on Chrome actually behind the spec
    • no "mode", no "totalframedelay"
    • "appendStream" behind a flag
    • no A/V trackList, because Blink doesn't support it yet.
    • there's currently no place that lists what's supported
      • maybe on Chromium.org?
    • All layout tests in Blink are written against W3C? spec.
      • Some are currently WebM? biased
    • IE seems pretty complete and based upon latest spec

4. How does memory management / garbage collection works?

  • Not clear how this works on Webkit.
  • Simple solutions don't always work well.
  • How to do AES decryption on top of MSE?
    • Using web workers to decouple.
    • Should be possible on Chrome/Android
    • FYI: updating bytestreams while appending not supported
      • (similar issue was raised with web audio)
      • Chrome actually makes copy instead of holding reference
      • Neutering is problematic, since other Views could be blown away (needs further defined first)

5. What about DASH?

  • YouTube? is doing it today.
  • DASH.js reference implementation on DASH evangelist site.
    • Contains several example streams.
  • Wowza servers are supporting DASH.

6. What about text tracks?

  • Currently reviewing VTT support in WebM?
    • Enabled in both SME and WebM? progressive.
    • No overlapping, just full cues
  • Framework in Chrome for in-band text tracks in development
    • Muxed or not
    • Landing soon...
    • would also open support for multiple A/V tracks.
  • Spec for VTT and TTML in MP4? written
    • Only supports full cues (no overlapping)
    • Spec lives on Text Tracks working group wiki (by Cyril)
  • What about in-band cues?
    • Just VTT track with the "kind" set to metadata