From FOMS 2013

Main: MediaSource

Media Source Extensions (Aaron)

1. What Changed since last year's FOMS

Many features added to SME since last year:

• appendMode attribute to sourceMode object (to handle transport streams; ignore timestamps;
  • to offload discontinuities handling to framework
  • also good for advertising insertion.
• All append operations are a-synchronous now
• Append Window: support appending sections of chunks.
• Better TrackList? management
• New API to remove segments (to out-smart the browser around garbage collection)
• Statistics functionality: videoPlaybackQuality object:
  • corruptedVideoFrames
  • creationTime
  • droppedVideoFrames
  • totalVideoFrames
  • totalFrameDelay

At risk for removal: "appendStream" and "totalFrameDelay".

videoPlaybackQuality is on video element, so could be used for non-MSE playback as well. Still part of MSE specification though.

2. Codec/container support in implementations:

• Chrome supports:
  • WebM? with VP8? and Vorbis
  • ISOBFF with H264 and MP3?.
  • VP9? only on desktop, Opus coming soon?
  • TS support only in Chromecast
  • MP3? audio-only support behind a flag (skips ID3? tags and Shoutcast headers)
• Internet Explorer 11 supports ISOBFF with H264/AAC
  • Only on Windows 8.1 (NOT Windows 7)
• Firefox working on implementations
  • See bug #855130 on Bugzilla
• Apple looking into MSE support
  • Some work done; desktop-only
  • Further down the road.
  • WebKit? gStreamer MSE work in progress too
  • All Webkit bugs tagged "mse"

3. Is API compatible between browsers?

• MSE on Chrome actually behind the spec
  • no "mode", no "totalframedelay"
  • "appendStream" behind a flag
  • no A/V trackList, because Blink doesn't support it yet.
  • there's currently no place that lists what's supported
    • maybe on Chromium.org?
  • All layout tests in Blink are written against W3C? spec.
    • Some are currently WebM? biased
  • IE seems pretty complete and based upon latest spec
    • YouTube? plays MSE in IE11?.

4. How does memory management / garbage collection works?

• Not clear how this works on Webkit.
• Simple solutions don't always work well.
• How to do AES decryption on top of MSE?
  • Using web workers to decouple.
  • Should be possible on Chrome/Android
  • FYI: updating bytestreams while appending not supported
    • (similar issue was raised with web audio)
    • Chrome actually makes copy instead of holding reference
    • Neutering is problematic, since other Views could be blown away (needs further defined first)

5. What about DASH?

• YouTube? is doing it today.
  • Including VP9?
• DASH.js reference implementation on DASH evangelist site.
  • Contains several example streams.
• Wowza servers are supporting DASH.
  • DASH264? only

6. What about text tracks?

• Currently reviewing VTT support in WebM?
  • Enabled in both SME and WebM? progressive.
  • No overlapping, just full cues
• Framework in Chrome for in-band text tracks in development
  • Muxed or not
  • Landing soon...
  • would also open support for multiple A/V tracks.
• Spec for VTT and TTML in MP4? written
  • Only supports full cues (no overlapping)
  • Spec lives on Text Tracks working group wiki (by Cyril)
• What about in-band cues?
  • Just VTT track with the "kind" set to metadata