Web Codecs

WebCodecs

    * Configure method on the VideoDecoder? with the mime type
    * Depends on the codec
    * There’s an API on both the decoder and encoder side
    * Not all codecs have an init segment like H264
        * For example, AV1?
    * WebGPU? + WebCodecs is the Golden Path
    * Large investment to convert MSE to WebCodecs
        * Video decoding + audio decoding + a/v sync
        * Audio sync is really hard
    * How can we make it easier for MSE players to use some of WebCodecs, e.g. VideoFrames? for importing into WebGPU?
        * requestVideoFrameCallback is great but it runs only on the main thread
    * MediaStream? API is great on paper but doesn’t work well in realtime
        * TrackProcessor? can’t capture above 30fps
        * TrackGenerator? seems to copy frames out of the GPU
    * VideoFrames? can’t be natively zero copied over the GPU
        * Can we use a shared ArrayBuffer?? Not sure
    * Use case: using WASM to decode natively unsupported codecs
        * Too slow if we need to copy frames
    * No, but it’s in the explainer as a non-goal of WebCodecs
        * Why?
  • Is there a way to make the audio stack easier?
    * The browser has more access to info - e.g. handling when the default audio device changes
    * AudioWorklet? isn’t a “sink”, need to make a ring buffer and handle it yourself
    * AudioContext? breaks when underflowing
    * Just grant access to the browser’s audio ring buffer
    * Low latency streaming (moq.js)
    * Post processing of VideoFrames? in WebGPU?
    * Online video editors
  • What are potential use cases?
    * WASM decoders vending VideoFrames?
    * MSE latency is too high
    * WebRTC? is too constrained
    * Can build your own solution with WebCodecs
  • What are some problems?
    * Implicit YUV conversion
        * Sometimes Chrome does the conversion wrong and colors are messed up
        * Unnecessary use of resources when we don’t need YUV
    * Can’t do DRM
    * No support for HDR
        * But HDR support is a nightmare anyway
    * Do SVC flags work? Has anyone tried this?
    * Video is “easy”, audio is hard
        * Reaching WebRTC? parity
  • “Once you escape MSE, you’re on your own!”
  • Making audio easier would make adopting WebCodecs way easier
  • Updates on our feedback from Chrome will be posted in #webcodecs in video-dev slack