WebCodecs
* Configure method on the VideoDecoder? with the mime type
* Depends on the codec
* There’s an API on both the decoder and encoder side
* Not all codecs have an init segment like H264
* For example, AV1?
* WebGPU? + WebCodecs is the Golden Path
* Large investment to convert MSE to WebCodecs
* Video decoding + audio decoding + a/v sync
* Audio sync is really hard
* How can we make it easier for MSE players to use some of WebCodecs, e.g. VideoFrames? for importing into WebGPU?
* requestVideoFrameCallback is great but it runs only on the main thread
* MediaStream? API is great on paper but doesn’t work well in realtime
* TrackProcessor? can’t capture above 30fps
* TrackGenerator? seems to copy frames out of the GPU
* VideoFrames? can’t be natively zero copied over the GPU
* Can we use a shared ArrayBuffer?? Not sure
* Use case: using WASM to decode natively unsupported codecs
* Too slow if we need to copy frames
* No, but it’s in the explainer as a non-goal of WebCodecs
* Why?
- Is there a way to make the audio stack easier?
* The browser has more access to info - e.g. handling when the default audio device changes
* AudioWorklet? isn’t a “sink”, need to make a ring buffer and handle it yourself
* AudioContext? breaks when underflowing
* Just grant access to the browser’s audio ring buffer
* Low latency streaming (moq.js)
* Post processing of VideoFrames? in WebGPU?
* Online video editors
- What are potential use cases?
* WASM decoders vending VideoFrames?
* MSE latency is too high
* WebRTC? is too constrained
* Can build your own solution with WebCodecs
* Implicit YUV conversion
* Sometimes Chrome does the conversion wrong and colors are messed up
* Unnecessary use of resources when we don’t need YUV
* Can’t do DRM
* No support for HDR
* But HDR support is a nightmare anyway
* Do SVC flags work? Has anyone tried this?
* Video is “easy”, audio is hard
* Reaching WebRTC? parity
- “Once you escape MSE, you’re on your own!”
- Making audio easier would make adopting WebCodecs way easier
- Updates on our feedback from Chrome will be posted in #webcodecs in video-dev slack