Web Codecs
Notes.WebCodecs History
Hide minor edits - Show changes to output
Added lines 1-53:
WebCodecs
* How do init segments work with WebCodecs?
* Configure method on the VideoDecoder with the mime type
* Depends on the codec
* There’s an API on both the decoder and encoder side
* Not all codecs have an init segment like H264
* For example, AV1
* Bridging MSE players to WebGPU with WebCodecs
* WebGPU + WebCodecs is the Golden Path
* Large investment to convert MSE to WebCodecs
* Video decoding + audio decoding + a/v sync
* Audio sync is really hard
* How can we make it easier for MSE players to use some of WebCodecs, e.g. VideoFrames for importing into WebGPU
* requestVideoFrameCallback is great but it runs only on the main thread
* MediaStream API is great on paper but doesn’t work well in realtime
* TrackProcessor can’t capture above 30fps
* TrackGenerator seems to copy frames out of the GPU
* WebCodecs and WASM
* VideoFrames can’t be natively zero copied over the GPU
* Can we use a shared ArrayBuffer? Not sure
* Use case: using WASM to decode natively unsupported codecs
* Too slow if we need to copy frames
* Can you write your own codec in WebCodecs?
* No, but it’s in the explainer as a non-goal of WebCodecs
* Why?
* Is there a way to make the audio stack easier?
* The browser has more access to info - e.g. handling when the default audio device changes
* AudioWorklet isn’t a “sink”, need to make a ring buffer and handle it yourself
* AudioContext breaks when underflowing
* Just grant access to the browser’s audio ring buffer
* What are people using WebCodecs for today?
* Low latency streaming (moq.js)
* Post processing of VideoFrames in WebGPU
* Online video editors
* What are potential use cases?
* WASM decoders vending VideoFrames
* MSE latency is too high
* WebRTC is too constrained
* Can build your own solution with WebCodecs
* What are some problems?
* Implicit YUV conversion
* Sometimes Chrome does the conversion wrong and colors are messed up
* Unnecessary use of resources when we don’t need YUV
* Can’t do DRM
* No support for HDR
* But HDR support is a nightmare anyway
* Do SVC flags work? Has anyone tried this?
* Video is “easy”, audio is hard
* Reaching WebRTC parity
* “Once you escape MSE, you’re on your own!”
* Making audio easier would make adopting WebCodecs way easier
* Updates on our feedback from Chrome will be posted in #webcodecs in video-dev slack
* How do init segments work with WebCodecs?
* Configure method on the VideoDecoder with the mime type
* Depends on the codec
* There’s an API on both the decoder and encoder side
* Not all codecs have an init segment like H264
* For example, AV1
* Bridging MSE players to WebGPU with WebCodecs
* WebGPU + WebCodecs is the Golden Path
* Large investment to convert MSE to WebCodecs
* Video decoding + audio decoding + a/v sync
* Audio sync is really hard
* How can we make it easier for MSE players to use some of WebCodecs, e.g. VideoFrames for importing into WebGPU
* requestVideoFrameCallback is great but it runs only on the main thread
* MediaStream API is great on paper but doesn’t work well in realtime
* TrackProcessor can’t capture above 30fps
* TrackGenerator seems to copy frames out of the GPU
* WebCodecs and WASM
* VideoFrames can’t be natively zero copied over the GPU
* Can we use a shared ArrayBuffer? Not sure
* Use case: using WASM to decode natively unsupported codecs
* Too slow if we need to copy frames
* Can you write your own codec in WebCodecs?
* No, but it’s in the explainer as a non-goal of WebCodecs
* Why?
* Is there a way to make the audio stack easier?
* The browser has more access to info - e.g. handling when the default audio device changes
* AudioWorklet isn’t a “sink”, need to make a ring buffer and handle it yourself
* AudioContext breaks when underflowing
* Just grant access to the browser’s audio ring buffer
* What are people using WebCodecs for today?
* Low latency streaming (moq.js)
* Post processing of VideoFrames in WebGPU
* Online video editors
* What are potential use cases?
* WASM decoders vending VideoFrames
* MSE latency is too high
* WebRTC is too constrained
* Can build your own solution with WebCodecs
* What are some problems?
* Implicit YUV conversion
* Sometimes Chrome does the conversion wrong and colors are messed up
* Unnecessary use of resources when we don’t need YUV
* Can’t do DRM
* No support for HDR
* But HDR support is a nightmare anyway
* Do SVC flags work? Has anyone tried this?
* Video is “easy”, audio is hard
* Reaching WebRTC parity
* “Once you escape MSE, you’re on your own!”
* Making audio easier would make adopting WebCodecs way easier
* Updates on our feedback from Chrome will be posted in #webcodecs in video-dev slack