Main.BrowserStreams History
Hide minor edits - Show changes to output
November 25, 2013, at 11:35 PM
by - Added link to Rachel's WebVideo suggestion
Changed line 28 from:
The current browser models and the painting flows mean that event loops are currently probably the best way to handle processing new frames. But there is an opportunity for a new sort of approach like a WebVideo API that's a parallel peer to the WebAudio API. And in just the way WebAudio has optimised common processing into well defined nodes and then leaves open the ScriptProcessorNode for deeper experiementation. This sort of model would work well for a WebVideo approach and could be tied in with the opaque texture id's discussed above.
to:
The current browser models and the painting flows mean that event loops are currently probably the best way to handle processing new frames. But there is an opportunity for a new sort of approach like a WebVideo API ([[http://lists.w3.org/Archives/Public/public-media-capture/2013Sep/0084.html|see earlier suggestion by Rachel Blum]]) that's a parallel peer to the WebAudio API. And in just the way WebAudio has optimised common processing into well defined nodes and then leaves open the ScriptProcessorNode for deeper experiementation. This sort of model would work well for a WebVideo approach and could be tied in with the opaque texture id's discussed above.
November 19, 2013, at 12:06 PM
by - Removed the moustache reference 8)
Changed lines 1-3 from:
There's lots of fragsolidation
to:
There's lots of fragmentation/consolidation going on around streams - WhatWG, W3C and MS.
November 19, 2013, at 04:11 AM
by - Updated comment about the texture id
Changed line 12 from:
Under the hood the browser (at least Chrome) currently uses GPU based opaque texture id's so references to textures can efficiently be passed around with minimal copies. If we could access this inside GLSL then we could potentially take a big step forward. But as soon as the pixels are made available to the Web Platform then a massive performance hit occurs.
to:
Under the hood the browser (at least Chrome) currently uses GPU based opaque texture id's so references to textures can efficiently be passed around with minimal copies. If we could access this texture inside GLSL without it being painted or copied into the Web Platform (e.g. just inside the shader) then we could potentially take a big step forward. But as soon as the pixels are made available to the Web Platform then a massive performance hit occurs.
Changed line 22 from:
NOTE: In the W3C spec handles back pressure using the "write pending flag".
to:
NOTE: the W3C spec handles back pressure using the (JS-invisible) "write pending flag" and thrown errors to indicate congestion; the whatwg spec handles it using an explicit writeability flag.
November 19, 2013, at 01:05 AM
by - First draft of notes from the Streams session
Added lines 1-34:
The time is now for putting a moustache on everyone on youtube, and we need a spec to make it so! 8)
There's lots of fragsolidation going on around streams - WhatWG, W3C and MS.
NOTE: This work is impacting or will impact many of the WGs.
Part of the problem here my come from the fact the term stream is used in many different media and programming ideas.
For an overview of "all" the current flows from source to sink with a focus on Post-Processing see http://www.slideshare.net/robman/web-standards-for-ar-workshop-at-ismar13/14
It's probably useful to separate this groups discussion into Streams (flow) and the Post-Processing that's built on top of (or into) these Streams. With a focus on video/computer vision specially contrast the (currently very inefficient) Video/Canvas model with the slightly more efficient Video/Shader model.
Under the hood the browser (at least Chrome) currently uses GPU based opaque texture id's so references to textures can efficiently be passed around with minimal copies. If we could access this inside GLSL then we could potentially take a big step forward. But as soon as the pixels are made available to the Web Platform then a massive performance hit occurs.
For much computer vision this is fine as all processing "could" happen in GLSL and only the resulting key features, or other meta data would be returned - not the actual pixels.
NOTE: Platforms with general hardware video decode will not be as efficient for post processing.
From a JS dev perspective the ability to connect ArrayBuffers to both ends of many types of end points (XHR, WebSocket, DataChannel, etc) makes the management of data exchange much simpler and more elegant.
The spec from the whatwg is doing more than just media, and the Damon says the JS community likes it.
NOTE: In the W3C spec handles back pressure using the "write pending flag".
Some comments from the group is that a great thing about whatwg streams is how generic it is. As usual collaboration between the two groups would be great.
Streams are obviously critical for media delivery and being able to do efficient post processing like shaped noise addition could be very useful for services like YouTube.
There is current work underway (see Ian's initial proposal and the following threads) to allow workers to use/access canvas elements/contexts.
The current browser models and the painting flows mean that event loops are currently probably the best way to handle processing new frames. But there is an opportunity for a new sort of approach like a WebVideo API that's a parallel peer to the WebAudio API. And in just the way WebAudio has optimised common processing into well defined nodes and then leaves open the ScriptProcessorNode for deeper experiementation. This sort of model would work well for a WebVideo approach and could be tied in with the opaque texture id's discussed above.
NOTE: There's nothing in browsers currently to decode media without rendering it. A WebVideo API/node would be "completely" independent of rendering.
Native client extension may be a good route to prototype a web video node...however the browser are currently very fragmented for this type of development NaCL vs Emscripten/asm.js.
There's lots of fragsolidation going on around streams - WhatWG, W3C and MS.
NOTE: This work is impacting or will impact many of the WGs.
Part of the problem here my come from the fact the term stream is used in many different media and programming ideas.
For an overview of "all" the current flows from source to sink with a focus on Post-Processing see http://www.slideshare.net/robman/web-standards-for-ar-workshop-at-ismar13/14
It's probably useful to separate this groups discussion into Streams (flow) and the Post-Processing that's built on top of (or into) these Streams. With a focus on video/computer vision specially contrast the (currently very inefficient) Video/Canvas model with the slightly more efficient Video/Shader model.
Under the hood the browser (at least Chrome) currently uses GPU based opaque texture id's so references to textures can efficiently be passed around with minimal copies. If we could access this inside GLSL then we could potentially take a big step forward. But as soon as the pixels are made available to the Web Platform then a massive performance hit occurs.
For much computer vision this is fine as all processing "could" happen in GLSL and only the resulting key features, or other meta data would be returned - not the actual pixels.
NOTE: Platforms with general hardware video decode will not be as efficient for post processing.
From a JS dev perspective the ability to connect ArrayBuffers to both ends of many types of end points (XHR, WebSocket, DataChannel, etc) makes the management of data exchange much simpler and more elegant.
The spec from the whatwg is doing more than just media, and the Damon says the JS community likes it.
NOTE: In the W3C spec handles back pressure using the "write pending flag".
Some comments from the group is that a great thing about whatwg streams is how generic it is. As usual collaboration between the two groups would be great.
Streams are obviously critical for media delivery and being able to do efficient post processing like shaped noise addition could be very useful for services like YouTube.
There is current work underway (see Ian's initial proposal and the following threads) to allow workers to use/access canvas elements/contexts.
The current browser models and the painting flows mean that event loops are currently probably the best way to handle processing new frames. But there is an opportunity for a new sort of approach like a WebVideo API that's a parallel peer to the WebAudio API. And in just the way WebAudio has optimised common processing into well defined nodes and then leaves open the ScriptProcessorNode for deeper experiementation. This sort of model would work well for a WebVideo approach and could be tied in with the opaque texture id's discussed above.
NOTE: There's nothing in browsers currently to decode media without rendering it. A WebVideo API/node would be "completely" independent of rendering.
Native client extension may be a good route to prototype a web video node...however the browser are currently very fragmented for this type of development NaCL vs Emscripten/asm.js.
