Automated testing
Selenium -> browser Mobile -> ??
Canvas; screen grab w/ control stream
Mozilla eideticker (monitors output of device)
- https://wiki.mozilla.org/Project_Eideticker
Remote developer console on devices
What to do with permission dialogs / extra-DOM stuff
JavaScript? unit frameworks
* qunit * tape
Instrumetnation in the wild (w/ YouTube?)
webplatform.org
* each browser vendor has suite of browser tests
Orchestrated multiple browser w/ mult. participants very hard to test (WebRTC)
KitKat? will solve a lot of problems (runs on older, weaker hardware)
Success Stories?
* Xiph: Testing for Opus * 160 core cluster w/ Jenkins * Each commit 1000's of hours of audio processed * Performance testing is harder * Multi-platform support (ARM, etc) * Lots of different chips * Consistent test results: hard * What test hooks can be added to codec implementations? * Support extensions for test hooks
Rdio: Capture HDMI output and compare to expected decoded frames
* not generalizable
Expose device hardware capabilities via API?
Performance tests / certification
