SketchScan is an application for mobile devices that lets you capture, clean, animate, and share sketches. Users can attach audio annotations to regions of an image. A remote server combines these annotations into a single video.
Currently I have only developed an Android version of the mobile client. The backend is a combination of PHP and Java services with judicious use of ffmpeg.
When the user is finished, he sets an email address at which to receive a response and presses a button to upload the image and annotations. The server first enhances the image, removing shadows and brightening strokes. It then creates clips that combine each region with its associated audio annotation. It combines these clips with a title screen into a single video, uploads the video to YouTube, and sends an email to the user that includes a link to the video as well as a copy of the original photo.
This video shows a finished SketchScan production.
Users can also interactively scan barcodes to attach text to their captures. When users send the captured photo and/or video to archiving services that provide an email interface (such as Evernote) this associated text can make captures much easier to find in the future.
SketchScan is described briefly in our ACM interactions paper on media bricoleurs.