SketchScan: Breathing life into everyday drawings

SketchScan is an application for mobile devices that lets you capture, clean, animate, and share sketches. Users can attach audio annotations to regions of an image. A remote server combines these annotations into a single video.

Currently I have only developed an Android version of the mobile client. The backend is a combination of PHP and Java services with judicious use of ffmpeg.

1

After taking a photo of a sketch, the user can swipe the screen to define a region, immediately after which the app opens a dialog to let the user attach an audio annotation (either spoken to text-to-speech). After attaching audio notes to several regions the user can return to an overview screen to review his annotations. Below, the user plays an annotation generated using Google's automated text-to-speech engine.

2

When the user is finished, he sets an email address at which to receive a response and presses a button to upload the image and annotations. The server first enhances the image, removing shadows and brightening strokes. It then creates clips that combine each region with its associated audio annotation. It combines these clips with a title screen into a single video, uploads the video to YouTube, and sends an email to the user that includes a link to the video as well as a copy of the original photo.

This video shows a finished SketchScan production.

3

Users can also interactively scan barcodes to attach text to their captures. When users send the captured photo and/or video to archiving services that provide an email interface (such as Evernote) this associated text can make captures much easier to find in the future.

SketchScan is described briefly in our ACM interactions paper on media bricoleurs.

4

While not related to SketchScan directly, I like Sunni Brown's argument that to doodle or sketch is "to make spontaneous marks to help yourself think." SketchScan augments this process, helping others (as well as your future self!) see how you came up with ideas through the marks you made.