“In 2016, after being inspired by a video of the Scanman Line Follower on YouTube, I started work on a synthesizer device using the Toshiba TCD1304 linear CCD to synthesize audio from spectrogram data (or graphical data interpreted as spectrogram data) using Michel Rouzic’s ARSS code (the source of his Photosounder application). This became overly bulky, hardware-wise, and really didn’t work as a standalone controller, so I put it on the back burner.
Recently I became aware that the sensors made by Agilent for optical computer mice do a lot of processing internally, both being able to provide a bitmap image (very slowly) and average darkness along with the change in X and Y using simple serial requests (much faster) rather than having to deal with a high-speed analog to digital conversion like the Scanman / Toshiba sensor. So, I decided to make a simplified version of the CCD synth using a mouse instead of the scanner. By modifying an Arduino library developed by Conor Peterson for reading pixel data from the Agilent sensor to read movement and average darkness I was able to grab the data fast enough for a simple yet responsive standalone gestural synthesizer
The components in this device can be purchased for less than ten dollars and the code is simple enough for almost anyone to modify, making this a quick and inexpensive sound maker for performance or as a prank.
Using the software below, the scrollwheel button switches between modes: 1 - pitch based on X-position, 2 - pitch based on camera input, 3 - a mix of the two. The left mouse button is a momentary trigger and the right is latching. The scrollwheel changes the frequency range and scrollwheel plus left button changes the center point of that range. Left button plus center button toggles volume modulation on the Y-axis.”