From Drawing to Sound: Creating a Visual Instrument in Python

I’ve always been fascinated by the intersection between art, code, and sound — the moment where a line drawn on a screen can become a musical gesture.

So when I set out to create an interactive synthesizer that turns drawings into sound, I faced a question that many audio developers encounter: should I build the sound engine from scratch using digital signal processing, or should I leverage sample-based synthesis to focus on the musical interaction itself?

Instead of choosing one over the other, I decided to explore both paths, resulting in two complementary versions of the Geometric Synthesizer — one focused on sound design, the other on performance and play.

©Alexas Fotos

Evolution from Shapes to Gestures

The first version — which I later called the Exploratory Edition — began as an experiment: could I build a complete synthesizer from scratch, using nothing but Python and math?

With NumPy and sounddevice, I implemented custom oscillators, ADSR envelopes, and real-time audio processing. Initially, I envisioned a system where geometric shapes — circles, squares, triangles — would trigger instruments, creating a visual-sonic vocabulary.

Users could draw a circle to hear piano, a square for guitar, a line for flute. Every waveform — sine, square, sawtooth, and triangle — was generated sample by sample, giving me complete control over the synthesis pipeline. It wasn’t just about making sound, but about understanding how sound is built — how simple mathematical curves can turn into music.

This seemed elegant in theory.

However, as the project evolved, I recognized that while procedural synthesis offered educational value and technical depth, it presented challenges in creating an intuitive musical interface.

Early attempts to map geometric shape recognition to instrument selection proved unreliable — distinguishing circles from ovals, or squares from rectangles, required complex algorithms that often misclassified user intent.

More importantly, I realized that shape recognition introduced an unnecessary layer of abstraction between intent and result.

If a user wanted to play piano, they had to remember which shape corresponded to piano, then carefully draw that shape correctly, hoping the algorithm would classify it properly.

Geometric Synthesizer v1

This cognitive overhead made spontaneous musical expression nearly impossible. The most natural musical interaction wasn’t about what shape you drew, but rather the continuous gestural motion itself — the vertical movement of your hand directly controlling pitch, like playing an invisible string instrument in the air.

That realization led me to develop a second version, the “Musical Edition,” which pivots from shape recognition to direct gestural control.

The Challenge: From Shape Recognition to Gestural Control

I redesigned the system around continuous gestural control — no more shape recognition, geometric analysis, or guessing user intent. The link between movement and sound is now immediate, predictable, and musically intuitive.

With FluidSynth and professional SoundFont libraries integrated, the canvas becomes a vertical musical keyboard: the Y-axis maps directly to pitch (higher near the top, lower near the bottom), while the X-axis controls stereo position (left sounds left, right sounds right).

This shift from discrete shape classification to continuous spatial control completely redefined the project. It’s no longer about drawing geometric forms, but about performing through motion — treating the canvas as a two-dimensional instrument where every gesture carries musical meaning.

Architectural Decisions: Non-Blocking Polyphony

For the Musical Edition, I designed a three-component architecture that cleanly separates concerns and enables true polyphonic playback.

The AudioEngine component wraps FluidSynth and manages note lifecycles using timestamp-based tracking instead of blocking delays. When a note is triggered, the engine stores its start time and expected duration in an active notes dictionary, then immediately returns control to the main loop.

This architecture allows dozens of notes to play simultaneously, while the user keeps drawing new shapes in real time. It’s what makes the system feel alive — responsive, layered, and continuously reactive to gesture.

def play_note(self, midi_note, velocity, duration, pan, instrument, is_drum):
    """Trigger a note without blocking playback."""
    channel = 9 if is_drum else 0
    self.synth.noteon(channel, midi_note, velocity)

    # Store timing info for later release
    self.active_notes.append({
        'midi': midi_note,
        'channel': channel,
        'start_time': time.time(),
        'duration': duration,
    })

def update(self):
    """Stop notes whose duration has expired."""
    now = time.time()
    for note in list(self.active_notes):
        if now - note['start_time'] >= note['duration']:
            self.synth.noteoff(note['channel'], note['midi'])
            self.active_notes.remove(note)

The ShapeAnalyzer translates spatial properties into musical parameters — vertical position into pitch, gesture length into duration, and horizontal position into stereo pan. It also manages scale quantization, snapping notes to the active key when enabled.

The GeometricSynth component ties everything together, running the main event loop, handling input, and coordinating analysis with audio playback.

This modular architecture keeps responsibilities clean: the audio engine plays sounds, the analyzer interprets gestures, and the main app orchestrates their interaction. It’s what makes the system easy to test, extend, and evolve — adding MIDI export or quantization required only local changes, without touching core audio or geometry logic.

Spatial Mapping: Turning Canvas into Keyboard

With robust audio architecture and intelligent gesture analysis in place, the creative challenge became designing the spatial-to-musical mappings that would determine the instrument’s musical character.

Vertical Axis: Pitch

The Y-axis maps to a five-octave range from C2 (65 Hz) to C7 (2093 Hz). This range was carefully chosen:

• Wide enough for melodic expressiveness
• Narrow enough for precise pitch control with mouse/touch
• Covers the comfortable singing range plus instrumental extensions

The mapping is inverted to match music notation conventions:

y_normalized = 1.0 - (y_pos / screen_height) # Top = 1.0, Bottom = 0.0 
midi_note = MIN_MIDI + y_normalized * (MAX_MIDI - MIN_MIDI)

Horizontal Axis: Stereo Pan

Rather than mapping X-position to time (which flows automatically during playback), I mapped it to stereo positioning:

pan = x_pos / screen_width # 0.0 = left, 0.5 = center, 1.0 = right

This creates spatial depth in the audio field. Draw on the left side, hear sound from the left speaker. Draw on the right, hear it from the right. This transforms the canvas into a three-dimensional sonic space where melody (Y), harmony (simultaneous notes), and spatial positioning (X) all contribute to the musical texture.

The pan system can be toggled on/off with the P key, allowing users to choose between stereo depth and centered mono output.

Continuous Live Feedback: Playing While Drawing

The most significant innovation in the Musical Edition is the continuous live drawing mode. Instead of pressing discrete keys, users draw to generate sound in real time — like a theremin or fretless string instrument, where pitch responds instantly to vertical motion.

The challenge was managing a sustained note that evolves continuously. Triggering a new note for every small movement would create chaos, so I built a smart note manager that:

• starts a note when drawing begins,
• restarts only after a ≥1 semitone pitch change,
• modulates velocity from drawing speed,
• and ends cleanly on release.

Geometric Synthesizer v2 — In free mode, every stroke’s position, color, and motion directly shape the sound and instrument choice.

The result is fluid, expressive control — the line itself becomes both gesture and melody.

Velocity follows the same principle: the faster you draw, the louder the sound. It’s a direct, physical link between motion and musical energy.

def update_live_sound(self, pos):
    """Update pitch and velocity during continuous drawing."""
    midi_note = self.analyzer.shape_to_midi({'center': pos})

    # Restart only if pitch shifts by ≥ 1 semitone
    if self.last_live_pitch is None or abs(midi_note - self.last_live_pitch) >= 1:
        if self.active_live_note_id:
            self.stop_live_sound()

        velocity = self.current_velocity  # From drawing speed
        pan = pos[0] / self.width if self.pan_enabled else 0.5

        self.active_live_note_id = self.audio.play_note(
            midi_note, velocity, duration=10.0, pan=pan,
            instrument=self.audio.current_instrument, is_drum=False
        )
        self.last_live_pitch = midi_note

The way velocity is calculated is key to how the instrument feels. Instead of assigning a fixed velocity to every note, the system measures drawing speed in real time — tracking the pixel distance traveled per frame.

Faster gestures produce louder notes, slower gestures result in softer dynamics. This creates a direct link between physical motion and sonic energy, much like how acoustic instruments respond to a performer’s touch.

def shape_to_duration(self, shape): 
 """Convert trace length to note duration""" 
 length = shape['length'] # Total Euclidean distance 
 duration = max(0.1, min(3.0, length / 100)) 
 return duration 

def shape_to_velocity(self, shape): 
 """Convert trace length to velocity with gamma curve""" 
 length = shape['length'] 
 normalized = min(1.0, length / 100.0) 

 gamma = 0.6 # Response curve (<1 = emphasizes small differences) 
 normalized = normalized ** gamma 

 # Interpolate from base velocity to maximum (127) 
 velocity = int(self.velocity_base + (127 - self.velocity_base) * normalized) 
 return max(30, min(127, velocity))

The gamma curve emphasizes small/medium differences while preventing large gestures from overpowering the mix.

This continuous feedback loop transforms the experience from “drawing shapes that play back later” to “performing music in real time.” Players instantly hear whether their gestures are too fast or too slow, whether their pitches are accurate, whether their movements form the melody they imagine.

The instrument becomes responsive — alive — almost conversational.

Multi-Note Gesture Detection: Melodic Phrases from Single Strokes

While continuous feedback worked beautifully for pitch exploration, users naturally began drawing melodic phrases — smooth gestures spanning multiple pitches, like a violinist sliding along the fingerboard. They expected each stroke to form a contour, not a single tone.

This led to one of the project’s most advanced features: automatic multi-note segmentation. The system analyzes each gesture for pitch variation and splits it into sequential notes, each tied to a local peak or valley in the path.

The algorithm follows two main steps:

1. Flatness check: if vertical variation is small (≈ 35 px), treat the gesture as one sustained note.
2. Smoothing + direction change detection: smooth Y-values (moving average = 5) to reduce tremor noise, then mark each change in direction as a potential note boundary.

In short: ignore flat gestures, smooth the motion, detect reversals, and turn each extremum into a distinct note.

In practice, it looks like this:

def extract_note_segments(self, shape):
    """Detect melodic segments from direction changes"""
    points = shape['points']
    y = [p[1] for p in points]

    # Skip nearly flat gestures
    if max(y) - min(y) < 35:
        return None

    # Smooth Y values
    window = 5
    smoothed = [np.mean(y[max(0, i-window):i+window+1]) for i in range(len(y))]

    # Detect direction flips
    extrema = [0]
    for i in range(window, len(smoothed) - window, window):
        if (smoothed[i] - smoothed[i - window]) * (smoothed[i + window] - smoothed[i]) < 0:
            if abs(smoothed[i] - smoothed[extrema[-1]]) > 25:
                extrema.append(i)
    extrema.append(len(points) - 1)

    # Convert to note segments
    return [
        {'midi_approx': self.analyzer.shape_to_midi({'center': points[i], ...}),
         'x_pos': points[i][0], ...}
        for i in extrema
    ]

This gives the system a kind of musical intuition — it recognizes when a single stroke contains multiple expressive pitches and turns it into a coherent phrase.

By ignoring flat gestures, smoothing noise, and detecting direction changes, it transforms wavy lines into melodic contours. The effect feels almost magical — a single gesture becomes a melody.

Instrument Organization: 108 Sounds Across 12 Presets

Moving from shape-based to gesture-based control freed instrument selection from geometric constraints. Users now choose instruments directly via number keys (1–9), with instruments organized into 12 thematic presets: Classical, jazz, rock, electro, Latin, country, soul, world music, drum kit, percussions, Latin drums, and miscellaneous.

Users cycle presets with arrow keys; each instrument has a unique color, and the current selection is shown for instant visual feedback.

This organization provides 108 distinct timbres while maintaining simple navigation. The preset categories help users quickly find the sonic palette they need — romantic classical, gritty rock, ethnic world sounds, or precise rhythm percussion.

Quantization, Tempo and other features

While free-form spatial positioning allows expressive timing, many musical contexts require rhythmic precision. A jazz drummer needs to stay on the beat. A electronic producer wants perfectly aligned loops. To support these use cases, I implemented a comprehensive quantization and tempo system.

Quantized playback in action — turning freehand drawings into rhythmically structured music.

Tempo Control

Users can cycle through standard tempos (60, 80, 100, 120, 140, 160, 180 BPM) using the T key. The tempo determines the duration of one beat, which serves as the foundation for all rhythmic calculations:

def get_beat_duration(self): 
 """Get duration of one beat in seconds""" 
 return 60.0 / self.bpm # e.g., 80 BPM = 0.75s per beat

Quantization Divisions

The D key cycles through note divisions: 1/2 notes, 1/4 notes, 1/8 notes, 1/16 notes (default), and 1/32 notes. The quantization step represents the temporal grid to which notes will snap:

def get_quantize_step(self): 
 """Get quantization step in seconds based on current division""" 
 beat_duration = self.get_beat_duration() 
 return beat_duration / (self.quantize_division / 4) 
 # Example: 80 BPM, 1/16 notes = 0.75s / 4 = 0.1875s grid

Visual Grid

Pressing G toggles a visual grid overlay that displays vertical lines at each quantization interval. Stronger lines mark beat boundaries, lighter lines mark subdivisions. This provides visual feedback about where notes will snap when quantization is enabled.

Quantized Playback

When quantization is enabled (Q key), the playback system modifies its behavior:

Event timing rounds to the nearest quantization step:

if self.quantize_enabled: 
 quantize_step = self.get_quantize_step() 
 quantized_time = round(start_time / quantize_step) * quantize_step

Stable grid visualization: When playback starts, the grid parameters (pixels_per_second and quantize_step) are frozen and saved. This ensures the visual grid remains aligned with the actual playback timing throughout the entire performance, preventing disorienting visual shifts mid-playback.

# In start_playback() - freeze grid parameters 
self.playback_pixels_per_second = pixels_per_second 
self.playback_quantize_step = self.get_quantize_step() 

# In draw_grid() - use frozen parameters during playback 
if hasattr(self, 'playback_pixels_per_second'): 
 pixels_per_second = self.playback_pixels_per_second 
 quantize_step = self.playback_quantize_step

Loop alignment: In quantized mode, loops snap to complete beat boundaries (the “thick lines” on the visual grid). If your composition spans 2.7 beats, the loop will extend to 3 full beats before repeating, creating clean, measure-aligned repetition. In free mode, loops wait for all notes to finish naturally before repeating.

if self.quantize_enabled: 
 beat_duration = self.get_beat_duration() 
 last_event_end = max(e['time'] + e['duration'] for e in events) 

 # Round up to next complete beat 
 beats_needed = math.ceil(last_event_end / beat_duration) 
 loop_duration = beats_needed * beat_duration

This system transforms the free-form drawing canvas into a metronomic sequencer when precision is needed, while preserving expressive rubato timing when quantization is disabled. The flexibility to toggle between modes makes the instrument suitable for both improvisational exploration and structured composition.

Scale Quantization The scale-lock feature (M key) snaps pitches to the nearest note in the selected key signature (C, G, D, F, Am, Em). This ensures harmonic coherence even with imprecise drawing:

def snap_to_scale(self, midi_note): 
 """Snap MIDI note to current scale""" 
 octave = midi_note // 12 
 semitone = midi_note % 12 

 # Find closest note in scale (e.g., C major = [0,2,4,5,7,9,11]) 
 closest = min(self.scale_lock_scale, key=lambda x: abs(x - semitone)) 

 return octave * 12 + closest

Musicians can quickly switch keys with the K key, exploring different tonal colors while maintaining scale coherence. When disabled, the full chromatic range becomes available for atonal or chromatic passages.

MIDI Export: Sharing Gestural Compositions

With 108 instruments organized into thematic presets, users can craft full arrangements directly on the canvas.

To make these creations shareable, the system supports exporting gestures as standard MIDI files. Each trace’s pitch, velocity, and timing are captured, producing a single-track, type-1 MIDI file.

This allows compositions to be opened in any DAW for further editing, instrument changes, or playback, bridging the gap between live gestural performance and professional production.

Lessons Learned & Future Enhancements

1. Keep It Direct — I spent too long perfecting shape recognition before realizing it added confusion, not creativity. The real joy comes when gestures instantly make sound — draw, and you hear it.
2. Smooth Polyphony — Non-blocking note playback was a turning point. Any delay kills musical flow, so tracking notes by timestamp keeps the performance responsive.
3. Balancing Multi-Notes — Deciding when a stroke becomes multiple notes took trial and error. Too sensitive, and gestures fragment; too loose, and melodies blur. The balance came only by playing and listening.
4. Quantization as a Choice — Adding optional quantization turned free sketches into rhythmic loops and grooves. Flexibility beats restriction.
5. Visual Stability — Locking the grid during playback made everything feel grounded. Small details like this dramatically improve perceived quality.
6. Context Matters — Loop behavior adapts: in quantized mode, it snaps to beats; in free mode, it breathes naturally. Different modes, different musical intentions.

Looking ahead, there are so many directions I’m excited to explore! I’d love to let users save and load their compositions, move or edit individual traces, and even record their sessions directly to WAV or MP3.

Conclusion

The Geometric Synthesizer V2 (aka the Draw-to-Sound Interface) shows that the most intuitive musical interfaces come from direct, physical relationships between gesture and sound — not from clever algorithms. By turning the canvas into a continuous performance surface, the instrument feels like an extension of the hand, not a puzzle to solve.

Under the hood, non-blocking polyphony, smart gesture segmentation, flexible quantization, and MIDI export make this expressiveness possible without forcing artificial constraints. The result is an instrument that’s responsive while improvising, precise when composing, and approachable for both musicians and non-musicians.

This project reminded me of a simple truth: technology should amplify human expression, not dictate it. When a gesture immediately produces rising pitches, when a melodic contour is heard as a coherent phrase, when a rhythm locks to the beat perfectly — the technology disappears, and the music simply happens.

Project Repository: [GitHub URL]

Live Demo Video: [YouTube URL]