What is this?

NISPS lets you teach a neural network how to map a 2D joystick position to a rich set of visual or audio parameters. Move the joystick, shape the outputs you want, and the network learns your preferences in real time.

How it works

A small neural network takes your joystick X/Y position as input and produces dozens of output parameters. You teach it by either:

• Examples — set the parameter sliders to what you want at a given joystick position, add the example, then hit Train. Do this a few times from different positions and the network interpolates between them.
• Feedback (RL) — move the joystick around. If you like what you see/hear, press + (keep this). If you don't, press − (explore more). The network gradually learns what you prefer.

What to expect

• Moving the joystick changes all outputs in real time through the network
• Randomize shuffles the network weights — instant new mapping
• Train fits the network to your saved examples
• + / − feedback nudges the network: + reinforces the current mapping, − adds exploration noise
• Switch between Visual (particle flow field) and Synth (C15 synthesizer) output modes
• The heatmap bar at the top shows all output parameters at a glance

Controls

Hand Tracking

Toggle to Hands mode in the bottom bar to use your webcam for input.

• Right hand controls parameters — palm position, finger curls, spread, rotation, and pinch map to 14 input dimensions
• Left hand gives feedback via gestures:
• 1 finger (index) held 0.4s → positive feedback (+)
• 2 fingers (index + middle) held 0.4s → negative feedback (−)
• If only one hand is visible, it's treated as the tracking hand — use keyboard/buttons for feedback
• Camera permission is requested only when you switch to Hands mode

Tips

• Start with Randomize a few times to hear/see different mappings, then use feedback to refine
• Use Follow mode to let the joystick wander automatically while you give feedback
• In Synth mode, press the play button (top left, pulses orange when audio is off) to start audio, then explore
• Tap a heatmap cell to see which parameter it controls

Synth Controls

• Presets — use the dropdown (top left, next to play) to choose a parameter preset. Lower tiers (1.x) expose fewer parameters for simpler exploration; higher tiers (3.x, 4.x) unlock cross-modulation, feedback, and the full engine. You can tweak any preset afterwards using the controls below.
• Hover any bar in the synth visualizer to see parameter name, current value, range, and curve
• Hover a group name (e.g. "Env A", "Osc B") at the top to open its control drawer:
• Group curve — drag the top graph vertically to shape the response curve for the whole group (adjusts all param curves relatively)
• Per-param curve — drag each small graph to shape that parameter individually (down = exponential, up = logarithmic)
• Min/Max range — dual-thumb slider constrains the parameter's output range
• Mute (M) — removes the parameter from NISPS control; its bar disappears and you get a manual value slider instead
• MIDI input — connect a MIDI controller and it will be detected automatically; MIDI notes trigger the synth directly
• Arpeggiator — expand the bottom bar to access the built-in arpeggiator with tempo, octave range, offset, and chord progression controls; press Pause to stop it

OSC Output

Send NISPS parameters to any OSC-capable software (SuperCollider, Max/MSP, Pure Data, TouchDesigner, Ableton, etc.) via a small bridge script that runs on your computer.

./nisps-osc-bridge

Configuration

OSC addresses

Each parameter is sent as /nisps/<Param_Name> <float 0–1>. All 126 synth parameters are available. Values reflect your current preset, curves, and tame settings. Examples:

/nisps/Env_A_Att 0.35
/nisps/SVF_Flt_Cut 0.72
/nisps/Reverb_Mix 0.15

// Listen for all NISPS params
OSCdef(\nisps, { |msg, time|
    msg.postln;
}, '/nisps/*');

// Map a specific param to a synth
OSCdef(\cutoff, { |msg|
    ~synth.set(\freq, msg[1].linexp(0, 1, 200, 8000));
}, '/nisps/SVF_Flt_Cut');

[netreceive -u -b 57120]
|
[oscparse]
|
[route /nisps]
|
[route /Env_A_Att /SVF_Flt_Cut ...]

[udpreceive 57120]
|
[OSC-route /nisps/Env_A_Att]
|
[scale 0. 1. 200. 8000.]