Cross-Disciplinary Creative Skills
The integration layer: bind audio to visuals, story to systems, data to perception - and prove it with one working artifact.
0 of 25 skills marked done · saved on this device
Start where the work stops being trivial and stops being impossible. That's your tier — jump in below.
Working tier
Apply it to your own project without a guide; debug under constraint.
L5 · Solid Control
4 activities
Educational Interactive Media
Teaching a concept demands disciplined integration of visuals, sound, and interaction with a measurable learning outcome.
Walk away with: Published p5 sketch link + a 3-line learning objective and a one-paragraph note on what the...
Build a single interactive p5.js explainer that teaches one concept (e.g. how a sine wave's frequency maps to pitch) where a slider changes BOTH the on-screen waveform AND the audible tone in real time.
Free tools only (p5.js + p5.sound, web editor); one concept, one screen; two media minimum (visual waveform + audible synthesis).
Published p5 sketch link + a 3-line learning objective and a one-paragraph note on what the learner should be able to do after.
Replicate one official Ableton Learning Music interaction (e.g. the rhythm sequencer) as your own minimal version, then write down which design choices made the concept click.
Time-box 4 hours; reproduce the teaching interaction, not the styling.
Working clone (p5/HTML) + a short critique comparing your version to the original on clarity.
Design a museum-style 'exhibit card': one tangible interaction (click/hover/keypress) that triggers a coordinated sound + visual change explaining a single fact, with an explicit success check ('the learner can now state X').
One interaction verb; must combine audio + visual; include an assessment line.
Runnable demo + exhibit card doc (concept, interaction, A/V mapping, success check).
Repair a confusing educational interaction: take a demo where the sound and visual disagree (e.g. pitch rises but the bar shrinks) and fix the mapping so the modalities reinforce the lesson.
Document the before/after mapping; keep the same concept.
Before/after clips + a problem note on why the original mapping misled learners.
L5 · Solid Control
4 activities
Interdisciplinary Collaboration & Teamwork
Cross-disciplinary artifacts die at the seams; the skill is making artist/engineer/musician handoffs explicit and versioned.
Walk away with: Repo with the contract doc, the audio asset, the visual code, and a retro on where the hando...
Run a 2-person 'role-swap' build: pair (or play both roles across two sessions) where a 'musician' delivers a stem and a 'visual dev' must build a reactive visual from it using only a written interface contract (no live explaining).
Communication only through a shared one-page A/V mapping worksheet + a Git repo; free tools.
Repo with the contract doc, the audio asset, the visual code, and a retro on where the handoff broke.
Author a reusable 'integration contract' template: a one-pager defining inputs/outputs between disciplines (e.g. audio FFT band -> named visual parameter, units, range, latency budget) and use it on a real mini-project.
Template must be tool-agnostic and version-controlled.
Filled template committed to a repo + the artifact it governed.
Set up a shared sync layer for a jam: connect Sonic Pi (audio) and a visual tool over Ableton Link or OSC so two collaborators stay in tempo without verbal coordination.
Use Ableton Link or OSC; both clients must follow one tempo source.
Screen recording showing tempo change on one client propagating to the other + a setup diagram.
Write a post-mortem on a real collaboration failure (yours or a documented one) categorizing it as 'one domain ornamental', 'brittle integration', or 'versioning broke', with a concrete prevention rule.
Use the brief's failure taxonomy; cite specific evidence.
Short post-mortem doc with the failure class, root cause, and the rule you'd adopt.
L5 · Solid Control
4 activities
Speculative Design & Critical Making
Posing a credible 'what if' artifact requires fusing design, fabrication, and a critical argument, not just a gadget.
Walk away with: Runnable prototype + a one-page critical-making note (premise, the question it raises, inten...
Build a 'diegetic prototype': a working interface for a fictional future product (e.g. a device that sonifies your daily carbon footprint) where the data drives both a sound and a visual readout.
Must function on real or simulated data; pair the artifact with a 150-word critical premise ('what does this provoke?').
Runnable prototype + a one-page critical-making note (premise, the question it raises, intended audience reaction).
Make a critique-first artifact: take a current 2026 trend (native audio-visual AI co-generation) and build a small provocation that exposes its rights/editorial risk (e.g. a generator that watermarks provenance into both image and audio).
Two media minimum; the critique must be legible without your verbal explanation.
Demo + critique notes connecting the artifact to a cited current signal.
Run a 'design fiction' worksheet: write a short newspaper-from-the-future clipping about your fictional artifact, then build the single most provocative scene from it as an interactive sketch.
Time-box 1 day; the built scene must be interactive and combine sound + image.
The clipping text + the interactive scene + a note on what reaction it's designed to spark.
Repair a 'gadget without an argument': take a flashy demo (yours or a tutorial) and rework it so it makes a clear critical point, documenting what changed from spectacle to statement.
Keep the tech; change the framing/interaction to carry meaning.
Before/after + a paragraph on how you moved from complexity to depth.
L5 · Solid Control
4 activities
Synesthetic Design (Cross-Sensory)
Systematic sense-to-sense translation (sound to color/light) is the atomic unit of every A/V integration in this domain.
Walk away with: Live sketch + the mapping table + a clip showing the same note always yields the same color.
Define and implement an explicit sound-to-color rule (e.g. pitch -> hue, loudness -> brightness, spectral centroid -> saturation) and render a live patch where a note's qualities map deterministically to a color field.
The mapping must be written down as a table first, then coded; free tools.
Live sketch + the mapping table + a clip showing the same note always yields the same color.
Recreate Hydra's audio-reactive example so that FFT bands drive distinct visual parameters (low->scale, mid->color, high->displacement) and verify each band moves the intended parameter in isolation.
Replicate first; isolate-test each band; browser-only.
Hydra code + a clip toggling each band to show clean parameter ownership.
Build a coherence test: design two competing mappings for the same sound and run a quick 5-person poll on which 'feels right', then keep the winner with a rationale.
Both mappings must use the same audio source; document the perceptual reasoning.
Two clips + poll results + a note on why the chosen mapping is more synesthetically coherent.
Repair an incoherent mapping: take a visualizer where the visual contradicts the sound (e.g. bass makes things calmer) and fix it so each modality reinforces the other; log the perceptual mistake.
Keep the source; change only the mapping.
Before/after clips + problem note.
L5 · Solid Control
4 activities
Visual Music & Audio Visualization
A visualizer that 'doesn't lie to the music' is the domain's flagship first artifact - real FFT driving honest visuals.
Walk away with: Hosted demo + a clip with audible sound showing bars responding to known frequency content (...
Build the canonical FFT-to-canvas frequency-bar and oscilloscope visualizer from the MDN guide, then swap in your own track and confirm the bars actually track the spectrum (not random motion).
Use real AnalyserNode FFT data; free tools; audible audio in the proof.
Hosted demo + a clip with audible sound showing bars responding to known frequency content (e.g. a sweep).
Compose an original 30-60s 'visual music' piece in p5.js where amplitude and at least two frequency bands drive separate visual systems, timed to a real track.
Two distinct audio features -> two distinct visual behaviors; document the mapping.
Rendered clip with audio + an A/V mapping note (which feature drives what).
Build a TouchDesigner network using the audioAnalysis palette so kick/snare detection and spectral bands drive a generated visual, proving beats land on visible accents.
Use the built-in audioAnalysis component; free non-commercial TD.
TD .toe screenshot/network export + a clip where the kick visibly hits the visual on the beat.
Repair a 'fake' visualizer: take one that animates on a timer rather than the audio, rewire it to true FFT input, and document the difference in honesty.
Prove with a silent passage where the visual must now go still.
Before/after clips incl. a silent section showing the repaired version stops moving.
Advanced tier
Teach it, template it, ship it under real constraints.
L6 · Advanced Control
3 activities
Adaptive Music & Sound for Games
Music that follows game state is composition fused with logic - the canonical middleware integration problem.
Walk away with: FMOD project + a clip sweeping the parameter showing a smooth, on-grid musical transition.
Build an FMOD Studio event with a continuous parameter (e.g. 'intensity') that crossfades between two musical layers, then prove the transition by sweeping the parameter live.
Free FMOD Studio; one parameter, two layers minimum; combine composition + logic.
FMOD project + a clip sweeping the parameter showing a smooth, on-grid musical transition.
Recreate a Wwise interactive-music structure (segments + transitions with a stinger) from the Learn Wwise 201 track and confirm transitions respect musical bars.
Follow the official tutorial first; verify bar-synced transitions.
Wwise project + a clip demonstrating a tutorial transition triggered at the right musical moment.
Author a reusable 'adaptive cue spec' template (states, parameters, transition rules, fallback) and apply it to a tiny prototype, teaching the system to another person via the doc alone.
Doc must be tool-agnostic and let a peer reproduce the behavior.
Spec template + the prototype it produced + a note from a test reader who followed it.
L6 · Advanced Control
3 activities
Audio-Reactive Lighting Design
Mapping beat/level analysis to light intensity and color is a direct sound-to-photons integration with real latency stakes.
Walk away with: Clip with audible audio where flashes hit the beat + a mapping note.
Build a virtual lighting rig (rectangles/cones as 'fixtures' in p5.js or TD) where beat detection flashes intensity and a frequency band drives color, syncing visibly to a track's downbeats.
Free tools; beat -> intensity AND band -> color (two mappings); proof must show beats landing.
Clip with audible audio where flashes hit the beat + a mapping note.
Drive real or simulated DMX/OSC fixtures from audio: emit OSC messages from your analysis so a lighting client (or a logged listener) receives intensity/color values in time with the music.
Use OSC; document the address scheme and value ranges per fixture.
OSC message log aligned to a clip + the address/range spec.
Build a latency + safety checklist for a lighting rig (smoothing to avoid seizure-risk strobing, fallback on audio dropout) and validate it against your rig.
Must address strobe-rate safety and a no-audio fallback state.
Checklist doc + a clip showing graceful behavior when audio cuts out.
L6 · Advanced Control
3 activities
Interactive Projection Art
Projection that reacts to people fuses motion graphics, sensing, and spatial calibration - a public-space integration test.
Walk away with: Clip of you interacting + a calibration note (mapping camera space to projection space).
Build a webcam-as-sensor interactive projection in TouchDesigner or p5.js where motion/blob detection makes projected graphics respond (e.g. ripples follow your hand), tuned for a flat surface.
Free tools; webcam input -> visual response; document calibration to the surface.
Clip of you interacting + a calibration note (mapping camera space to projection space).
Add a sound layer so the same interaction triggers both a projected visual AND a responsive sound, making the installation audiovisual rather than visual-only.
One trigger -> coordinated A/V; two media minimum.
Clip with audible audio showing synchronized response + a signal-flow diagram.
Write a deployable setup + calibration checklist (surface alignment, ambient light, sensor placement, participant flow) and dry-run it as if installing in a public space.
Cover space, calibration, latency, and participant behavior per the brief's core concepts.
Checklist + a setup diagram + a short note on a calibration problem you hit and fixed.
L6 · Advanced Control
3 activities
Multimedia Storytelling (Transmedia)
A coherent story across film, web, and interactive pieces needs a story bible that keeps each medium contributing, not repeating.
Walk away with: Story bible + playable Twine + the second-medium artifact, with a one-line note on what each...
Build a transmedia story bible (one canon doc) plus two small entry points in different media: a Twine branching scene AND a static web/social artifact that reveal different facets of the same world.
Two distinct media; each must add new information, not restate; free tools.
Story bible + playable Twine + the second-medium artifact, with a one-line note on what each medium uniquely contributes.
Script a branching narrative beat in Ink and embed it in a web page, then cross-link it to a second medium (e.g. an image or audio clue) so the story only fully resolves across both.
Use Ink for logic; require both media to complete the beat.
Ink file + running web build + a note on the cross-medium dependency.
Create a reusable 'transmedia matrix' template (rows = story beats, cols = media, cells = what each medium reveals) and fill it for your world to prove no medium is ornamental.
Every medium column must own at least one unique reveal.
Filled matrix + a critique flagging any cell that's mere decoration.
L6 · Advanced Control
3 activities
Real-Time Interactive Visuals
Instant visual response to input/music/sensors is the live-feedback backbone of VJ and creative-coding practice.
Walk away with: Screen recording of a 60s live steer + the Hydra code.
Build a live Hydra patch driven by both audio FFT and mouse/keyboard so a performer can steer the visuals in real time without stopping.
Two live inputs (audio + manual); must run uninterrupted; browser-only.
Screen recording of a 60s live steer + the Hydra code.
Recreate a real-time generative scene in Three.js where audio amplitude modulates a shader uniform, confirming the response is per-frame (no perceptible lag).
Replicate first; measure/observe latency; free tools.
Hosted demo + a clip showing tight audio-to-visual coupling.
Make a reusable 'VJ input router' starter: a small patch that normalizes audio + MIDI/OSC inputs to 0-1 control values other visuals can subscribe to, and document the routing.
Normalize inputs to a common range; OSC/MIDI optional but documented.
Starter file + routing doc + a clip of one input driving two different visuals.
L6 · Advanced Control
3 activities
UX Sound Design (Sonic UX)
Interface sounds must align tightly with UI actions and brand feel - audio in service of usability, not decoration.
Walk away with: Web prototype clip + a 1-page rationale on timbre/length/loudness per cue.
Design a coherent sonic set for one small UI (success, error, notification, toggle) in Sonic Pi or a DAW, then wire them to a real HTML prototype so each action plays its sound on the visible event.
Sounds must share a sonic identity; bind audio to actual UI events (two media: UI + sound).
Web prototype clip + a 1-page rationale on timbre/length/loudness per cue.
Create a sonic-logo pair: a 1.5s brand mnemonic plus a shortened earcon variant, and demonstrate them in two contexts (app open vs notification).
Derive the short earcon from the full logo; consistent identity.
Two audio files + a clip placing each in its UI context.
Build a Sonic UX checklist (timing vs visual feedback, frequency masking, mute/accessibility fallback, do-not-annoy limits) and audit an existing app's sounds against it.
Must include an accessibility/silent-fallback item.
Checklist + a short audit of a real app with at least 3 concrete findings.
L6 · Advanced Control
3 activities
Voice and Visual Interaction
Driving visuals from speech/vocal sound binds recognition or pitch tracking to real-time graphics.
Walk away with: Clip showing your voice changing the visual + code + a note on the fallback when mic is deni...
Build a browser sketch where microphone input drives visuals: use the Web Speech API for keyword commands ('red', 'faster') OR Web Audio pitch/volume to morph a generative scene.
Free, browser-only; mic input -> live visual change; handle the no-permission case.
Clip showing your voice changing the visual + code + a note on the fallback when mic is denied.
Make a vocal-pitch instrument-visualizer: humming a higher note pushes a shape up/right and changes color, proving continuous (not just on/off) control.
Continuous mapping from pitch -> at least one spatial + one color parameter.
Clip demonstrating smooth control across a pitch range + mapping note.
Document a robustness pass: list false-trigger and noise problems your voice-visual demo hits, then add smoothing/thresholds and record the improvement.
Address at least two real failure cases (background noise, accent/word miss).
Before/after clips + a problem-and-fix note.
L7 · Expert
2 activities
Alternate Reality Game (ARG) Design
ARGs weave web, print, and physical clues into one narrative - the hardest test of multi-channel coherence.
Walk away with: Live links + a tester walkthrough confirming someone solved it unaided + a puzzle-flow map.
Design and ship a 3-step micro-ARG: a public web page hiding a clue that points to a second medium (an image with steganographic/QR data) that resolves into a final Twine-based reveal.
Three media minimum (web + image/print + interactive); each clue must be solvable by a fresh tester; free tools.
Live links + a tester walkthrough confirming someone solved it unaided + a puzzle-flow map.
Author a 'trailhead + rabbit-hole' design doc and clue-dependency graph for a small ARG, validating there are no dead ends or unsolvable jumps.
Graph must show every clue's prerequisite and payoff; no orphan clues.
Dependency graph + a playtest note where a tester confirmed each hop was reachable.
L7 · Expert
2 activities
Augmented Physical Sculpture
Augmenting a real object with projection/AR/sensors fuses fabrication with digital responsiveness on a physical surface.
Walk away with: Clip of the lit object reacting to a track + a calibration note on aligning to geometry.
Projection-map a simple physical object (box, mask, paper model) so its surfaces gain dynamic light/visuals that react to sound, calibrating the projection to the real geometry.
Free TD or mapping tool; visuals must register to the object AND react to audio (two media on a physical surface).
Clip of the lit object reacting to a track + a calibration note on aligning to geometry.
Build a marker-based WebXR/A-Frame AR overlay that adds an animated digital layer to a printed image or object, viewable on a phone.
Free A-Frame; overlay must track the physical marker; document the alignment.
Phone capture of the AR overlay registered to the object + the A-Frame source.
L7 · Expert
2 activities
Collaborative Online Art Platforms
Real-time shared creation demands networked state and an interface that stays intuitive under concurrent edits.
Walk away with: Two-window screen recording of synced drawing + a note on how conflicts are handled.
Build a minimal real-time shared canvas (e.g. WebSocket or WebRTC datachannel) where two browsers draw/place elements and both see updates live, then stress it with simultaneous edits.
Two clients minimum; show concurrent edits resolving; free/open libs.
Two-window screen recording of synced drawing + a note on how conflicts are handled.
Create a shared Flok session with Hydra as the target, then co-edit and co-perform a live visual patch with one collaborator.
Use two browser sessions; define parameter ownership and document handoffs and latency.
Recording of the shared session + a roles/latency note.
L7 · Expert
2 activities
Performative Wearables & Interactive Costume
Costumes that turn movement/bio-signals into live A/V effects fuse wearable sensing with performance timing.
Walk away with: Clip of movement driving synchronized sound + visuals + a mapping note (gesture -> effect).
Use a phone's motion sensors (DeviceOrientation/Acceleration via web) strapped to the body as a stand-in wearable, mapping movement to live audio + visual effects on a connected screen.
Free; phone sensor -> A/V effect in real time; two media minimum.
Clip of movement driving synchronized sound + visuals + a mapping note (gesture -> effect).
Choreograph a 30s 'performance cue map': specify which gestures trigger which A/V states and rehearse so the effects land on intended beats, documenting timing tolerances.
Map gestures to states with timing windows; account for performance reliability.
Cue map + a rehearsal clip where effects fire on the planned moments.
L7 · Expert
2 activities
Physical Computing for Music
New tangible instruments turn sensors/microcontrollers into expressive sound control - hardware fused with sound synthesis.
Walk away with: Clip performing an audible phrase by moving the controller + a mapping doc.
Build a tangible-ish instrument: map continuous inputs (mouse position, phone tilt, or a webcam-tracked hand) to synthesis parameters in Sonic Pi or Web Audio so physical motion shapes the sound expressively.
At least two simultaneous expressive parameters; free tools (no hardware purchase required).
Clip performing an audible phrase by moving the controller + a mapping doc.
Send sensor/control data over OSC into Sonic Pi to drive synthesis, proving a documented, reusable control protocol between the 'instrument' and the sound engine.
Use OSC with a documented address/range scheme.
OSC spec + a clip where moving the control changes the sound as specified.
L7 · Expert
2 activities
Real-Time Audio-Visual Performance
Live-mixing music and visuals together is the domain's headline expert artifact - synchronized A/V under performance pressure.
Walk away with: Unbroken screen recording with audible audio showing synced A/V transitions + the patch/code...
Perform a 2-3 minute live set where Sonic Pi (audio, live-coded) and Hydra (visuals) are tempo-locked via Ableton Link/OSC so musical changes and visual changes hit together.
Both audio and visuals generated/steered live; one shared tempo source; free tools.
Unbroken screen recording with audible audio showing synced A/V transitions + the patch/code.
Build and document a performance rig as a reusable case study: signal flow, failure/fallback plan (what happens if a layer crashes mid-set), and a recovery you actually tested.
Must include a tested recovery from a mid-performance failure.
Rig diagram + case-study doc + a clip of the recovery working.
L7 · Expert
2 activities
UX Design for AR/VR
Spatial UX (gaze, gesture, controllers, comfort) is its own discipline once interaction leaves the flat screen.
Walk away with: Capture of the interaction working + a note on the chosen interaction model and comfort cons...
Build a small WebXR scene in A-Frame with one gaze-or-controller-driven interaction (select, teleport) and at least one spatial-audio cue that confirms the action.
Free A-Frame; combine a 3D interaction + spatial/positional audio feedback; runs on phone or headset.
Capture of the interaction working + a note on the chosen interaction model and comfort considerations.
Produce an AR/VR UX heuristics checklist (reachable UI placement, motion-sickness mitigation, clear affordances in 3D, audio cue redundancy) and run a teardown of an existing WebXR demo against it.
Include a comfort/safety item and an accessibility-redundancy item.
Checklist + teardown with at least 4 concrete findings.
Frontier tier
Fuse across domains and extend the field. Earned, not entered.
L8 · Cross-Domain Mastery
1 activity
Cinematic VR Experience Design
Narrative VR fuses film language (composition, continuity, pacing) with 360 space and spatial audio - two deep crafts at once.
Walk away with: Headset/phone capture + a director's note on how audio + composition steered the viewer.
Assumes prior comfort with both film/cinematography fundamentals (shot composition, continuity) AND a WebXR/3D toolchain (A-Frame/Three.js). Do the UX Design for AR/VR and Visual Music activities first.
Build a 60-90s seated cinematic VR scene in A-Frame: a 360/3D environment with a directed focal event AND positional/spatial audio that guides the viewer's attention without a forced camera.
Free A-Frame; must use spatial audio to direct gaze; no locomotion (comfort-first); document the attention-direction technique.
Headset/phone capture + a director's note on how audio + composition steered the viewer.
L8 · Cross-Domain Mastery
1 activity
Data Sonification & Visualization
Rendering one dataset as BOTH honest sound and matched graphics is the domain's signature cross-domain artifact pair.
Walk away with: Hosted chart + audio render (or live scrub clip) + a mapping/scaling doc justifying that the...
Assumes data-wrangling basics, audio synthesis (Web Audio/Sonic Pi), and a viz library (Observable Plot). Build the Visual Music and Synesthetic Design artifacts first so the sound-mapping discipline is in place.
Take one real open dataset (e.g. monthly temperatures or earthquake magnitudes) and produce a synchronized PAIR: an Observable Plot chart and a Web Audio/Sonic Pi sonification using the SAME mapping, scrubbable together.
Free tools; identical data->parameter mapping across both senses; document units and scaling so the sound doesn't misrepresent the data.
Hosted chart + audio render (or live scrub clip) + a mapping/scaling doc justifying that the sonification is honest (e.g. linear vs log choice).
L8 · Cross-Domain Mastery
1 activity
Immersive Theater & Entertainment Tech
Themed/immersive shows weave sensors, props, lighting, and media into live storytelling - systems integration as theatre.
Walk away with: Clip of the cue firing all media together + the cue map doc + a fallback note for a misfire.
Assumes working knowledge of show control (cues), audio, lighting, and sensing, plus the Audio-Reactive Lighting and Interactive Projection artifacts as building blocks.
Design and prototype one immersive 'scene beat': a sensor/input (a button, motion, or sound cue) that triggers a coordinated cue stack - lighting change + sound + projected media - and build a cue map that any operator could run.
At least three coordinated media on one trigger; deliver an operator-runnable cue sheet; free tools (TD/OSC/OBS).
Clip of the cue firing all media together + the cue map doc + a fallback note for a misfire.
L8 · Cross-Domain Mastery
1 activity
Interactive Audio-Visual Installation
Installations where audience action makes sound and image influence each other are this domain's portfolio centerpiece.
Walk away with: Clip with audible audio showing a participant changing the system and the A/V co-evolving +...
Assumes installation/space design and sound-design experience plus the Interactive Projection and Visual Music artifacts. This combines sensing, A/V mapping, and spatial calibration into one piece.
Build a closed-loop A/V installation prototype: audience input (webcam motion / mic) changes the visuals, and the visual state in turn shapes the sound (and vice-versa), so the two media genuinely influence each other rather than running in parallel.
Demonstrate bidirectional influence (not just two reactive layers); free tools; document the feedback loop and a safety/latency note for public use.
Clip with audible audio showing a participant changing the system and the A/V co-evolving + a signal-flow + calibration doc.
L9 · Visionary
1 activity
AI-Driven Multimodal Art
Generating linked image + sound from shared intent is frontier work - employable only with editorial control, eval rubrics, versioning, and rights awareness.
Walk away with: The image+audio artifact + a versioned decision log + the filled eval rubric + a rights/prov...
Visionary tier: assumes ML basics and expert command of at least one art domain. Master Visual Music + Data Sonification first. Bring editorial judgment - the value is curation, evaluation, and provenance, not raw generation.
Produce a coherent multimodal piece from ONE shared concept: generate (or AI-assist) a still image and a matching short audio bed, then hand-edit both for a unified mood - and ship it with a full provenance + rights log and a 5-criterion eval rubric you scored it against.
Free/accessible tools; the editorial decisions (what you accepted, rejected, fixed) must be documented; include prompt/process log + source-rights notes per the domain ethos.
The image+audio artifact + a versioned decision log + the filled eval rubric + a rights/provenance statement.
L9 · Visionary
1 activity
Generative Audio-Visual Systems
A system where one rule-set/data source generates synchronized sound AND visuals in real time is the domain's deepest integration and a publishable contribution.
Walk away with: Public repo with the system, a demo video (audible audio), mapping/architecture notes, and a...
Visionary tier: assumes fluent generative coding AND sound synthesis. Build the Visual Music, Real-Time Interactive Visuals, and Real-Time A/V Performance artifacts first. The aim is a reusable, open system - not a one-off render.
Design and open-source a small generative A/V system where a SINGLE source of truth (one parameter set, seed, or data stream) deterministically drives both the audio synthesis and the visual generation in tandem - same seed yields the same A/V outcome.
Shared state must provably drive both modalities (not two separate generators loosely synced); ship runnable code + README + a license; free tools (p5.sound/Three.js/Hydra/Sonic Pi).
Public repo with the system, a demo video (audible audio), mapping/architecture notes, and a determinism check showing one seed reproduces the same A/V output.
Everything Here Opens. No Cost to You.
30 curated, free-to-open references. Filter by what you need.
Canonical FFT / frequency + time-domain analysis in the browser; getByteFrequencyData, fftSize, bins.
developer.mozilla.org/en-US/docs/Web/API/AnalyserNodeFull FFT-to-canvas oscilloscope and frequency-bar examples; the reference recipe for binding audio to visuals.
developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/Visualizations_with_Web_Audio_APIBuilt-in component for low/mid/high levels, kick/snare detection, spectral centroid - ready audio features to drive visuals.
docs.derivative.ca/Palette:audioAnalysisFrequency spectrum of input channels; the core node for audio-reactive TD networks.
docs.derivative.ca/Audio_Spectrum_CHOPAdaptive game audio: parameters, instruments, transitions. Brief's Foundational Reference.
fmod.com/docs/2.03/studio/authoring-events.htmlInteractive music structures, segments, transitions; free Audiokinetic learning track.
audiokinetic.com/en/learning/learn-wwise/wwise201Recipes for branching/interactive narrative; brief's Foundational Reference. (Bot-blocks fetchers; canonical and live.)
twinery.org/cookbookConcise JS data-viz grammar (ISC license); the visualization half of a sonification+viz pair. Brief's Foundational Reference.
observablehq.com/plotBrowser-based, AGPL; audio-reactive functions + webcam/stream inputs. Repo: github.com/hydra-synth/hydra.
hydra.ojack.xyzFree code-based live music; MIDI/OSC + Ableton Link networked metronome.
sonic-pi.netp5.FFT exposes waveform + spectrum bins for binding sound to sketch parameters.
p5js.org/reference/p5.sound/p5.FFTFree creative-coding sketchbook (Java/Python modes); Minim/Sound libs for audio-reactive work.
processing.orgNode-based real-time A/V; free non-commercial license. The integration-layer workhorse.
derivative.caFree/OSS capture, scene compositing, streaming - proof-capture for any A/V rig.
obsproject.comFree/OSS 3D, animation, geometry nodes, VSE; doubles as a generative + render engine.
blender.orgFree/OSS HTML + entity-component framework for VR/AR in the browser, any headset or phone.
aframe.ioFree interactive browser lessons: rhythm, harmony, structure - sense of music timing for A/V.
learningmusic.ableton.comFree PDF; Hermann/Hunt/Neuhoff - the foundational text for turning data into sound.
sonification.de/handbookFree JS 3D/WebGL incl. WebXR + TSL shaders; the visual engine for generative + immersive work.
threejs.org/docsFree MIT narrative language for branching/transmedia story logic; Inky editor + engine integrations.
inklestudios.com/inkWiring adaptive audio into engine/game state via the Studio API.
fmod.com/docs/2.03/api/studio-guide.htmlActive TD network sharing, debugging, audio-reactive patterns.
reddit.com/r/TouchDesignerp5.js / Processing / shader sketches and critique.
reddit.com/r/creativecodingGenerative + algorithmic art, including audio-driven systems.
reddit.com/r/generativeOfficial tutorials, projects, and AI/audio-reactive walkthroughs.
derivative.ca/communityLive visuals / VJ rigs, Resolume, real-time A/V performance.
reddit.com/r/vjingThe interoperable standard underpinning browser audio analysis + synthesis.
w3.org/TR/webaudioStandard for VR/AR access on the web; basis for cross-headset XR UX.
w3.org/TR/webxrMessage protocol linking audio, visuals, lighting, sensors across tools.
opensoundcontrol.stanford.edu/spec-1_0.htmlAccessibility baseline; multi-sensory work must not strand users on one channel.
w3.org/TR/WCAG22Keep your progress
Create a free account to sync completed skills across devices. The library stays free, and you can keep learning without one.
We couldn't request a link. Try again in a moment.
Check your inbox
If that address can receive email, a sign-in link is on the way. Open it to sync your progress.