Data Knot is the best of AI and machine learning: low-latency, optimized, performance-friendly, responsive — a set of building blocks for gestures and sounds that you customize. Built on the Fluid Corpus Manipulation project (FluCoMa) by composer/artist and self-described “crazy person” Rodrigo Constanzo, it’s something else musicians and artists love, too: it’s free.
Built for Max (more on Pd below), you get a collection of low-latency, real-time machine learning tools:
Works with any audio input, “including optimizations for Sensory Percussion sensors and ordinary drum triggers.”
Low-latency onset detection, onset-based, and real-time descriptor analysis, classification and clustering, regression and prediction, corpus analysis and querying.
Synthesis and modeling, audio, data, and CV processing.
Basically, drum and bang on stuff, play, sing, bang, hook up voltages, gesture, dance in front of a camera, do all that and turn them into inputs you can use and analyze in your Max patches, all with performance that doesn’t make you miss a beat. (Yes, both CV as in computer vision and CV as in control voltage are applicable here!)
Data Knot is built around the FluCoMa Toolkit and requires v1.0.8 or higher to be installed for this package to work. You’ll find the full Data Knot in the Max Package Manager (which, like Pd’s deken, has made life so much easier).
But the bottom line is, all of this lets you be a musician, with, like, musical instruments, which to most people — even children and novices — is the main draw of any music or interaction and AI.
And this is awesome. It opens up a ton of interesting musician/patcher collaborations, too. It feels almost like a return to the heydey of human-computer interaction and why it was fun in the first place, only now the group of people want to make crazy sounds has grown to the general population, not just a few privileged people in nice private college laboratories.
Machine learning at human scale
Part of what makes AI confusing is that it’s a catch-all term for a lot of different processes. But AI for music, when it’s built by people who don’t seem to really like music all that much, winds up… well, wrong. It’s slow, hard to customize. Generative AI focuses on non-realtime rendering with few parameters, a black box with as little expression and customization available, so that it runs autonomously. All other complaints about genAI aside, and there are many, it’s the exact opposite of everything we want.
I’m generalizing more than a little. There are also a lot of really interesting machine learning processes that are nonetheless still difficult to configure or that don’t run well in real-time. And that’s what makes Data Knot so cool. Everything here is optimized for low latency and real-time performance.
The stuff that interests big data, big money, and big tech the least is what interests musicians the most. We’ve seen this for years, back to the brilliant Wekinator and research by Rebecca Fiebrink, which remains a great way to understand machine learning. (See also projects like ml5.js.) Far from blindly cheerleading for big tech and AI as used in surveillance capitalism, teaching machine learning this way can demystify what machine learning actually is.
How this developed, artist collaborations
You don’t tie yourself in a Data Knot overnight. Rodrigo has been at this for years. He tells CDM more about how this came about:
I’ve been using Data Knot for all my performances/videos over the last 5 years (the development period of Data Knot). Here are some relevant projects/videos/performances.
An ongoing series on improvisation with conversations/interviews embedded called `play talk play`, where even when my personal instrumentation changes, there’s nearly always an electronic component using Data Knot.
If anyone thinks pop music has gotten dumber, I think all the complexity went to the play talk play series. Actually, this could be the future of pop music. Let’s go! (chosen slightly at random, there are dozens of great artists…)
And there’s more:
Some hardware explorations around a Landscape NOON, the goal of which is to embed a bela (or gem more realistically now) in a CNC’d enclosure with the NOON so it becomes a standalone machine learning-powered instrument:
Even when the project is oriented around a different idea or approach (ambisonic feedback), Data Knot is still involved at some point in terms of analysis and processing:
Pd users, yes, there’s hope of this eventually working across both platforms, which I think would be terrific for community, collaboration, and teaching (teaching both inside and outside institutions). A start to that:
I did make pd versions of many of the core abstractions in the previous version of Data Knot (SP-Tools), primarily to be able to embed some of the code onto a bela using pd.lib (this vid is using pd as I was prepping the code for bela embedding)
That little jam is fresh. Yeah, let’s get a crack Pd team on this, too.
Rodrigo’s compositional history
I love that Rodrigo opened up his own practice and collaborations; the musical process around this is as compelling as the software. It’s almost as though the software was just a catalyst for finding new ways for people to play together — a way to challenge habits and open up even deeper human listening and improvisation. So it’s helpful to listen in as he explains the composition that led to this, and that in turn informed the FluCoMa project:
I was one of the first five composers commissioned at the very start of FluCoMa, so a large part of what Data Knot became was birthed through those early experiments of both FluCoMa and me. At the very start, the package was much more limited in scope as this was around 1 year into the project, so the tools I had to work with didn’t have the depth of what they would later, but it did lead to a lot of components that are still present in it now (mainly super-optimized onset detection and corpus-based sample playback).
Here’s the performance of the piece that I made for the FluCoMa commission (Kaizo Snare):
I suppose some of the biggest takeaways are that optimizing for low latency at every step of the way is the kind of rabbit hole that you can easily get lost in (which I’m currently writing this email from still). Machine learning, broadly speaking, and machine listening, more specifically, are fairly well established fields even in the world of music. So a lot of the ideas, improvements, and developments within Data Knot have been around optimizing existing processes and creating workflows to leverage those use cases in a musical, powerful, and intuitive way.
Actually, this may also come as a surprise to those just hopping on the “AI” bandwagon as they see massive investment on the big data side: the most effective tech works as well as it does because it’s mature, not novel. This process of developing listening techniques and optimizing them for performance has involved decades of work by computer scientists, musicians, and people straddling those and other fields, working collaboratively and testing the technology in performance.
Do Knot
But enough talking: this is one to play with. Max patchers, I’ll be eager to see what you come up with. And maybe these duos and so on are something other venues want to replicate. After a lot of patching alone — a pleasantly solitary experience, it’s true — you do sometimes want to then go out and play with people.
And what’s all this about Descriptors? Rodrigo is also a great teacher:
And just out this month, v1.0 — here’s what’s new if you saw an earlier version (and a glimpse into how this has developed):
Okay, someone’s interactive Halloween setup next week is going to be lit.
Looking for somewhere to start with Max before you get into this? The Max for Live series I’m doing this fall might be what you need! And you could use the two… together. Oh, yes.