This Amazing Machine Lets You ‘Paint’ Mozart’s Music

Move over, Fantasia—a composer has figured out how to translate Mozart and Bach into moving artworks.

11.07.15 5:01 AM ET

Music is supposedly a universal language, a kind of transcendent tongue that encodes meanings mere words cannot capture. Hans Christian Andersen famously voiced this idea in the 19th century—“where words fail, music speaks”—and his remark is just one of dozens of variations on a theme.

When NASA sent the Voyager hurtling deep into space in 1977, it included the so-called “Golden Record,” essentially a mixtape of humanity’s greatest musical compositions, in the hope that alien life-forms might understand our species through music. This gesture reveals something important about our belief in the transcendent reach of music—perhaps its power is such that even vastly different beings could appreciate it.

Opposite the first page of my score of Bach’s Goldberg Variations is a quote on music from the 17th-century polymath Sir Thomas Browne: “In brief, it is a sensible fit of that harmony which intellectually sounds in the ears of God.” For Browne, music’s universality encompassed even the divine realm.

Music may speak to deities and extraterrestrials alike, but what exactly is it saying? Attempts to translate music into fallen human language tend toward one of two extremes: poetic rapture or the technical-mathematical. Consider this description of Beethoven’s Fifth Symphony from E. M. Forster’s novel Howard’s End: “Gusts of splendor, gods and demi-gods contending with vast swords, colour and fragrance broadcast on the field of battle, magnificent victory! Magnificent death!” Novelists are not the only ones to get excited when writing about music; the cognitive scientist Douglas Hofstadter evokes a Chopin étude like this: “To me, it was a soft, rushing whisper of notes, a fluttering, like the leaves of a quaking aspen in a gentle breeze. Yet it was not just a scene of nature; it expressed a human longing, a melancholy infused with strange and wild yearnings for something unknown and remote.”

A harmonic analysis of a Bach suite from a cello blog lies at the opposite extreme: “The G major harmony in the second half of bar 25 is this of a tonic and changes at the bar line to bar 26 to a Subdominant function in the temporary key of D major.” Though perhaps not as forbidding as the austere, insect-like squiggles of black ink on a page of sheet music, this is far from the lush accessibility of metaphor and image. Walt Disney’s original 1941 film “Fantasia” did not simply show scrolling footage from the musical scores of Bach, Beethoven, and Stravinsky for good reason. The vaguely psychedelic images of flying ponies and dehydrated dinosaurs may correspond less precisely to music, but they are far more visually interesting.

Somewhere between these extremes is a technology called the Music Animation Machine, invented by composer and programmer Stephen Malinowski. His software translates the primary formal elements of music—pitch, rhythm, dynamics, harmony, timbre, etc.—into a visual language of interconnected lines, colors, and shapes. Each visual feature models a particular element of the music; the size and position of the shapes model rhythm and pitch, for instance, which helps make the structure of even dazzlingly complex compositions intuitively apparent. A rising flurry of green circles might signify an ascending melodic line in a Mozart sonata. The dense polyphony of a Bach fugue can be clearly decomposed into constituent strands of red, orange, and green triangles that loop and intertwine. The result is a visual rendering of music that’s both poetic and precise, a way of mapping sounds onto images that combines the mathematical and the artistic.

Malinowski has made over 350 YouTube videos with the Music Animation Machine; collectively they have been viewed over 140 million times. His animations have been projected on massive screens during live performances at Carnegie Hall, and his work forms part of a permanent display at MOMA featuring Björk’s music. His aspirations, however, do not center on concert halls and museums: he wants to make the usually rarefied realm of classical music broadly accessible to a wide audience.

“The main thing I’ve learned is that comprehension increases enjoyment,” he told me at his East Bay home, where the available space in two full rooms has been colonized by computers, cables, pianos, musical scores, and electronic keyboards. “You could understand nine out of ten things in a piece but not like it until you get that tenth element. And when you don’t like a piece you literally don’t know what you’re missing.” This is especially true of famously complex and difficult works, like Stravinsky’s Rite of Spring or Beethoven’s Grosse Fuge, both of which he has animated. The videos help to make apparent the patterned intricacies that give such pieces their tremendous power.       

Malinowski traces the inspiration for the Music Animation Machine to an afternoon in the early 1970s when he took LSD.  He was at a friend’s house listening to a recording of Bach’s Chaconne in D-minor for solo violin, following along with a printed score of the music. A music composition major in college, he was in his early 20s at the time. Neither studying scores while listening to music nor taking mind-altering drugs were unusual activities for him, but this time the conjunction of the two produced an extraordinary experience.

What happened next is hard to describe. Malinowski uses a technical term: saccade synchronization. Saccades are the tiny, jerky motions of the eyes as they switch focus between points. Their “synchronization” gave him the feeling of possessing supernatural visual powers. “It was like I could direct traffic at some neurological level,” he said. “Wherever the ‘now’ moment in the music was, I could track it perfectly. There was a sense of motion in the score, a dancing feeling, as if the notation was dancing to music. It was an intense and wonderful experience.  Not the sort of thing that would happen without taking drugs.”

The Music Animation Machine approximates Malinowski’s drug-induced episode of preternatural visual acuity, making it available to anyone with Internet access. Of course not everyone watching his videos will have the same subjective experience that a highly-trained composer on drugs once did. Malinowski thinks the videos only approach what he experienced during the episode, which has never happened again.

Yet his vision of music was not simply the eccentric whim of a drug-addled mind. He has developed a formal system of rules governing the correspondence between notes and images, a way of making the deep grammar of music visually comprehensible. This precision and rigor separate his idea from other visual fantasies inspired by music.

Get The Beast In Your Inbox!
By clicking "Subscribe," you agree to have read the Terms of Use and Privacy Policy
Thank You!
You are now subscribed to the Daily Digest and Cheat Sheet. We will not share your email with anyone for any reason

In the 1970s the technology did not exist for him to program and disseminate his software, and he had no training in programming. All he could do was chart on graph paper an early prototype of the Music Animation Machine, which he still keeps rolled up in his office, an analog trace of what is now a digital technology. The precise geometry and rich color scheme give it the air of an illuminated medieval manuscript.  In the 1990s he sold mail-order videos of his animations, but the technology came into its own with the rise of YouTube in the early 2000s. He has now retired from his day job as a programmer and devotes himself full-time to the animations.

One hundred and forty million YouTube views provide one sort of validation; he also gets a steady supply of notes that range from requests for future animations to testimonials. A music appreciation teacher in San Francisco told him that he used to teach the basic elements of music in three classes and expected roughly half the class to grasp them. With the videos, over 90 percent of students understood the material after a single class.

Malinowski’s girlfriend, Christine, is a retired teacher from Kansas who used to work with special needs children at an elementary school. She and Stephen began corresponding after she used his videos to get more kids interested in classical music. “I would always play recordings of Mozart or Bach on the anniversaries of their birthdays. I like to get the kids painting as they listened to the music. These were fourth-grade boys, so I was competing with soccer and other sports,” she said. “They were not that interested until they watched the videos. Once they saw them, they couldn’t get enough. It was fascinating to watch them do their own paintings after seeing Stephen’s videos. You could tell just by looking at the paintings the difference between the ones that Mozart and Bach inspired.”

Malinowski thinks that musically-inspired graphical design is in its artistic infancy. He’s currently collaborating with Swiss musician Etienne Abelin and a Swiss company, iart, to develop an intuitive user interface that would allow anyone to make their own visualizations of music. On his website he has a funny chart that condenses human history into two columns. For fashion, he shows a caveman clad in furs in one column and a suave man in a tailored suit in the other. For architecture, he juxtaposes a simple straw hut with the soaring interior of a cathedral. The last entry is for music visualization, which he places firmly in the hides and huts phase of progress. In one column he shows a frame from one of his videos; in the other column, the future of the field, he leaves a question mark.