At the end of the semester, students in the Industrial Mathematics class presented their work on using a neural network to compose music in the style of J.S. Bach.
This past semester, 11 Colorado State University master’s and Ph.D. students composed a suite of musical pieces that sound strikingly like they were written by J.S. Bach, the prolific German composer of the Baroque period.
These weren’t music students, though. They were math students.
The course was Industrial Mathematics, and the students were challenged by their professor, Michael Kirby, to apply a mathematical theory to a practical problem: teaching a computer to compose music with specific style, rhythm and structure parameters. To do it, the students used a type of machine-learning algorithm familiar to any iPhone owner who uses Siri, Apple’s voice recognition software.
What does composing music have to do with voice recognition software? It boils down to a type of neural network, called an LSTM – a long short-term memory network. Together, the class read an academic paper that explains the background of the LSTM algorithm and how it works, then Kirby asked them to reproduce the results. The students chose a Bach dataset, and created computer code to back up their results.
An LSTM is like a brain being trained by feeding it data – in this case, a repertoire of 129 chorales from the J.S. Bach canon. Unlike other neural networks, the LSTM has a series of gates that lets it learn relevant information and disregard what is irrelevant to the task at hand. Making predictions and being corrected over and over by the original dataset, the neural network learns, identifies and eventually corrects its own mistakes.
Learning like Siri
Patrick Rosse, a master’s student in the class, explained that this type of neural network is what Apple uses in the Siri software, which takes into account previous words spoken and makes decisions about what was probably said. Each time the user talks to Siri, the recognition gets better and more accurate.
“Another example I’ve seen is that someone fed [an LSTM] a bunch of lyrics from the rapper Fabolous and it ended up writing a verse that retained his rhyming scheme,” said Rosse, a former math teacher who is also studying LSTMs as part of his thesis. “The LSTM is good at remembering relevant things, and forgetting irrelevant things.” Yet another internet-popular example of the LSTM’s power: Using it to spit out a book chapter that reads very much like it could be part of the Harry Potter series.
At the end of the semester, the class presented their results, walking through the processes they used and the challenges they encountered. Examples of the music, in midi format, were played for the class.
Kirby is an expert in pattern analysis who became intrigued with LSTMs after reading the relevant paper.
“I want students to have exposure to the types of technologies being used in the world,” Kirby said. He said he wants them to be able to answer the question, “Can you apply the math you have learned to make discoveries?”
Broad range of uses
Kirby explained that the LSTM neural network is applicable to a wide variety of problems and datasets, and he is exploring whether LSTMs could be useful for studying a broad range of data that have more than one time-dependent variable. Music is one example of such a multivariate time series.
Currently, Rosse is working with Kirby on a Department of Defense-funded project to study whether the circadian rhythm of a mouse could be used to predict infection from certain pathogens. “In this case, the mouse’s circadian rhythm is being remembered, like the structure of the music,” Kirby said.
Ten years ago there was less student demand for a course like Industrial Mathematics, Kirby said. Yet the field of data science is growing, and the College of Natural Sciences recently launched its first data science major. “I think the field is recognizing that more and more fundamental questions are mathematical in nature,” Kirby said.
Even composing music that sounds like Bach.
