23 Nov 2017

Javascript Audio Visualization pt1, Web Audio API

Ever wanted to use javascript to create rich audio visualizations?

Ever wanted to use javascript to create rich audio visualizations?

I’ve been interested in electronic music ever since I was about 12 years old, and I sure do miss the old days of winamp visualization plugins. I found them fascinating to just sit and watch for hours on end. The artistry was on full display, but so was the raw technical skill. I used to think that something that complex would never be achievable in web browsers with javascript, but times have changed.

The web browser has evolved, introducing new APIs and capabilities that are starting to more closely resemble an operating system than a browser. Tools like Backbone.js, Ember.js, and React have completely reimagined what building frontend applications looks like. Libraries like D3.js and p5.js have shown us that we can render complex graphical information like never before. All the while, transpilers and toolchains like CoffeeScript, webpack, Babel, and asm.js have pushed the boundaries of what “javascript” even means. At this point, people have starting implementing crazy stuff like Quake, or even an entire Linux kernel, all in javascript.

Basically, the time is right for some seriously cool experimentation.

All the examples used in this article can be found in my exploring-audio-vis github repository

Getting Started

For this project, I decided to use React with Uber’s react-vis library. The reason for this choice is simply that these are the tools I’ve been working with a lot during at Synack, so I’ve grown pretty familiar with them. I’m also using Storybook, simply because it makes testing components in isolation very simple, and provides most of the webpack/build configuration I need out of the box.

Utilizing the Web Audio API

(Much thanks to Patrick Wied)

Before we can even begin to look at data visualization, we need to be able to extract the frequency data from the audio file we want to play. The tool that makes this possible is the new Web Audio API’s AudioContext. To construct it is easy enough.

const audioContext = new AudioContext()

Next, we need to bind our <audio /> element to our new AudioContext to produce a MediaElementAudioSourceNode.

const audioSource = audioContext.createMediaElementSource(audioPlayerRef.audioEl)

Here, audioPlayerRef is the instance of ReactAudioPlayer we use to control audio playback, and audioEl is an instance property of that component that represents an <audio /> element.

Next, we create (and cache) our audioAnalyser from our newly created audioContext

this.audioAnalyser = audioContext.createAnalyser()

Finally, we connect() our audioSource to our audioAnalyser, as well as the target audioContext destination.

Note: Failing to connect() your audioSource to audioContext.destination will result in no sound during playback.

audioSource.connect(this.audioAnalyser)
audioSource.connect(audioContext.destination)

Note: There already exist some libraries that wrap the new Web Audio API in React. While writing thig blog, I came across react-audio. I’m sure there are more.

Now that we have our audioAnalyser, all we need to do is query it for the frequency data currently being played at any given point in time. To do this, we invoke the getByteFrequencyData, and pass it a properly sized Uint8Array to inject the data into.

const frequencyData = new Uint8Array(this.audioAnalyser.frequencyBinCount)
this.audioAnalyser.getByteFrequencyData(frequencyData)

At this point, we can query the audioAnalyser at any time to extract the frequency data. Running console.log(frequencyData) produces something that looks like this: [ 137, 172, 187, 176, 143, 120, ... ]

Polling the audioAnalyser

We need to continually poll the audioAnalyser for new data as our audio component continues to play. While javascript provides setTimeout and setInterval for time-based operations, it’s better in this case to prefer requestAnimationFrame simply because we intend to render the data that the audioAnalyser produces.

If you’re unfamiliar with requestAnimationFrame and why it’s used for rendering-specific tasks, check out the MDN article on it.

onAudioFrame = () => {
  if (!this.state.playing || !this.audioAnalyser) return
  const frequencyData = new Uint8Array(this.audioAnalyser.frequencyBinCount)
  this.audioAnalyser.getByteFrequencyData(frequencyData)
  doSomethingWith(frequencyData)
  requestAnimationFrame(this.onAudioFrame)
}

Invoking this function will cause onAudioFrame to continue to be invoked at every available animation frame as long as this.state.playing is true, and this.audioAnalyser is defined.

Building a React Component

This section assumes you are familiar with basic React concepts. Comments for this component are inline. Check out the latest source for bugfixes/errata.

import React, {
  PureComponent
} from 'react'

import PropTypes from 'prop-types'

import ReactAudioPlayer from 'react-audio-player'

export default class AudioAnalyser extends PureComponent {

  static propTypes = {
    ...ReactAudioPlayer.propTypes,
    onFrequencyData: PropTypes.func.isRequired
  }

  // Track our playing state
  state = { playing: false }

  // Analyse a single "frame" of audio
  onAudioFrame = () => {
    // Don't do anything if we're paused, or if we don't have an analyser
    if (!this.state.playing || !this.audioAnalyser) return

    // Create a new Uint8Array to inject the frequency data into
    const frequencyData = new Uint8Array(this.audioAnalyser.frequencyBinCount)

    // Inject the frequency data into the array
    this.audioAnalyser.getByteFrequencyData(frequencyData)

    // Invoke our `onFrequencyData` prop
    this.props.onFrequencyData(frequencyData)

    // On the next animation frame, repeat the process
    requestAnimationFrame(this.onAudioFrame)
  }

  // Handle the audio player ref once it's rendered
  onAudioPlayerRef = audioPlayerRef => {
    if (!audioPlayerRef || !audioPlayerRef.audioEl) return

    // Allow the audio element to read data from dubious sources
    audioPlayerRef.audioEl.crossOrigin="anonymous"

    // Interface with the Web Audio API
    const audioContext = new AudioContext()
    const audioSource = audioContext.createMediaElementSource(audioPlayerRef.audioEl)
    this.audioAnalyser = audioContext.createAnalyser()

    // Connect our audio source to the analyser
    audioSource.connect(this.audioAnalyser)

    // And also to the audio destination
    audioSource.connect(audioContext.destination)
  }

  // Create a simple interface for tracking play state
  onUpdatePlaying = (playing, callback) => (...args) => {
    // Figure out what the current/new play state is
    const isCurrentlyPlaying = this.state.playing
    const shouldNowBePlaying = playing

    // Cache the new play state
    this.setState({ playing })

    // Start tracking the audio frequencies if we started playing
    if (!isCurrentlyPlaying && shouldNowBePlaying) {
      this.onAudioFrame()
    }

    // Invoke the callback with whatever args we were passed
    if (typeof callback === 'function') {
      return callback(...args)
    }
  }

  render() {
    return (
      <ReactAudioPlayer controls
        { ...this.props }
        ref={ this.onAudioPlayerRef }
        onAbort={ this.onUpdatePlaying(false, this.props.onAbort) }
        onEnded={ this.onUpdatePlaying(false, this.props.onEnded) }
        onPause={ this.onUpdatePlaying(false, this.props.onPause) }
        onPlay={ this.onUpdatePlaying(true, this.props.onPlay) }
      />
    )
  }

}

Conclusion

In this article we covered basic implementation and strategy for extracting audio frequency data from a playing audio file, and provided a component that performs the task using React conventions. In the next article we’ll start to build a simple data visualization. From there we’ll format and aggregate the data to match our data visualization’s component interface. Hopefully this won’t end up like my last blog series, and I actually complete it.