Real-time spectrogram (and other audio-related data) visualization using Marsyas and OpenCV

Visualizing the output of Marsyas networks can be a tricky thing, because data is streamed in real time. I have found that a fast way to do that is by using python's OpenCV bindings, so that we can view, for example, a spectrogram being streamed in real time. For this to work, you will basically need to get data from a mrs_realvec to a numpy array. To do that, after you have created the network you will use:

 
net.tick()
out = net.getControl("mrs_realvec/processedData").to_realvec()
out = numpy.array(out)

That means that you will need to have: import numpy to get that functionality. You will have to pre-define a 2-dimensional numpy array that will store past values of your spectrogram. In our solution, we want the spectrogram to flow from right to left, hence we will need a 2-dimensional array where the columns represent time and the lines represent frequencies. The array may be initializes as:

 
Int_Buff = numpy.zeros([DFT_size, nTime])

where DFT_size is the size of your DFT and nTime is the number of time sample you want to store. After getting the out array, you should remove the first column of Int_Buff and append the new data to the last position. Before doing so, it is necessary to add a dimension to out and transposing it (this is due to the way numpy is implemented - one-dimensional arrays cannot be appended to two-dimensional array, and the conversion assumes the output is a line array, which is not what we want at this point). So, we will have:

 
if numpy.ndim(out)==1:     # If out is a 1-dimensional array,
 out = numpy.array([out]) # convert it to 2-dimensional array
Int_Buff = Int_Buff[:,1:] # Remove first column of Int_Buff
Int_Buff = numpy.hstack([Int_Buff,numpy.transpose(out)]) # Transpose / horizontal stack 

From that, you may yse the function array2cv(), defined in http://opencv.willowgarage.com/wiki/PythonInterface, to convert from a numpy array to cv's image format. Of course, to deal with that you will need to have:

 
import cv
im=array2cv(Int_Buff)

Remember that before dealing with images you will need to create a window where things will be displayed. For that, use:

 
cv.NamedWindow("Marsyas Spectral Analysis", cv.CV_WINDOW_AUTOSIZE)

So, the following lines tell OpenCV to show your data:

 
cv.ShowImage("Marsyas Spectral Analysis", im)
cv.WaitKey(10)

If you only do the steps above, you will probably get a black screen. You will want to normalize your output array before stacking it to your memory, using:

 
out = out/numpy.max(out)

Also, you may notice that, so far, the bass frequencies are on top while the trebles are on the bottom of the screen. That is the reverse of what we usually want. We will need to reverse the order of the output array, using:

 
out = out [::-1]

All of these ideas are coded in the spectral_analysis.py utility, already in the Marsyas repository. The actual implementation adds some other utilities, for example, the possibility of trimming the spectrogram so that only a certain frequency range is shown. The current program is an example implementation, and may be expanded for other uses, if necessary. If you just want to see how your voice's spectrogram looks like, try it:

 
python spectral_analysis.py

Installing the Marsyas Python bindings on Windows 7

In this post I assume you have followed the instructions from the previous post about installing Marsyas on Windows 7. To install the Python bindings you need to perform the following steps (I used Python 2.6 but I suspect that things should also work with Python 2.7). The python bindings to Marsyas do not require NumPy/SciPy and Matplotlib to work however having these package enables the creation of beautiful plots similar to MATLAB that help a lot with prototyping.

1. Install Python 2.6 using the MSI Installer 
2. Install NumPy using the superpack binary numpy-1.6.1-win32-superpack-python2.6.exe
3. Install SciPy using the superback binary  scipy-0.10.0rc1-win32-superpack-python2.6.exe
4. Install Matplotlib. I used matplotlib-1.1.0.win32-py2.6.exe from the Download link of the matplotlib website. 

Now you should be able to get nice MATLAB looking graphs in Python. To test start the Python command-line (or IDLE or whatever other method you have of editing Python code) and do:

>>> from pylab import *;
>>> x = randn(1000);
>>> hist(x);
>>> show() 

You should see a histogram plot of a rough bell-shape curve that will look something like this:

Congratulations you have a working Python/NumPy/SciPy matplotlib system. 

The next step is to install swig that is the program that generates the Python bindings. Download the archive that contains a pre-built executable for windows: http://prdownloads.sourceforge.net/swig/swigwin-2.0.4.zip Unzip the file. The next step will help ensure that swig and python can be found by CMake and consists of editing your PATH environment variable to point to the Python executable as well as the swig executable.

Right-click on Computer then select Properties and then select Advanced System Settings and Environment Variables.Under System Variables you will see the Path. Edit the path to add Python2.6 and swig.exe to it.

For example my Path looks like:
%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\ProgramFiles\Internet Exporer;C:\Program Files\CMake 2.8\bin;c:\Program Files\Microsoft SQL Server\100\Tools\Binn\;c:\Program Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\TortoiseSVN\bin;C:\Python26;C:\swigwin-2.0.4;

 Now we are ready to configure Maryas with the Python bindings using CMake. This can be done by ticking the WITH_SWIG option. If everything gets detected correctly the pink options will turn into grey and swig and python will be detected. This is how it looks on my machine:



 

Click generate and the Microsoft Visual Studio C++ Express solution file will be updated accordingly. 

The last part requires that you run Visual Studio as an administrator in order to be able to install the bindings and corresponding DLL to the default Python location for modules. To do that right-click on the icon for Visual Studio and select Run as Administrator. If needed confirm that you want to run as administrator and reopen the marsyas.sln file from the Marsyas build directory.

Right click on the ALL_BUILD target and select build to compile Marsyas. When all projects are finished then right click on the INSTALL target and select build.





Now all the Marsyas related stuff is installed. Check that the python related entries at the end go to the right directory especially if you have multiple version of Python installed.

Now you are ready for the final step. Start the Python 2.6 command-line and type:

import marsyas 

If there are no problems the Marsyas python bindings are installed. Now you are ready to explore the code in src/marsyas_python. For example you can load src/marsyas_python/windowing.py into Idle and click F5 to run. You should see a nice looking plot demonstrating the effect of windowing to the spectrum of sinusoid signals that are not aligned with the DFT bins.



Happy hacking !!!