Show Audio Information While Playing SoundHack





Show Audio Information While Playing SoundHack

Providing visual feedback for JavaSound audio, or at least trying to….

You might want to play a clip without any corresponding visuals; for example, if you were using it to signal the end of a long-running process such as uploading a file. On the other hand, if the sound is the focus of the application, as in a music-player application, you might need to show the user some information about the audio he's playing.

The Code

You already know how to play audio from a file or stream [Hack #76]; building on that, you can create a simple GUI that shows some of the basic traits of the audio, by pulling fields out of the AudioFormat object, which can be retrieved from the Line once it has been created. These fields include the audio format, bits/sample, frame size and rate, and endianness (which indicates how two-byte values are to be interpreted: big-endian means the first byte is more significant, and little-endian means the second is).

More impressively, DataLine provides a getLevel() method that returns the current level of the audio being played, as a float from 0.0 (silence) to 1.0 (maximum volume). You can use this to create a graphical level meter by getting the level and coloring in that percentage of a component. For example, if the level is 0.5, you'd fill in half of the component.

Drawing this level meter is pretty straightforward: create a JPanel whose paint() method clears the Graphics, gets the line level, and fills a rectangle starting at (0,0) with a height equal to the component's height and a width equal to the level times the component's width. Then you need to set up an animation loopa javax.swing.Timer is convenient because it avoids any thread-safety issues while doing the paintingto repeatedly call repaint() on the meter.

Combine this together and you have the DataLineInfoGUI, seen in Figure. Note that to play the audio, it uses the PCMFilePlayer class from the previous hack, so you can use an arbitrarily long AIFF or WAV, as long as its contents are uncompressed PCM data.

Displaying audio format information
import javax.sound.sampled.*;

public class DataLineInfoGUI extends JPanel {

	PCMFilePlayer player;
	JButton startButton;

	public DataLineInfoGUI (File f) {
		super();
		try {
			player = new PCMFilePlayer (f);
		} catch (Exception ioe) {
			add (new JLabel ("Error: " +
				 ioe.getMessage()));
            return;
		}
		DataLine line = player.getLine();
		// layout
		// line 1: name
		setLayout (new BoxLayout (this, BoxLayout.Y_AXIS));
		add (new JLabel ("File: " +
                         player.getFile().getName()));
		// line 2: levels
		add (new DataLineLevelMeter (line));
		// line 3: format info as textarea
		AudioFormat format = line.getFormat();
		JTextArea ta = new JTextArea();
		ta.setBorder (new TitledBorder ("Format"));
		ta.append ("Encoding: " +
                   format.getEncoding().toString() + "\n"); 
        ta.append ("Bits/sample: " +                   
				   format.getSampleSizeInBits() + "\n"); 
		ta.append ("Endianness: " +                   
                  (format.isBigEndian() ? " big " : "little") + "\n");       ta.append ("Frame size: " + 
                   format.getFrameSize() + "\n");
		ta.append ("Frame rate: " +         
                   format.getFrameRate() + "\n");
        add (ta);

		// now start playing
		player.start();
	}
	public static void main (String[] args) {
		JFileChooser chooser = new JFileChooser();
		chooser.showOpenDialog(null);
		File file = chooser.getSelectedFile();
		DataLineInfoGUI demo =
			new DataLineInfoGUI (file);

		JFrame f = new JFrame ("JavaSound info");
		f.getContentPane().add (demo);
		f.pack();
		f.setVisible(true);
	}
	class DataLineLevelMeter extends JPanel {
		DataLine line;
		float level = 0.0f;
		public DataLineLevelMeter (DataLine l) {
			line = l;
			Timer timer =
				new Timer (50,
				   new ActionListener (){
				   public void actionPerformed (ActionEvent e) {                                   
						level = line.getLevel();
						repaint();
					} 
				});
            timer.start();
		}
		public void paint (Graphics g) {
			Dimension d = getSize();
			g.setColor (Color.green);
			int meterWidth = (int) (level * (float) d.width);
			g.fillRect (0, 0, meterWidth, d.height);

		} 
	} 
}

This is a pretty straightforward implementation of the strategy sketched out previously: the class is a JPanel with a BoxLayout to which you can add an arbitrary number of rows. The first is the name of the file, the second is the level meter, and the third is a JTextArea to which you can append various fields pulled from the AudioFormat.

The level meter's constructor takes care of setting up its own repaint callbacks, so there's no babysitting required on the part of the caller. All that's left for the constructor is to start the player to begin feeding bytes to the Line.

Testing It Out

Launch the DataLineGUI application and you'll get a file-selection dialog. Choose a suitable AIFF or WAV, and you'll see the GUI shown in Figure.

Audio player display with format information


This is all well and good for a simple GUI, but there's one problem: where the heck is our level meter?! It should be between the filename and the text area, but it's totally not there!

Initially, I suspected my repaint code was hosed, but it all seemed correct. So, right after figuring out the meter width, I added a sanity-check debug line:

System.out.println ("level = " + level);

And when I ran it, I got a result that I really didn't want to see:

	[aeris:HacksBook/Media/x11] cadamson% java DataLineInfoGUI
	got PCM format	
	got info
	got line
	opened line
	level = 0.0
	level = 0.0
	level = 0.0
	level = 0.0	
	level = 0.0

And that was on a really loud song, so it wasn't just a slow fade in. I looked around to see if there was something special you have to do for getLevel() to work, but there wasn't.

Then I Googled, and found this post to the javasound-interest mailing list from February 2003:

Date:

Mon, 17 Feb 2003 22:31:21 -0800

Reply-To:

Discussion list for JavaSound API

 

<[email protected]>

Sender:

Discussion list for JavaSound API

 

<[email protected]>

From:

Florian Bomers <[email protected]>

Organization:

Sun Microsystems Inc.

Subject:

Re: DataLine.getLevel( )?

Comments: To:

[email protected]

Content-Type:

text/plain; charset=us-ascii


Unfortunately, it is not implemented. (actually, in my private opinion, it is a questionable method anyway: usually soundcard drivers do not provide such a primitive, so the Java Sound implementation has to calculate this "level" on its own. But there are many different algorithms to do so, suited depending for what the "level" is needed for, and it would possibly eat unnecessarily processor resources. So I guess it's best if everybody does the calculation of the "level" on his own on the buffers received by the TDL or written to the SDL, respectively. Easy and fast algorithms are maximum, moving average,block average, power).

sorry…
Florian

Knute Johnson wrote:
>
> Anybody know if DataLine.getLevel() is implemented?  All I get is 0.0
> on SourceDataLines and -1.0 on TargetDataLines.
>
> Thanks,
>
> Knute Johnson

In fact, a little further research shows that the fact that DataLine.getLevel() always returns UNKNOWN_LEVEL was filed as bug 4297101 in the Java Bug Parade on December 6, 1999. Five years later, it's still not fixed, though it looks like there was at least an attempt to fix it for Tiger (J2SE 5.0)a fix that was abandoned in August 2003.

By the way, wouldn't it have saved a lot of people a lot of time if they disclosed in the Javadoc that this method is a noop? But I digress


So, the level meter is not going to worknot because of the graphics, but because there's no way to get an accurate level. Or is there?

Hacking the Hack

Florian's message to javasound-interest says it is best if "everybody does the calculation of the 'level' on his own[, based] on the buffers received by the TDL [(TargetDataLine, usually used by capture devices)] or written to the SDL [(SourceDataLine)], respectively."

Setting aside the argument of duplication of effort, note that the buffers he speaks of are available in the hack code; it's what the PCMFilePlayer reads from the file and writes to the Line (specifically, a SourceDataLine, as Florian's message notes). So, in theory at least, this can be done. But it's not going to be pretty.

First, create a new DataLineInfoGUI2 class that is identical to the one from earlier in this hack, except that instead of using a PCMFilePlayer, it uses a PCMFilePlayerLeveler, a class that will be defined next.

This new class is pretty much the same as the old PCMFilePlayer, except that on each time through the while loop, as it reads the buffer and writes it to the line, it will call a method to scan through the buffer and determine a level for this group of samples. So, after reading the bytes from the input stream but before writing them to the line, add:

	// calculate level
	calculateLevel (buffer, readPoint, leftover);

As Florian argues in his message, the idea of a level is up for interpretation, but there is a general sense that it should represent the loudness or quietness of the audio at a certain time. Making the problem worse is the fact that the sample values will always be going up and down because the samples represent how much a speaker should be excited or relaxed, and it's the sample's periodic change that creates sound waves we hear. Put another way, even the loudest sounds can have some 0 samples at the bottom of their waves.

As a crude attempt at approximating a level, this hack's implementation gets the maximum amplitude (on either speaker, if the source is stereo) in the entire buffer. To make this a little more fine-tuned, this version of the player figures out a buffer size suitable to provide 1/20 of a second of audio, rather than the flat 32 KB used earlier. To do that, add this after getting the Line in the constructor:

	// figure out a small buffer size
	int bytesPerSec = format.getSampleSizeInBits() *
                      (int) format.getSampleRate();
    System.out.println ("bytesPerSec = " + bytesPerSec);
	int bufferSize = bytesPerSec / 20;
	buffer = new byte[bufferSize];

This needs to sync with the line as wellif the line's buffer is nearly full, it won't accept this entire buffer on the write() without blocking. So, you can tune the while loop to do a read-and-write only if the Line will accept a bufferful of data. Do this by adding the following block after the if (playing) statement:

	// only write if the line will take at
	// least a buffer-ful of data
	if (line.available() < buffer.length) {
		Thread.yield();
		continue;
	}

Now, the only problem is implementing calculateLevel()i.e., doing the actual iteration through the buffer to calculate a maximum value. This, frankly, is a huge pain in the butt, because to determine each sample value, you have to deal with four issues you hadn't cared about before:

  • Channels (i.e., mono versus stereo)

  • Sample size

  • Endianness

  • Signing

This is handled in the calculateLevel() method of PCMFilePlayerLeveler, listed in Figure.

Method to calculate a crude "level" of sample bytes in a buffer
private void calculateLevel (byte[] buffer, 
				 int readPoint, 
				 int leftOver) {
   int max = 0;
   boolean use16Bit = (format.getSampleSizeInBits() == 16);
   boolean signed = (format.getEncoding() ==
                     AudioFormat.Encoding.PCM_SIGNED);
   boolean bigEndian = (format.isBigEndian());
   if (use16Bit) {
	   for (int i=readPoint; i<buffer.length-leftOver; i+=2) {
	   int value = 0;
	   // deal with endianness
	   int hiByte = (bigEndian ? buffer[i] : buffer[i+1]);
	   int loByte = (bigEndian ? buffer[i+1] : buffer [i]);
	   if (signed) {
		   short shortVal = (short) hiByte;
		   shortVal = (short) ((shortVal << 8) | (byte) loByte);
		   value = shortVal;
	  } else {
		  value = (hiByte << 8) | loByte;
	  }
		  max = Math.max (max, value);
	  } // for
   } else {
	  // 8 bit - no endianness issues, just sign
	  for (int i=readPoint; i<buffer.length-leftOver; i++) {
		  int value = 0;
		  if (signed) {
		      value = buffer [i];
          } else { 
		      short shortVal = 0; 
			  shortVal = (short) (shortVal | buffer [i]); 
     		  value = shortVal;
          }
		  max = Math.max (max, value);
	  } // for
   } // 8 bit
   // express max as float of 0.0 to 1.0 of max value
   // of 8 or 16 bits (signed or unsigned)
   if (signed) {
       if (use16Bit) { level = (float) max / MAX_16_BITS_SIGNED; } 
	   else { level = (float) max / MAX_8_BITS_SIGNED; }
   } else {
       if (use16Bit) { level = (float) max / MAX_16_BITS_UNSIGNED; }
	   else { level = (float) max / MAX_8_BITS_UNSIGNED; }
   } 
} // calculateLevel

This crude implementation just reads all the samples in order, meaning the stereo casesamples alternating between left and rightis ignored. Thus, the maximum value wins, regardless of what channel it's on.

Figuring out the value is still a bit-munging pain because of the three outstanding issues that must be dealt with. For 16-bit audio, the samples should be read two at a time. You arrange the "high" (most significant) and "low" (least significant) bytes based on the endianness of the format, and then cast to a Java int or short based on whether you need to maintain the sign bit (in a 32-bit int, the 16 bits won't be signed; in Java's 16-bit short, the sign will be maintained). Eight-bit audio spares you the endianness hassle, though you still have to be aware of signage, and cast to a byte or short based on whether you need to preserve a sign.

All of this, just to figure out the value of a sample. As you might expect, the only thing left to do on each loop is to compare the sample's value to the maximum for this buffer, and to reset the maximum if this value is higher. At the end, you divide the maximum value against the maximum possible value for that combination of bits and signage to get the level as a value between 0.0 and 1.0.

Running the Hacked Hack

When you run this hack, you finally get a player with a level meter, as seen in Figure.

Audio player display with format information and level meter


While this looks OK in a book, it really isn't very satisfactory when you're watching the audio as it plays. It doesn't seem to relate to the music that closely; that is, it seems to follow softer music OK, but it really falls apart on rock music.

Part of the reason is that this "maximum" algorithm is quite crude; an approach such as averaging the samples in the buffer might be more realistic.

But the real problem is that the access JavaSound gives you is doomed to be hopelessly behind what's being played. Think about the available() method, which reports how much you can write to the SourceDataLine's buffer without blocking. What's happening is that you're refilling one end of its buffer, while it drains out the other end to the speakersor more accurately, to the native sound system (which may have its own buffers, and thus more latency). This arrangement is illustrated in Figure.

Flow of samples through buffers in JavaSound


So, you can calculate the level for the samples in the buffer, but it will be some time until those samples are played, so you have a mismatch of what's being measured and what's actually being played. What you need is access to the SourceDataLine's buffer, so you could run the level check on the bytes that are just about to be played. Until and unless that's available, the suggested workaround isn't really going to work.

Of course, Sun could just go and actually implement getLevel()…wouldn't that be nice?


     Python   SQL   Java   php   Perl 
     game development   web development   internet   *nix   graphics   hardware 
     telecommunications   C++ 
     Flash   Active Directory   Windows