Prototyping Code in Clojure – A Simple Audio Displaying App

By | February 24, 2008

Clojure is a new Lisp, created by Rich Hickey. The primary portal to all things Clojure is here on Source Forge.

I just got through prototyping a little app that takes in sound samples, and displays a stacked series of frequency spectra, and a 2d ‘sonogram’ type display. The running app looks like:

Frequency Spectra in Clojure

Not so easy to see, but the lower part of the screen is a 2d sonogram.

With the help of the KJ DSP library (which has a homepage on SourceForge), the app runs in real-ish time, and does a creditable job of displaying frequency spectra.

I hoped to try and get it running on Windows too, but the Java sampling api under Windows has defeated me for now. The code below works on a Mac Book Pro. Nothing else is tested.

The code is split into two main parts, which deal with getting sound samples, and with displaying a simple gui. I’ve reproduced the code below.

Note that this code isn’t complete, since I have a separate jar KJ.JAR which has the FFT code, and a NetBeans front end that provides isometric.gui (controls: start, stop, test:JButton, inputs:JCombobox, isopanel, sonopanel:JPanel).

; The following file is written by Jonathan Watmough
; This file is free for any use, providing this notice is preserved.
;
; This file works with the sound system on a Mac Book Pro.
; Nothing else is tested.



; imports
(import '(java.lang Thread))
(import '(java.nio ByteBuffer ShortBuffer))
(import '(javax.sound.sampled DataLine AudioSystem LineEvent LineListener AudioFormat))
;(import '(javax.sound.sampled.DataLine Info))

; supported audio filetypes
(def filetypes (. AudioSystem (getAudioFileTypes)))
(print (str "Supported audio: " (seq filetypes)))
(newline)

; supported mixers
(def mixer-info (seq (. AudioSystem (getMixerInfo))))

; get mixer-info, name, description of each mixer
(def mixer-info-list
  (map #(let [m %] { :mixer-info m 
                     :name (. m (getName))
                     :description (. m (getDescription))}) mixer-info))

; 
;---- user interface ----

; create a desired pcm audio format
; -> float sampleRate, int sampleSizeInBits, int channels, boolean signed, boolean bigEndian
(def format (new AudioFormat 44100 16 1 false true))
(def buffer-size (* 22050 2))   ; 44k = 1/2 sec x 2 bytes / sample mono

; set a ref for the open line
(def input (ref nil))

; from each mixer-info, get the target data lines (inputs)
(def input-lines
  (mapcat #(let [mix-info (:mixer-info %)
                 mixer    (. AudioSystem (getMixer mix-info))
                 targets  (. mixer (getTargetLineInfo))]
              (map #(let [target-info     %
                          get-line    (fn[] 
                                      (let [mixer (. AudioSystem (getMixer mix-info))
                                            line  (. mixer (getLine target-info))]
                                      (do (. line (open format buffer-size))
                                          (. line (start))
                                          (sync nil
                                            ; ### should close old input - When @input ...
                                            (set input (. mixer (getLine target-info)))
                                            (. @input (start))))))
                          mixer-name  (. mix-info (getName))]
                      {:mixer-name  mixer-name
                       :get-line    get-line}) targets)) mixer-info-list))

; get a set of samples for level-checking in ui
; assume we are dealing with signed 16 bit samples (short = 2 x bytes)
; may want to sync on 'input'
(defn get-inst-samples ([num-samples]
  (do 
    (if (not (. @input (isOpen)))
      (. @input (open format buffer-size)))
    (if (not (. @input (isRunning)))
      (. @input (start)))
    (let [bytes   (make-array (. Byte TYPE) (* 2 num-samples))
          bcount  (. @input (read bytes 0 (* 2 num-samples)))
          bbyte   (. ByteBuffer (wrap bytes))
          bshort  (. bbyte (asShortBuffer))
          shorts  (make-array (. Short TYPE) num-samples)
          ]
      (do (print "Read: " (str bcount) " bytes.")
          (newline)
          (. *out* (flush))
          (. bshort (get shorts 0 num-samples))
          shorts)))))







(comment

; "Built-in Microphone"  <-- we'll use this

; get the mixer info for the mic
(def mic-mixer-info
  (:mixer-info (first (filter #(= "Built-in Microphone" (:name %)) mixer-info-list))))

; get the built in mic mixer
(def mic (. AudioSystem (getMixer mic-mixer-info)))

; get the supported source and target lines for the mixer
(def sources (seq (. mic (getSourceLineInfo))))   ; nil
(def targets (seq (. mic (getTargetLineInfo))))   ; (interface TargetDataLine supporting 72 audio formats)

; get a target line
(def line-info (first targets))
(def mic-line (. mic (getLine line-info)))

; add a line listener for events on the line
(. mic-line (addLineListener
              (implement [LineListener]
                (update [evt]
                  (do (print "Event: " (. evt (getType)))
                      (newline)
                      (. *out* (flush)))))))

; check if we can get this format from the built in mic
(. mic-line (open format buffer-size))

; start the input
(. mic-line (start))	
 
; try looping and counting available samples
(dotimes i 100
  (print "Available data: " (. mic-line (available)))
  (newline)
  (. *out* (flush))
  (let [buffer  (make-array (. Byte TYPE) 2048)
        bcount  (. mic-line (read buffer 0 2048))
        bbyte   (. ByteBuffer (wrap buffer))
        bshort  (. bbyte (asShortBuffer))
        ]
    (print "Read: " bcount " bytes. Buffer state:" (str bshort))
    (print " ... Converted to short: "  (str (. bshort (get 0))))
    (newline))
  (. Thread (sleep 20)))   ; 1 milli sleep = 1/1000 of a sec = 44 samples

; stop the input
;(. mic-line (stop))

; close mic
;(. mic-line (close))

)

GUI Code
The following is the GUI part of the code. It generates some functions that draw over a couple of passed in controls. Create a simple GUI interface, then pass a couple of labels.

; osx sampling code
(load-file "iso-source.clj")

; load in the windows sampling code
;(load-file "iso-source-win.clj")

; graphics code
(import '(java.awt Color Polygon))
(import '(java.awt.event ActionListener))
(import '(java.awt.image BufferedImage VolatileImage))
(import '(javax.swing JLabel JFrame ImageIcon))
(import '(java.lang Thread))
(import '(java.util.concurrent Executors))

; make a set of functions that are associated with a graphcs in the ui
; this function is called with a couple of labels that we will draw on
(defn get-oscilloscope ([iso sono]
  (let [c   iso
        s   sono
        of  false
        oc  100
        pa  (ref nil)       ; store previous array so we can undraw
        ox  (ref 0)
        oy  (ref 0)

        ; might be neat to 'fade out'
        turn-off 
          (fn []
            (print "Turned off. Fading.")
            (newline))

        ; draw an oscilloscope grid
        turn-on
          (fn []
            (let [g   (. c (getGraphics))
                  x   (. c (getX))
                  y   (. c (getY))
                  w   (. c (getWidth))
                  h   (. c (getHeight))
                  r   7
                  xs  (/ (- w 1) (- r 1))
                  ys  (/ (- h 1) (- r 1))]
              (do 
                (. g (clearRect x y w h))
                (. g (setColor (. Color gray)))
                (. g (fillRect x y w h))
                (. g (setColor (. Color lightGray)))
                (dotimes ix r
                  (. g (drawLine (+ x (* ix xs)) y (+ x (* ix xs)) (+ y h))))
                (dotimes iy r
                  (. g (drawLine x (+ y (* iy ys)) (+ x w) (+ y (* iy ys)))))
                (. g (setColor (. Color white)))
                (. g (dispose)))))

        ; draw a set of samples, then xor out next time round
        draw-samples
          (fn [samples]
            (let [g (. c (getGraphics))
                  x (. c (getX))
                  y (. c (getY))
                  w (. c (getWidth))
                  h (. c (getHeight))
                  n (count samples)
                  s (/ w n)
                  m 32768                     ; signed 16-bit samples
                  ]
              (do (print "Drawing at " x "," y " size: " w " by " h ". Sample count: " n )
                  (newline)
                  (. *out* (flush))
                  (. g (setColor (. Color gray)))
                  (. g (setXORMode (. Color white)))

                  ; undraw the old one
                  (when @pa
                    (. g (drawPolygon @pa)))

                  ; draw the new one
                  (loop [i    0
                         xa   (make-array (. Integer TYPE) n)
                         ya   (make-array (. Integer TYPE) n)]
                    (if (< i n)
                      (let [nx (int (+ x (* i s)))
                            ny (int (- (/ h 2) (/ (* (aget samples i) h) m)))]
                        (do (aset-int xa i nx)
                            (aset-int ya i ny)
                            (recur (+ i 1) xa ya)))
                      (let [p (new Polygon xa ya n)]
                        (do (sync nil (set pa p))
                            (. g (drawPolygon p))))))

                  ; clean up graphics context
                  (. g (dispose)))))

        ; draw a 2d sonogram - need a better color scheme
        si  (ref 0)
        draw-sono
          (fn [samples]
            (let [g   (. s (getGraphics))
                  w   (. s (getWidth))
                  h   (. s (getHeight))
                  x   (. s (getX))
                  y   h
                  n   (/ (count samples) 4)
                  ss  (/ h n)
                  m   32768                     ; signed 16-bit samples
                  sx  (+ x (* ss @si))          ; start pos
                  ]
              (dotimes iy n
                (let [sy  (- y (* iy ss) ss)
                      gain   3                  ; sonogram gain
                      c   (int (* gain 255 (/ (aget samples iy) m)))
                      c   (if (> c 255) 255 c)]
                  (do (. g (setColor (new Color c c c)))
                      (. g (fillRect sx sy ss ss)))))
              (. g (dispose))
              (if (> (* @si ss) w)
                  (sync nil (set si 0))
                  (sync nil (set si (+ 1 @si))))))

        ; draw a frequency spectra and let it 'walk' down screen +x+y
        ; should clamp either end of spectra to zero to fix graphic glitches
        draw-frequencies
          (fn [spectra]
            (let [g (. c (getGraphics))
                  x (. c (getX))
                  y (. c (getY))
                  w (. c (getWidth))
                  h (. c (getHeight))
                  n (/ (count spectra) 4)
                  s (/ (- w 100) n)
                  m 32768                     ; signed 16-bit samples
                  ]
              (do ; bump iso
                  (sync nil 
                    (set ox (+ @ox 1))
                    (set oy (+ @oy 1))
                    (when (> @ox (- (/ h 2) 40))
                      (turn-on)
                      (set ox 0)
                      (set oy 0)))

                  ; Fill a red polygon, and outline in white
                  (loop [i    0
                         xl   (make-array (. Integer TYPE) n)
                         yl   (make-array (. Integer TYPE) n)]
                    (if (< i n)
                      (let [nx  (+ x (* i s) @ox)
                            ny  (- (+ (/ h 2) @oy) (/ (* (aget spectra i) h) m))]
                        (do (aset-int xl i nx)
                            (aset-int yl i ny)
                            (recur (+ i 1) xl yl)))
                      (do (let [p   (new Polygon xl yl n)]
                          (. g (setColor (. Color red)))
                          (. g (fillPolygon p))
                          (. g (setColor (. Color white)))
                          (. g (drawPolygon p))))))

                  (. g (dispose)))))
        ]
        
    ; return our 'vtable' associated with the two labels
    { :turn-on      turn-on 
      :draw-samples draw-samples 
      :draw-sono    draw-sono 
      :draw-frequencies draw-frequencies})))


;---- lame utility functions ----

; convert an array of shorts to floats
; there *has* to be a better way. But how?
(defn get-as-floats ([shorts]
  (let [n       (count shorts)
        floats  (make-array (. Float TYPE) n)]
    (dotimes idx n (aset floats idx (aget shorts idx)))
    floats)))


;---- load up the GUI, capture and analyze samples ----

; create an analyzer
(def fft (new kj.dsp.KJFFT 512))

; running flag
(def running (ref nil))

; oscilloscope
(defn oscilloscope ([]
  (let [gui       (new isometric.gui)
        ipanel    (. gui isopanel)
        iw        (. ipanel (getWidth))
        ih        (. ipanel (getHeight))
        iimage    (new BufferedImage iw ih (. BufferedImage TYPE_3BYTE_BGR))
        iicon     (new ImageIcon iimage)
        ilabel    (new JLabel iicon)
        spanel    (. gui sonopanel)
        sw        (. spanel (getWidth))
        sh        (. spanel (getHeight))
        simage    (new BufferedImage sw sh (. BufferedImage TYPE_3BYTE_BGR))
        sicon     (new ImageIcon simage)
        slabel    (new JLabel sicon)
        startbutton   (. gui start)
        oneshotbutton (. gui test)
        stopbutton    (. gui stop)
        inputs        (. gui inputs)
        pool      (. Executors (newFixedThreadPool 1))
        ]
    (. ilabel (setSize iw ih))
    (. ipanel (add ilabel))
    (. ipanel (doLayout))
    (. slabel (setSize sw sh))
    (. spanel (add slabel))
    (. spanel (doLayout))
    (. gui (pack))
    (. gui (setVisible true))
    ; populate input sources
    (doseq inp input-lines
      (. inputs (addItem (:mixer-name inp))))
    (let [funcs     (get-oscilloscope ilabel slabel)
          turn-on   (:turn-on funcs)
          draw-freq (:draw-frequencies funcs)
          draw-samp (:draw-samples funcs)
          draw-sono (:draw-sono funcs)
  
          ; worker function that samples and calls the draw functions
          worker
            (fn [] 
              (turn-on)
              (loop []
                (let [samples (get-inst-samples 512)
                      floats  (get-as-floats samples)
                      spectra (. fft (calculate floats))
                      ]
                  (aset-float spectra 0 0)
                  (draw-freq spectra)
                  (draw-sono spectra)
;                  (draw-samp samples)
                  (. Thread (sleep 10))
                  (if @running
                      (recur)))))

          ; starts a worker, who will quit if @running == nil
          start
            (fn []
              (when (not @running)
                (do
                  (sync nil (set running true))
                  (. pool (submit worker)))))

          ; one-shot, sample and analyze a single 'frame'
          one-shot
            (fn []
              (when (not @running)
                (do
                  (. pool (submit worker)))))

          ; kil the worker by setting @running = nil
          stop
            (fn []
              (sync nil (set running nil)))]

          ; set up UI buttons
          (do 
            (. inputs
              (addActionListener
                (implement [ActionListener]
                  (actionPerformed [evt]
                    (let [mixer-name  (. inputs (getSelectedItem))
                          sel-line    (:get-line (first (filter #(= mixer-name (:mixer-name %)) input-lines)))]
                      (do (print "Selecting: " mixer-name)
                          (newline)
                          (sel-line)))))))

            (. startbutton
              (addActionListener
                (implement [ActionListener]
                  (actionPerformed [evt]
                    (start)))))

            (. oneshotbutton
              (addActionListener
                (implement [ActionListener]
                  (actionPerformed [evt]
                    (one-shot)))))

            (. stopbutton
              (addActionListener
                (implement [ActionListener]
                  (actionPerformed [evt]
                    (print "=== STOP ===")
                    (newline)
                    (. *out* (flush))
                    (stop)))))
 
            (print "Hit start to go..."))))))
         

; comment out the following to have it not start automatically
(oscilloscope)

5 thoughts on “Prototyping Code in Clojure – A Simple Audio Displaying App

  1. Gryphon


    the Java sampling api under Windows has defeated me for now. The code below works on a Mac Book Pro. Nothing else is tested

    So much for the much acclaimed “Write Once Run Anywhere”

    Reply
  2. Coen Stevens

    Looks great! I’m trying to run it on my mac but I miss the isometric.gui
    Any chance you could put it online or send it to me by email? That would be very much appreciated.

    Reply
  3. http://www.deaffaith.info/

    Are you considered the percentage of car insurance can be an extremely difficult for you and your compact car will make your final decision. keya doubt, hear the term of six months you should know and trust our service providers. Do not take your list of the purchase of your claims are charged with thetime commitment, and if you have all of the discounts as well. After this there are times when insurance companies to look for then you can cancel the policy at belife during retirement: Move into a providers research. Another place you want on your wallet? Adding teenager car insurance quote. The other big city where there are no longer have worrythat decision and requires insurance) your GDS ratio, but then the higher your premiums should go through this article tries to see what your car is used for fair credit Butvehicle insurance and the negative items are of the United States, residents of the major vehicle coverage news letter on your personal credit score worthiness to a premium’s rate especially thatfirst of course arisen from the Same Company. Most individuals likes to be removed from its drivers at all times. It is a matter of real estates, the cost of ofall the time, it is deemed to be determined. If any of these methods to be a problem. As with other drivers into the road and living without it will maintainbacks the car insurance quotes, or at least once. They are also more likely to grow in confidence in yourself that quick car insurance company. Risk free drivers with no ofgetting from this facility. Select an ideal vehicle you plan on using.

    Reply
  4. http://www.sorethumbsblog.com/

    If you are purchasing your car insurance. They can also put restrictions on full-time vs. part-time students and new drivers. canto rely on. One way to all your letters until the end you may see in these cases. But if you hear anything surprising. It’s important to highlight however, that injuredat the same company year after year to regret your decision. You must be also worth while effort. So how can you do have to know is how it can moneycan see exactly where each driver’s attitude is because what we are concerned about the value of your vehicle for business and not some get their data back if I todriver policy. Providing a telematics box monitors the insurance companies available. Since the prospect of savings to boot. While it is stolen. Third party legal liability resulting from an insurance thatyou will be able to find cheap car insurance rates would be in the accident. The amount of information to get a better risk for getting caught the the highest ofsans auto. Many insurance companies not providing your car’s insurance, and even post something on sale will be filing a claim, and the cost of the cheaper the insurance policies different- this cover provides for legal action.). To call it ‘standard’ something or someone. There is no guarantee that legal and safe drivers.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *