I’ve moved …

I’m now blogging at http://www.friendlyvirus.org/miguelnegrao/blog/. This blog will not be updated in the future.

A look back on my work with ScalaCollider

I have just recently completed my first project using ScalaCollider. ScalaCollider is a front-end written in Scala, a hybrid object-oriented/functional language that runs on the Java virtual machine. I would like to share my impressions of using ScalaCollider now that I have come to use it for quite some time. I will mostly compare it to SuperCollider (using SCApp on the mac).

Positive:
– I can use all the infrastructure around the JVM and Scala. It’s easy to manage dependencies and to bootstrap a project using Simple Build Tool (SBT). There are several extremely competent IDE’s available that are used by developers at companies like Twitter. There are performance analysis tools available that are used by professionals. It’s possible to build an executable that can be distributed and that runs with a simple doble-click easily. This was possible in SuperCollider using standalones but it’s a complicated process, which meant I didn’t do it often enough. With scala it’s so easy that I keep binary copies of my project during development at several stages and after each concert (it’s one command in sbt). I also keep a copy of scsynth and the plugins which means that I have an executable that is exactly the same setup I had at that particular concert.
– I personally use the Intellij IDEA IDE, which is amazing. Coming from SuperCollider.app, IDEA just blew me away. It’s easy to edit code, it can reformat code easily, refactor code (change name of variables, classes, etc across multiple files), there’s no more tabbing and hitting space to format text, just place the cursor anywhere on screen and start writing. Because scala is a static language, the autocompletion is mind-boggling at first (in a good way !). Also help is available just by hovering the cursor over class name and hitting a key combination. Help is also available during auto-completion, so it’s easy to check what each method available does (at least after the help system is configured correctly, which might take some time).
– Scala is a new language that is still evolving. It has many new features that are ground-breaking such as implicit conversions, which allows having type classes like in Haskell and extending libraries that we cannot change. It’s a cutting-edge language, full of new ideas and possibilities. SuperCollider in comparison feels old. For instance, the scalaz library is incredibly powerful and advanced. It’s basically pure math. It’s quite a challenge to use it, but what you get in return is a new way of looking at things. Through scala I came in contact with a lot of new concepts (realized in libraries that I can use) such as Option/Maybe, Algebraic Data Types, Futures/Promises, Monads, Applicative Functors, Functors, Foldables, Functional Reactive Programming (Events and EventStreams), etc.
– Even if scala is a language that compiles down to byte-code, it’s still possible to use it as if it was interpreted using a REPL (Read Evaluate Print Loop), which is an interactive prompt where we can evaluates snippets of code. It’s therefore possible to write small scripts without declaring any classes. The environment to do this is not as nice as in SuperCollider, for instance, it’s not currently possible to select a portion of code and evaluate that, although that is just because no one implemented that yet (although sciss has tried already to make a plugin for IDEA to do this). I wouldn’t use ScalaCollider for live coding at this point, since the type system can become cumbersome when trying to code things quickly, although most of the times it’s quite invisible.
– It’s possible to use any Java or Scala library together with ScalaCollider that means that there is already code out there to do almost anything that we can think off, and it’s easy to interface with those libraries because they are all written either in Java or Scala (no need to go down to c or c++). I was astonished when I implemented the functionality of discovering my iphone when using TouchOSC with Bonjour/Zeroconf with just a few lines of code by importing a java library. The same was true of json parsing. It’s possible for instance to do graphical stuff using processing (Java) or SPDE (a Scala version of processing). The possibilities are endless. And most of the times these libraries are actually written by computer scientists, or people that work in companies or institutions, so the quality standards are quite high.

Negative:
– Some things are missing, such as Routines (there are some implementations available, but not final yet) and Patterns and others are still work in progress. Most of what is implemented is already working quite well though, I detected few bugs.
– It takes a bit of time to setup the development environment. Things are quite smoother now then some months ago though, and there is more documentation available to guide us through the process. It took me quite some time to understand how to get the help system going. Now the latest version of the SBT plugin automatically pulls the source jars when generating an IDEA project, which means I get the help there automatically. It took me some time to figure out how to use the continuations plugins in IDEA, which is necessary to use Routines. Hopefully all of this will be ironed out with time.
– It’s not as easy to just evaluate pieces of code, and check what is going on in the code by checking the value of variables. This is in part because the coding style used (which I copied from a sample application) has more classes inside classes, which makes the code more organized, but also more difficult to check things just by evaluating expressions. On the other hand, there is a debugger available, something which is quite handy to check things. It’s magic stepping through the execution process and actually looking at the lines. I confess I still don’t fully understand how to properly use a debugger and many times end up putting a lot of println’s all over the place, but that’s more a limitation that I have.
– The Scala language is more complex then SCLang, there are more things to grasp in order to use ScalaCollider. For instance, scala is statically typed, which means that I got a lot of errors about something being of the wrong type (usually collections) and I didn’t always know how to solve this issues. It gets better with time, but still it might take more time to code something (unless off course we are using an advanced feature of scala that actually makes it less code anyway). I don’t think the average SuperCollider user would have an easy time in scala.
– I found the GUI code a LOT more complex than in SuperCollider. The place where I lost the most time was with scala swing, which is a GUI toolkit. It’s badly documented, and it’s just more complex, although also more powerful. Scala swing has very powerful layout managers, so when I started using it I was delighted that I would never type pixel locations in Scala. But after some time it turned out that it actually takes more time to make the layout system do the right thing then actually typing the hardcoded pixel values by hand (provided I would never change those values, which is often not true).
– ScalaCollider runs in an environment (the Scala development environment) which is not specifically made for music. That means that many things are more complex then they need to be for audio work (things like the GUIs, selecting files from disk, serialization of objects, etc).

All in all, using ScalaCollider has been a quite positive experience. If not for anything else, I learned a lot along the way. I also lost a lot (a LOT !) of time, so ScalaCollider is not for those that are in a hurry to get something done. I had to figure out a couple of things on my own which took extra time. For instance to save settings to disk I had to learn on how to do serialization using the JSON serialization provided by liftweb, and to write some custom code for my classes (in SuperCollider all I needed to do was, .writeArchive or .asCompileString).

ScalaColliderZ

ScalaColliderZ is a library that provides some added features for ScalaCollider based on the scalaz library. Scalaz is a library for pure functional programming in Scala, which takes a lot of inspiration from Haskell. It has several type classes such as Monad, Monoid, Functor, etc.

ScalaColliderZ provides some methods (many more could be implemented) to deal with asynchronous tasks. Things such as sending synthdefs to the server or loading buffers are done in ScalaCollider using responder functions: we pass a function to the method that gets executed whenever the task has completed. The problem with this is that it’s not easy to sequence several tasks. For example it’s convenient to be able to load a buffer and send a synthdef and then play a synth when both tasks have completed. Promises are perfect for this. A promise is a kind of container that will have a value of certain type at some moment in the future. This is similar to Lists that are containers that can have multiple items of the same type or Option which is a container that might or might not have an item. It’s therefore also possible to use map and flatMap to get a new Promise from an existing one and foreach to execute an action when a promise get it’s value. Example:

promise{ /* do something that takes some time*/; 4 }.map(_+1).foreach(println)

this will print 5 after some time. It’s also easy to combine multiple promises (that might yield at unknow times in the future) to form another promise.

val p1 = promise{1}
val p2 = promise{2}
val p3: Promise[Int] = (p1 |@| p2){ (x, y) => x+y }
val p4: Promise[List[Int]] = List[p1,p2].sequence.

Currently there are methods to load Buffers (BufferPromise.load) and to send synthDefs (mySynthDef.loadP) that return a a promise:

import de.sciss.synth._
import ugen._
import scalaz._
import Scalaz._
import concurrent._
import org.friendlyvirus.mn.ScalaColliderZ._

object Test1 {
  def main(args: Array[String]) {

    Server.test{ s=>
      //Create two promises, one for loading a buffer and another for sending a SynthDef
      val p1:Promise[Buffer] = BufferPromise.read(Server.default, "sounds/a11wlk01.wav")
      val p2:Promise[SynthDef] = (SynthDef("play"){ Out.ar(0, PlayBuf.ar(1, "bufNum".kr(0) )) }).loadP(s)

      //Combine the two promises using an Applicative Functor. The function supplied will only be executed once
      // both promises are fulfilled. The function supplied takes a buffer and a Synthdef and returns a Synth, therefore
      // x will be a Promise[Synth]. We can then continue to compute on the Synth as if it was already here by using
      // map, flatMap, fold, etc
      val x:Promise[Synth] = (p1 |@| p2){ (buffer, synthDef) => Synth.play("play",Seq("bufNum"->buffer.id)) }
      x.map( x => println(x) )
    }

  }
}

And an example of sequencing:

import de.sciss.synth._
import scalaz._
import Scalaz._
import concurrent._
import org.friendlyvirus.mn.ScalaColliderZ._

object Test2 {
  def main(args: Array[String]) {

    Server.test{ s=>
      //change this to a big audio file of your own
      val path = "/Volumes/More Stuff/Audio Projects/Audio Logbook/2011-2012/FullCycle/20/20.aiff"

      println("Creating Promise")

      //load the file into 10 buffers.
      val ps:List[Promise[Buffer]] = List.fill(10)(path).map( BufferPromise.read(Server.default, _) )

      //sequence takes a List[Promise[Buffer]] and returns a Promise[List[Buffer]]
      //sequence is available for instances of the Traverse type class.
      //this promise will get it's result when all promises in the list get their results.
      ps.sequence.map( buffers => println("All buffers loaded:"+buffers ))

      println("Waiting for buffers to Load")

    }

  }
}

ScalaColliderZ also adds type classes instances for GE for Monoid:

import de.sciss.synth._
import ugen._
import scalaz._
import Scalaz._
import org.friendlyvirus.mn.ScalaColliderZ._

object TestZeroSemigroup {
  def main(args: Array[String]) {

    //because GE is semigroup we know how to sum lists of GEs, etc
    //Unfortunately we need to use .asMA since List already as a sum method.
    (List(SinOsc.ar,Saw.ar):List[GE]).asMA.sum.println

    //Also we can sum Maps with Maps
    val m1:Map[String,GE] = Map("osc"->SinOsc.ar(400))
    val m2:Map[String,GE] = Map("osc"->Saw.ar(500))
    println(m1 |+| m2)

    //because GE is a monoid it knows about it's zero
    val o = none[GE]
    //the same as o getOrElse 0.0
    println(~o)

  }
}

I hope to add more stuff in the future.

How to configure Ubuntu 11.10 for using 2 RME PCI Hammerfall DSP MADI cards together

The objective is to be able to use the 2 RME cards as one single 128 channel sound card. Ubuntu 11.10 does not detect the RME HDSPM out of the box, so we need to compile a new vanilla linux kernel obtained from git. It is also a good idea to make a small detour from the instructions below and configure the kernel for low-latency, but I will leave that for the brave souls (although it’s easy).

git clone https://github.com/tiwai/sound
cd sound
cp /boot/config-`uname -r` .config
yes '' | make oldconfig
sed -rie 's/echo "\+"/#echo "\+"/' scripts/setlocalversion
make-kpkg clean
CONCURRENCY_LEVEL=`getconf _NPROCESSORS_ONLN` fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers
sudo dpkg -i linux-image-3.3.0-rc1-custom_3.3.0-rc1-custom-10.00.Custom_amd64.deb 
sudo dpkg -i linux-headers-3.3.0-rc1-custom_3.3.0-rc1-custom-10.00.Custom_amd64.deb
sudo update-initramfs -ck 3.3.0-rc1-custom
sudo update-grub

Reboot and you should be running the new kernel which should detect the cards. Doing

cat /proc/asound/cards

should list the cards.

Next we need to make sure we can name the cards individually since hw:0 will possible refer a different card after every reboot. To do this we place the following inside /etc/modprobe.d/hdspm.conf:

options snd_hdspm id=foo,bar

To merge the two sound cards into one for ALSA we create a ~/.asoundrc.

In our case we have the hdspms connected to RME’s ADI-648 units via optical madi cables. One of the ADI’s is the wordclock master for the the other ADI-648 and the pci cards, so we need to set the clock source and the inputs source:

amixer -c foo set 'System Clock Mode' 'AutoSync' && amixer -c bar set 'System Clock Mode' 'AutoSync'
amixer -c foo set 'Input Select' 'optical' && amixer -c bar set 'Input Select' 'optical'

Now we unmute the cards:

for i in `seq 1 64`; do amixer -c foo set 'Chn',$i 64 && amixer -c bar set 'Chn',$i 64; done

To get rid of xruns jack should run with realtime priorities (change johndoe to your user name):

dpkg-reconfigure -p high jackd
usermod -a -G audio johndoe

You can check your privileges with:

ulimit -r -l

You can now start jackd for the 128 channels sound card

jackd -R -P 70 -d alsa -r44100 -dmergedcards -p128 -o128 -i128

and have fun !

Thanks to Adrian Knoth and Florian Faber for all the help with the hdspm driver !

It took me about 2 months to compile this information, so I hope this is useful to someone.

OSX internet sharing under the hood

I was curious this week to know what exactly is OSX’s internet sharing doing. It starts 3 processes as follows:

bootpd -d -P
natpmpd -d -y en0 en1
named -c /etc/com.apple.named.proxy.conf -f

From the man pages:

bootpd:  DHCP/BOOTP/NetBoot server
natpmpd: natpmpd is a daemon implementing NAT-PMP. NAT-PMP is a protocol for handling port forwarding requests from clients behind a NAT. It is normally
invoked by InternetSharing(8).
named: named is a Domain Name System (DNS) server.

The internetSharing process is responsible for starting these processes:

InternetSharing:

InternetSharing is the back-end for the Internet Sharing feature. It is
responsible for configuring the network interfaces, the DHCP server
bootpd(8), the network address translation daemon natd(8), and the Inter-
net domain name server named(8). named(8) is run in caching-only mode
and allows the DHCP server to always offer the same DNS server address to
the DHCP clients, regardless of the value of the actual DNS server
addresses.

more /etc/com.apple.named.proxy.conf
// Declares control channels to be used by the rndc utility.
// It is recommended that 127.0.0.1 be the only address used.
// This also allows non-privileged users on the local host to manage
// your name server. An empty control set means the utility is disabled.
//
controls {
};

options {
directory "/var/named";
/*
* If there is a firewall between you and nameservers you want
* to talk to, you might need to uncomment the query-source
* directive below. Previous versions of BIND always asked
* questions using port 53, but BIND 8.1 uses an unprivileged
* port by default.
*/
// query-source address * port 53;

listen-on { 192.168.2.1; };
forward first;
forwarders { 10.0.2.1; };
};

//
// a caching only nameserver config
//
zone "." IN {
type hint;
file "named.ca";
};

zone "localhost" IN {
type master;
file "localhost.zone";
allow-update { none; };
};

zone "0.0.127.in-addr.arpa" IN {
type master;
file "named.local";
allow-update { none; };
};

acl can_query {
any;
};

edit1:
The reason I made this investigation was that I was curious about what in internet sharing causes disruption of networks. Turns out that the DHCP server is configured to through out  ip’s in any of the mac’s network interfaces. If instead it was doing

bootpd -d -P -i en1

then it would give out ip addresses only to clients on the wireless interface. I’m I missing something or why didn’t apple think of this ?

Functional fibonacci numbers

There are many ways to calculate the fibonacci numbers sequence in scala, but I really like this one:

def fibs(n:Int)= {
    lazy val fibSeq:Stream[Int] = 0 #:: 1 #:: (fibSeq zip fibSeq.tail).map{ case (a,b) => a+b }
    fibSeq.take(n).toList
}

#:: is the lazy cons method, and fibs zip fibs.tail creates a stream of pairs with the latest two numbers in the sequence.

In haskell it can be even more compact:

fib :: Int -> Integer
fib n = fibs !! n
  where
    fibs = 0 : 1 : zipWith (+) fibs (tail fibs)

Using streams in scala for functional style programming

The paper “why functional programming matters” by John Hughes is a very nice example of using streams to calculate an approximation of the square root of a number using the Newton method.

The #:: method is a lazy cons, that only calculates the element to be added to the head of the list when needed. The repeat function creates a stream by repeatedly applying f to the last element of the  stream starting from the value a. The within function keeps looking for better matches in the stream until the successive values are close enough to each other. By chaining these functions you get the result:

/**
 *
 * Newton-Raphson Square Roots
 * as is shown in "why functional programming matters"
 * http://www.cs.utexas.edu/~shmat/courses/cs345/whyfp.pdf
 */

object NewtonRoots {

  //algorithm to approach the square root of n
  def next(n:Double)(x:Double) = (x + n/x)/2

  //create a stream composed of repeatedly applying f
  def repeat(f:Double=>Double, a:Double):Stream[Double] = a #:: repeat(f,f(a))

  //look for a good enough match in the stream
  def within(eps:Double, s:Stream[Double]):Double = s match {
    case a #:: b #:: rest => if( (a-b).abs < eps) b else within(eps, b#::rest)
  }

   //look for a good enough match in the stream.
   // A better way is to see how close a/b is to 1,
   // to avoid the rounding errors
  def relative(eps:Double, s:Stream[Double]):Double = s match {
    case a #:: b #:: rest => if( (a/b -1).abs < eps) b else within(eps, b#::rest)
  }

  //find the square root of 3 starting from the approximation
  // value 1, until approximations are within 0.01
  def main(args: Array[String]) {
    println( within(0.01, repeat( next(3), 1 ) ) )
  }

}

scala 2.9.1 + sbt 0.11 + Intellij IDEA (10/11) + scala continuations plugin

It took me some time to figure out how to compile a project in Intellij IDEA with the scala continuations plugin turned on. I was getting this error when I ran the project:

bad option: -P:continuations:enable

Here is what you need to do to make it work:

1. In your build.sbt file add:

   autoCompilerPlugins := true
   
   addCompilerPlugin("org.scala-lang.plugins" % "continuations" % "2.9.1")
   
   scalacOptions += "-P:continuations:enable"

2. In sbt run

   reload

   and then

   update

3. Generate the Intellij IDEA project using the sbt command

   gen-idea

4. Open the Intellij IDEA project and go to “Project Structure” -> “Modules” ->”Scala” and add the continuations plugin jar file in the “Compiler Plugins” pane. The jar should be in ~/.ivy2/cache/org.scala-lang.plugins/continuations/jars/continuations-2.9.1.jar .
5. Tick the “Enable Continuations” option in the “Compiler Options” pane.

Your project should now compile with support for continuations.

Some examples of pure functional programs that actually do something.

I’ve been struggling with functional programming lately. The part about using pure functions is very easy for me to get, what was/Is difficult is to understand how does a pure program contact with the outside world and manages to keep state.
I then studied a bit about the IO monad, and I managed to understand how to contact with the outside world. The final piece of the puzzle that I’m still missing is out to piece all this together and how to keep the state in a functional way. I’ve coded some very simple examples to teach myself how these things are done. The examples are scala code and use the scalaz library.

In this first example I just read one line at a time from the prompt and then write it back. The app will do this forever until it is killed. The part that causes it looping forever is done by the recursive function forever, which just repeats the same action over and over:

 def forever(theIO:IO[Unit]):IO[Unit] = theIO >>=| forever(theIO)

I find it quite magical way the way this compact syntax encapsulates the action of just doing something over and over again.

import scalaz._
import Scalaz._
import effects._

/*

This program just echo the input

 */

object Test1 {

  def forever(theIO:IO[Unit]):IO[Unit] = theIO >>=| forever(theIO)

  def functionalMain = {
   forever(for(
      line

On the second example I read one line from the prompt and then add it to a list. I was looking for ways to do this without storing this list anywhere. The method that causes the looping in this case is slightly more complicated, because it has to keep propagating an argument forever, by just passing it to the next action. This way we manage to keep state, a list with all the previous strings, without actually storing this list anywhere. Again, I find this quite magical.

import scalaz._
import Scalaz._
import effects._

object Test2 {

  def getStringAndAddToList(list:List[String]):IO[List[String]] =
    for(
      line (A=>IO[A])):A=>IO[A] = makeIO(_) >>= forever(makeIO)

  def functionalMain = {
    val action = forever(getStringAndAddToList _)
    action(Nil)
  }

  def main(args: Array[String]) {
    functionalMain.unsafePerformIO
  }

}

lift-json serialization pretty printing

The built-in serialization of lift-json prints a string with no newlines. I prefer to have a nicely printed string so that I can edit the file if I want. For this end I created a new function

writePretty

:

object JsonPimps {

  def writePretty[A <: AnyRef](a: A)(implicit formats: Formats): String =
    (writePretty(a, new StringWriter)(formats)).toString

  def writePretty[A <: AnyRef, W <: Writer](a: A, out: W)(implicit formats: Formats): W =
    Printer.pretty(render(Extraction.decompose(a)(formats)), out)
}