Digital & Interactive Art Experimentation

Over the last few months we have been exploring interactive & digital art here at RoomCR6 (#CR6labs) using a combination of Xbox Kinect and the programming language Processing. In this blog post we discuss some of our experiments, offer a quick glimpse into what we’ve been up to in the studio via a mini showreel and will also be giving away free downloads of three Sketches we have been playing with.

This is a long blog post so we have included some jump links for you. Enjoy!

Our Interactive Experiment Showreel
Setting up Kinect & Processing
Sketch 01 – Colour Wheel
Sketch 02 – Star Field
Sketch 03 – Digital Painting

What is Processing?

Processing is an open source programming language and development environment that is aimed at providing individuals with software literacy within the visual arts. It also aims to help unlock creative coding amongst designers and developers alike, bridging the gap between aesthetics and functional coding. Processing has also built a great online community of creatives that love to share ideas and also challenge what is possible to achieve with code. For more information on Processing visit the Processing website over at www.processing.org

We have thoroughly enjoyed working with Processing and creating/adapting Processing Sketches that work with the Xbox Kinect, a 3D infa-red depth enabled camera that allows us to pass real world 3D data points into Processing. We have put together a mini showreel covering some of our recent experiments (below) and we are, in the spirit of the open Processing community, giving away our sketches, mostly adaptations from other contributing members of Open Processing, a website where creatives share their sketches, for more info visit http://www.openprocessing.org. Download our Processing Sketches from – Processing Sketch Download.  Continue reading for more details on each of the Processing Sketches and also how we set up Kinect to run within the Processing IDE.

Our Interactive Experiment Showreel

 

Setting up Kinect & Processing

Throwing ourselves into the deep end we quickly realised that there are hundreds of tutorials explaining how to set up the Kinect & Processing IDE, however a lot of these were out of date and we ran into several issues from the start. Processing often gets updated and this makes most of the previous tutorials and set up guides obsolete, but luckily there is Google and an awesome community of creatives out there willing to help, we will discuss this a little later in the post.

First things first you’ll need to get your hands on an older Xbox Kinect, model number 1414 specifically (the model number can be found on the base of the Kinect – see image below). We managed to grab a couple Kinects off of eBay for around £25 each.

xbox-kinect-CR6Labs01

xbox-kinect-CR6Labs02

 

Once we received our Kinects we hooked them up and thought it would be a simple plug and play approach… We were wrong! Firstly, to get the Kinect talking to Processing, we needed to add the Open Kinect for Processing Library by the legendary Daniel Shiffman. You can add the library directly from Processing or visit http://shiffman.net/p5/kinect/ for more details. We also added another library, SimpleOpenNI which comes with a wider range of programming abilities, this library can also be added inside Processing or by visiting https://code.google.com/p/simple-openni/

All excited that we had everything set up and ready to go we walked into another issue… We kept on receiving the following error message when trying to run any of the Processing Sketches (Note we are working on Apple Mac OS X systems, result may be different on Windows or other platforms) –

A library relies on native code that’s not available.
Or only works properly when the sketch is run as a 32-bit application.
error-32bit

 

 

 

 

We spent a bit of time trying to figure this one out and eventually came across a workaround on a thread over at Daniel Shiffman’s github – visit https://github.com/shiffman/OpenKinect-for-Processing/issues/2 for full details.

To make things easier for you here is the workaround in step by step format supplied by Jesper Almstrom:

  1. Install OpenKinnect in Documents/Processing/
  2. Replace the libKinect.jnilib with the 64-bit version (Download) in the /Documents/Processing/libraries/openkinect/library/ folder
  3. Install Homebrew http://brew.sh/
  4. Inside terminal – Install XCode Command line tool – $ xcode-select –install
  5. Inside terminal – Install libusb using brew – $ brew install libusb
  6. Reboot with Kinect connected

After following those steps we were up and running! The easiest way to test everything has worked is to simply run any of the sketches supplied in the examples folder in Processing under Open Kinect, go to File > Examples then open up Contributed Libraries.

cr6labs-001

 

Having set up the Kinect’s we were excited and ready to begin experimenting with Processing and body/motion tracking. We had collected some sketches from the http://www.openprocessing.org – a website where users share sketches they have produced.  We starting adding the Kinect to these sketches as a controller instead of using a mouse or web cam which is usually the case.

Below are three examples we created and are sharing with you to experiment with in your own time. Have fun and please do share your results with us on Twitter – www.twitter.com/roomcr6

Sketch 01 – Colour Wheel (Download Sketch)

xbox-kinect-CR6Labs03-colour-wheel

We were exploring the idea of using a colour wheel in which a user can control with their body. Initially we tried using Average Point Tracking, however the results were not perfect and we found it difficult to accurately control the choice of colour.

This lead us to use SimpleOpenNI’s Kinect library and we built a Skeleton tracker offering more control of the colour choice. Currently the sketch tracks the left hand and outputs the selected colour to the small block on the top left side of the screen. We also connected the Sketch and colour block with Syphon, which allows you to output the block into other software applications such as MadMapper (for more info visit http://syphon.v002.info/). We used MadMapper to display the colour on a projected screen as seen in our Interactive Experiment Showreel above.

Please feel free to edit and play around with the code and don’t forget to share it with us as we would love to see what you have come up with. Tweet us at www.twitter.com/roomcr6

Sketch 02 – Star Field (Download Sketch)

xbox-kinect-CR6Labs04-starfield

Based on the original Processing Sketch from Samyukta Madhu (http://www.openprocessing.org/sketch/147342)

We adapted the original sketch and used Open Kinect’s – Average Point Tracking as the controller of the star field. This can be seen as a red depth image in the image above. This can also be turned off by removing the following line of code:

// Show the image
tracker.display();

To adjust the depth threshold you can either use the up and down arrow keys whilst the sketch is running or edit the code within  KinectTracker.pde changing the initial threshold, currently set to 745 – see code below:

// Size of kinect image
int kw = 640;
int kh = 480;
int threshold = 745;

Please feel free to edit and play around with the code and don’t forget to share it with us as we would love to see what you have come up with. Tweet us at www.twitter.com/roomcr6

Sketch 03 – Digital Painting (Download Sketch)

xbox-kinect-CR6Labs05-paint

Based on the original Processing Sketch from Martin Schneider (http://www.openprocessing.org/sketch/6742)

We adapted the original sketch and once again used Open Kinect’s – Average Point Tracking as the controller of the digital ‘paint brush’. The sketch also uses a threshold which you can edit within the KinectTracker.pde, currently set to 600, edit the following code:

// Size of kinect image
int kw = 640;
int kh = 480;
int threshold = 600;

On initial load of the sketch it will load in Black & White, however you can change this by pressing C on the keyboard and then it will load into full colour mode.

Please feel free to edit and play around with the code and don’t forget to share it with us as we would love to see what you have come up with. Tweet us at www.twitter.com/roomcr6

We are currently working on a couple concept projects and are always on the lookout to collaborate with creative coders and developers, so please do feel free to get in touch with us – Contact RoomCR6 here