Using computer vision to detect faces, TwitBubble draws a thought bubble next to the face, and displays the most recent tweet from the Twitter user entered in the GUI. Made with Processing, OpenCV (face detection), ControlP5 (GUI), Tiwtter4J, in about 5 hours, most of that time was spent on figuring out the twitter4j library.

download the source code

an early screen shot:

TwitBubble was inspired by CloudMirror.

update [04-Aug]: I built a GUI which allows you to manipulate the visualization in real time. Video to come soon.

I made this visualizer over a weekend. It’s built in Processing with the OpenGL 3D library.

The little circles, the dancers, are a basic implementation of a particle system built by Daniel Shiffman.

This was my first time working with the Sonia music library and OpenGL. Pulse will evolve in to a component of a larger interactive, generative piece I’m working on. I created the video with Snapz Pro X. I went this route because the MovieMaker Processing library slowed down the visuals and threw off the synchronization with the audio.

musical credit: Wolves (Song of the Shepard) by Iron and Wine from The Shepard’s Dog

download the source code

Some early screen shots of the visualizer:

/sneak preview.

Definition file (.ghx).

Referenced assembly (.dll).

Please note that the referenced assembly is based entirely on code from luvtechno.net.

This was my final project for the Digital Fine Art elective I took as part of my master’s degree. It was shown at Skrol. This was a solo project.

A Revolt Against Actuality is in interactive video and sound installation that explores our relationship to artwork and artists.  Viewers of the work are captured on video and thrown into the work while visitors moving through the larger gallery space affect and distort audio elements. Real time background subtraction and motion capture are the technical underpinnings.

Placing the viewer in the work  demands an analysis of how we engage with complexity and specifically complexity in art. It expressed the need to look past the surface and pull back the layers of meaning. I’ve used Joyce’s work as the foundation of the piece as it is the paradigm of the complex. Without deeper analysis it is gibberish, but  with insight the gibberish evolves into brilliance.

The forms of interaction, visual engagement and audio distortion, reflect the necessity to both engage with artistic material from the perspective of the creator – get inside their head- and the need to distort work through our own interpretations and perspectives -externaldistortions.

A Revolt Against Actuality was built in the Processing programming environment.

The inner workings of ARAA incorporates separate audio and video systems. The the video system, by its nature, is obvious. The camera captures the viewer and places them inside the image on screen every few seconds. However, the audio system is a little more subtle and to fully appreciate the idea of the piece let me explain how it works.

There is a camera and screen outside the viewing booth. The camera captures the scene in the gallery area while the text from the Anna Livia Plurabella chapter of Finneagan’s Wake is displayed on screen word by word. The system captures the movement in the gallery space and manipulates the size and colour of the text on screen in addition to the audio recording of Joyce himself reading the same text. The viewer in the booth experiences the unintentional manipulation of the audio by the viewers on the exterior. Extending the notion of influence of external forces on our interpretation of art.

this post was originally published on my blog at the Centre for Digital Media

PITTSBURGH, PA – I don’t know where to start. I’m sitting in the airport waiting for my flight to Chicago and then home to Vancouver. I spent the past four days in Pittsburgh for the Art and Code symposium at Carnegie Mellon University. To paraphrase the words of Golan Levin, organizer of the event and director of the STUDIO for Creative Inquiry @ CMU, the conference was a gathering to workshop and discuss programming tools designed for artists, young people, and the rest of us.

It was a conference focused on the computer as a means for creativity as apposed to simple utility, evolving the computer from efficient tool to artistic medium.

Presenting at the conference were the thought leaders of this vibrant community. The founding developers of Processing, openFrameworks, VVVV, Hackety Hack, Alice, and Scratch along with leading educators and researchers taught workshops and presented their platforms.

I attended the ActionScript workshop by Ira Greenberg, the author of Creative Computer Coding, Daniel Shiffman’s Advanced Processing Workshop, and“hello world” workshops for openFrameworks and PureData. It was intense and exhilarating and exhausting and profoundly inspiring.

Emphasis throughout was placed on computer literacy as apposed to simply computer proficiency. Just as literacy in the context of language does not simply mean that a person can speak but that they can read and write – express themselves. So too, computer literacy should mean that we don’t just know how to use the computer to get things done but to create software and control it. To use the computer as a means in fulfilling the will to self-expression, no different than using language to write. With that mindset, there was a persistent distinction between computer science and programming and the importance of breaking down the barrier to programming amongst non-computer scientists.

Personally, I owe Daniel Shiffman a beer or two for the work that he has done to break down that barrier. It was his book, Learning Processing, that I devoutly pored over during winter break, only a couple of months ago; it gave me the confidence to tackle programming and break down the barrier ‘line of code by line of code’.

Other than the obvious rough and tumble technical learning that comes from intense workshops, the most valuable take away was the sense of community and passion that percolated throughout the weekend. Nothing exemplifies this more than the fact that the majority of the platforms are entirely opensource and non-commercial. It is this spirit of openness and collaboration and sharing which is the foundation of creation and innovation in this field. The essence is synthesis; the synthesis of the artistic and the technical in both mindset and form.

Unlike the capitalist market place where IP and privacy are the foundations of innovation, in this atmosphere it felt like people were competing for openness. The more they give away the taller they stand.

This shift in perspective owes itself to a few factors. Firstly, the development of many of these tools comes from the academic world, namely MIT, NYU and CMU. It is within the academic environment that developers are released from corporate expectation and are encouraged to freely disseminate their programs.

Additionally, these toolkits are developed by small core groups of people. Processing was developed by a team of two, Ben Fry and Casey Reas, of hundreds of developers, making the process cheaper and more sustainable.

But, most central to the success of these endevours are the communities that have evolved around them. Hundreds of people, all volunteers, write libraries to extend the features of these programs, answer questions in forums and share their code with others. And the true success of these platforms lies not in the technology, per say, but the communities they built and foster.

At a closing panel on the development of new tools, Zachary Lieberman, a founding developer of openFrameworks, stressed the importance and necessity of social interaction in the development of these platforms and in education. in the openFrameworks community, social gathering in computing manifests in the organization of programming ‘knitting circles’ where people get together and jam with code.

On Sunday night we held a conference specific ‘open mic -Dorkbot – Pecha Kucha’ hybrid where people presented their various projects and experiments. I took the mic and proudly shared one of our projects from last semester, Fluxus – the art of conversation; I not only demoed the virtual conversation space we designed, but also showed some photos from the Midforms Festival where Fluxus was on display. My final slide captured two avatars inside each other creating art; the perfect image to represent my experience at Art and Code where 200 people from all over the world got together to talk about creating art with code and the art of coding. I eagerly anticipate next year.

Ryan Nadel is a first year student in the MDM program.

I was introduced yesterday to http://processing.org/. After messing with python and C++ it is interesting to mess with a system that allows me to quickly access libraries that manipulate graphics. I’m not even sure what Processing really is. It looks like a framework with some cool built in modules sitting on top of java. I haven’t come up with anything amazing yet, but this is definately something that has peaked my interest. Today I wrote this program that draws a circle and filles it with random sized smaller circles:

float radius;
int r;
int g;
int b;
int s;

strokeWeight(s);
stroke(color(r,g,b));
fill(color(r,g,b));
ellipse(250,250,radius,radius);
}

Today I’m going to work on drawing moving shapes with changing colors.

In the best known example of Cellular Automata – Conway’s Game of Life, each cell has a binary state – it is either On or Off.
However, it is possible to explore similar automata where the state of each cell can be any real number in a given range – Continuous Cellular Automata.

The video above shows such a CCA in grasshopper. Each cell has a height value, which interacts with the values of its neighbours according to a simple* equation. This is a way of generating 3-Dimensional forms even though the cells only use a 2-dimensional Moore neighbourhood.

You can download the grasshopper definition here:
CAheights.ghx

I made this by taking a Game of Life definition which Baldino had already made and simply copy-pasting some code from one of my earlier processing sketches into the VB component (well, I had to change a couple of bits of syntax, but surprisingly little).

The processing sketch uses colours instead of heights to show the value of each cell, and is mouse reactive:

Click image for Live Interactive version + sourcecode

Its strange the way it varies between periods of calm and chaos, without ever completely settling down or degenerating to noise.

* though it took me a fair bit of trial and error to come up with this particular equation. Just like the different rulesets explored by Wolfram, little changes can give quite different results, and I ran through all sorts of odd glitchiness before finding one I liked.

This visualization was made by Robert Hodgin with Processing while working on a way to integrate lyrics into visualizers. To read about his process click HERE.

Click image for Live Interactive version

A little toy which demonstrates how complex and chaotic behaviour can arise from very simple rules and local interactions.

This is a type of Continuous Cellular Automata. Each cell has a scalar value which changes, based on the values of its neighbours. Then by taking the difference between the values of neighbouring cells, this scalar field is converted into a vector field.

You might also like to try the fizzy version, the musical version, the many glitchy variations, or the monster version, and I’ve also started playing with a 3D version

Source Code provided (just follow the links)

2008 ended with style : a full week workshop around Julien V3GA’s Vision Factory API.

LiveCoding is an extension of Vision Factory framework (used by Julien V3GA in his studio for professional purposes and for Vjing in a personal purpose) and exists in two different “beta”-versions : 
● On a single computer;
● Up to 5 computers (Up to 4 computers using Processing plus an extra one where the API actually runs).

In order to use Vision Factory, you have to edit a javascript file where you enter your code lines. When you save the javascript file, SpiderMonkey re-interprets the script and sends a “bug report” to Vision Factory. If no bugs were found, the script is validated and displayed, else the previous script is kept running and Vision Factory stands by for a new/corrected bugs-free version of the JavaScript file.

We use pre-programmed Vision Factory functions as base structures for the javascript files. The most basic ones are equivalent to the Processing’s or OpenFramework’s “void setup()”, “void update()” and “void draw()” functions. The init() -equivalent of setup()- is run only once, but it can be recalled in any moment by the programmer. Both update() and render() -equivalent of draw()- are loops and work like in OpenFrameworks. Here you can find some new features recently added by Julien V3GA.

Video from Julien V3GA

Another characteristic of Vision Factory is the fact that it was built on a layer structure concept. Vision Factory has a built-in OSC protocol management. It is simple to get access to layer proprieties and to change them using OSC messages. In the network version, each computer gets assigned to a layer (their javascript is assigned to the specific layer). Using several computers, each computer receives a layer, and like in photoshop, the higher layers mask the lower ones. Each computer runs a Processing client that sends the script whenever it is saved, Vision Factory receives them and treats them like the non-network version. Finally, we can control Vision Factory’s final render : splitting the screen, giving a render for each JavaScript or assembling the different layers into one unique screen, having superpositions from the different scripts.

Vision Factory is currently not released.

Julien V3GA also showed us an iPhone/iPod Touch application called Mrmr that he uses when he makes some Vjing stuff. The aplication allows one to configure an interface and send data using Open Sound Control. The devices (iPhone/iPod touch) interface (multitouch screen) is highly adapted to control sound and visualization’s parameters (like a MIDI controller) with the advantage is that you can do it wireless.

Considering all this, I am more then willing to start a project to implement Open Sound Control into a Blackberry mobile phone (if someone heard about an opensource project on this direction, please leave me a comment presenting this project). The advantage of a Blackberry over other phones is the full (and comfortable) QUERTY/AZERTY keyboard. Allowing you to type fast enough to make LiveCoding. The first stage is to implement OSC, then to make a programming interface (the built-in notepad?) to finally add the command features (connect to server, reload the “setup()” function of the script, disconnect, open a script, etc).

Well, I believe that I have some work to do now…

Working on the concept of surfaces with processing and eskimoblood’s surfacelib.
See the other pictures here at my flickr Surface Works gallery :

Now that we didn’t win anything (so it seems) I can publish this project. It was great working on it, though a bit stressful towards its final stages…

Here it goes:

This project is a collaboration between Veronica and me.

Creation

The urban bacteria is a responsive structure that is not built; it grows following the path dictated by an algorithm that takes into account the geometry of the site as well as the available sunlight. It adapts to almost any given urban space, evolving into an organism fit for the conditions it encounters.
We created in processing (processing.org) an autonomous system that is emergent (it demonstrates an “internal will to reach coherence” – Cecil Balmond). It was used to compute the structure of the bacteria and simulate its growth in a variety of conditions.

Pulse

It has a life of its own. It pulses along with the variations in sunlight: when there’s a excess/high amount of sunlight available it increases its volume, regaining its initial form as a direct result of a decrease in available sunlight (caused by clouds, sunset). In its “expanded” form it offers more shadow to the pedestrian space below when it is most needed, increasing its quality and, therefore, inviting people to use it.

Daylight > nightlight; natural light > artificial light

During the day, the “urban bacteria” stores the excess energy resulted from sunlight and releases it during the night. The quality of an urban space is linked with the amount of light it receives during the night – this “living structure” proposes an ecological/economical way of transforming daylight into night light, sunlight into artificial light.

Materials

The membrane of the structure has a multiple role: absorb sunlight and create shadow during the day and release light during the night. We propose a multi-layered material consisting of two layers of polarizer sheets (in between which there’s another layer of cellophane) superimposed over a photovoltaic lattice that transforms sunlight into energy. The double layer of polarizing material creates an intense visual effect (by speculating small shifts in geometry and sunlight angle) that can be used to enrich the surrounding urban space.

“Urban bacteria” is an autonomous shape living in concordance with the rhythm of the environment.

Final boards:

This project was developed for the Velux IVA competition.

Scripts

RhinoScript and Processing

Process

Existing geometry in which the bacteria would evolve was constructed in Rhino and then custom-exported as simple rectangular planes in a specially built p55.in file. This would be custom-imported in Processing.

The growth algorithm was programmed in Processing. Running inside the imported geometry the bacteria would grow. The thus-grown (see above for details) structure would be custom-exported for Rhino.

Via RhinoScript, the “bacteria” would be imported and given a form via a custom script that took into consideration the time of day. Fin.

The second semester of my academic year turned around a “group project” called “Laboratoire des Fictions”.

We had to get together to think on the fiction laboratory topic. The project ends up with a public presentation in the Studio space (in Aix en Provence School of Art).

We (the class) had to think on different art works and space arrangement in this big room that is the Studio. Each of us presented projects, ideas, concepts, definitions and we debated on theme in order to get a coherent set.

We decided to build a space where the visitor is manipulated, and used as laboratory object of study. The visit starts with charming attendants dressed like nurses/scientist that gives the visitor a placebo pill explaining that the active substance in the pill will make theme feel never before felt sensations. Then the visitor passes through a set of two corridors built with elastic fabrics with the objective to disturb the visitor’s senses. The visitor passes through a third corridor where the soil is soft, the left wall captures the visitors’ shadows and the right wall (nearly invisible because of the darkness)  keeps falling on the visitor’s head. This third corridor leads the visitor to the main room where several installations are establish. The first installation the visitor sees is an anamorphosis activated by a giant hamster wheel where visitors are invited to run. In the same room, the visitor can see the other side of the falling wall, this side of the wall has mirrors, when it falls (this time away from the visitor) our perception of verticality is disturbed and we feel like if we were falling. Another installation is a set of green neon lights that turn off after some time in order for the visitor to see the “Vie en rose” (there is an effect of persistence of vision, when the green light is switched off, the eyes have a adaptation time and you see everything in the opposite color – pink). When the neon light is turned off, we can hear a specialized quadriphonic mix of several versions of  “La vie en rose”. On the floor, a crazy robot drives around avoiding walls and visitors, presenting bugs and errors, annoying people. Finally, a door, in the center of the room, is used to play a Tetris game projected on a waterwall. The waterwall is also the exit. The visitor has to choose between waiting for a random waterfall stop, playing the game to see if it makes the water stop or just cross it with water anyway. Once outside the room, the visitor could find an evaluation machine called “I.D.I.O.” (in french it means “stupid”), where, during a performance, some of the visitors are randomly chosen to get evaluated by a group of (crazy) scientistes.

Since it’s hard to explain the exhibition with words, here are some videos showing it :

Here is a 3D animations made by Floriane Rebatue:

My work in this project is the Tetris waterwall projection. I have designed, built and programmed a big part. The game was programmed using Processing, the door has sensors linked to a PIC 16F876. The waterwall is 4 meters width and 3 meters high, it’s a closed circuit using a water-pump to inject water in a pipe with 1500 holes (1mm diameter). The collector is 4 meters width (in the central part) and is 1 meter large. A small bridge allows visitors to cross the water. The sound is generated by Pure Data (thanks to François Parra). I also got help from Jane, Pierre Loup and Florent.

Here are some sketches :

Here is a video :

Here you can find the code source (addapted to use with a computer keyboard and with a PIC) - Requires oscP5 (project site) library and Processing

Here is the PureData patch (activation of the OSC Library and PureData required).

Oh, this was a long night! As I was about to go to sleep yesterday, I was just checking my blogroll on Bloglines the last time, when I saw John Resig’s brand new blog post, announcing that he had ported the wonderful Processing.org to JavaScript. This is absolutely great news. Some time ago I read that a port of Processing to JS had been started, but I wasn’t able to follow the news back them. So this came as a nice surprise.

I really like Processing and have been doing some small experiments with it for quite some time. But what I mostly like about it, given my total lack of artistic sense, is to appreaciate all the creativity and beautiful things which it empowers artists to do (see photos and videos on flickr, vimeo and their site).

So, having a bit of experience with Processing and a deep love for JavaScript, I took last night to explore Processing.js and see what I could quickly do with it. I picked up three examples of code I had with Processing and put them to run under Processing.js. And I also created a brand new one to explore the API from the JS perspective.

Porting the code from Processing was considerably easy. It was not totally flawless, of course, because we simply can’t expect everything to run perfectly without the need of any modification. However, with some simply exploration and effort, everything was running ok.

The greatest part of it is that it is a double API and language under only one core. It is weird to look at it, but you can code in Java or in JavaScript to the exact same API. Under the hood, the Processing.js parses the Java code (at least most of the idiom common to Processing) and converts it to JavaScript. It is funny that, even if you made a mistake in your Java code, you’ll get a JavaScript syntax error!

I’ll go into more details about the code and the porting in another post. Here I’ll mention that most of the Processing 2D API (3d not supported) works flawlessly on it (even the text services, only for Firefox 3), but there are some things still lacking. As a quick example, two of my Processing code used the mouseButton variable to know which button of the mouse was pressed. This variable, however, is lacking in this first version. Anyway, it was a very easy addition to the core with just two lines of code. The frameCount is also missing, which is another easy addition. Anyway, the great thing is that, knowing that we are only targeting the most modern web browsers, we can use advanced JS and DOM features, without the need for ugly hacks all over the place. As an example, we can even use the DOMContentLoaded event to watch for the best time to start to run the code.

Let’s go to the examples! Just a note, however: these are all ultra-alpha code, and by using it, you understand you are going to the deep underground world of your web browser, and that it may crash or get extremely slow. All the examples work in Firefox, Safari and Opera, but some of them needs the text services and will only make sense if you view with Firefox 3 (also, they were only thoroughly tested in Firefox 3).

Brownian Movement

This is a very basic one. A Brownian Movement example comes in the default examples of Processing, and when I was learning the Processing API, I used that code as an example to create a more aesthetically pleasing version, putting some random colors to the mix, a fade-out effect and changed the straight lines to Bezier curves. The code worked without any modification at all! I just picked up the Processing code, throwed it to the Processing.js core and it worked on the first run.

Chaos Theory

A simple game I coded in 2006 inspired by a similar flash game I saw back then. When the game starts, 75 balls are randomly launched into space, and you must click somewhere into the screen to generate a explosion. When the explosion hits a ball, it generates another explosion, creating a chain reaction. The goal is to destroy the most number of balls with the smallest number of clicks (generated explosions).
The original code used some Java features unavailable at Processing.js, but converting it wasn’t much of a deal (except for squashing some original bugs!). Also, the balls had some drop shadows which I disabled here for the game to run quickier. You will need to view it in Firefox 3 if you wish to see the ball counter (which is somewhat essential for the fun), but I’ll change this soon to a text on the page, as I did in the next example.

Snake

A simple version of the traditional snake game. Here the snake grows automatically (and fast!) and you should avoid crashing in itself or on the walls for as long as possible. A counter tells you how good you did on each try. This was the most interesting example because I didn’t had the code for it before. I coded it from scratch for these Processing.js demos, and took the opportunity to code it entirely using the API from JavaScript only (so, in this example, there is no Java code being parsed and converted). Coding Processing in JavaScript feels a bit different than traditional Processing, and requires some small tricks, but it is also very interesting because it is easier (at least, more natural) to interface with other HTML elements on the page via the DOM.

Electric Field

I did this code for an assigment (in a course at university) to create some visualization tool for an Electric Field. Each student used some tool and approach. As I used Processing, and the results looks cool (!), I also went on an effort to port this one to Processing.js. This is actually a simplified demo, because the original code was huger and crashing the browser frequently. It is extremely slow because there are gazillions of per-pixel calculations on these, but if you want to take a look beyond the screenshot, and you are patient enough, take a look. Add some electric charges using the left and right buttons (middle click works as right button too, if you wish to avoid the popup menu), and after that, click on the checkbox to view the Electric Field. Each charge takes some time to add, and the “View Electric Field” takes a long time*. The ammount of calculation, though, is always the same (it is not proportional to the ammount of charges), so you can add as many charges you want.
*The browser will say 3 or 4 times that a script is running for too long, and will ask if you wish to continue.
Please note that this example in special is very slow and may crash your browser, so open at your own risk!

I hope you find all these examples interesting. If you wanna view more of these, there are many many more examples at the Processing.js project page.

(Oh, all of these remember me once when I did an implementation of the Conway’s Game of Life all in Obscure C Code for a contest! =) Story for another day.)

We started out on this by analyzing the area and its surroundings from the point of view of circulation (more on how we did this). How would people move around? The conclusions from this study where integrated later on in the project.

Next we took on the task of (re)creating the landscape – modify the terrain in a functional yet unobtrusive way. Formal concepts revolved around radiolaria, foam, water, waves etc. Luckily we had a working circle packing applet in processing ready (which was used for this). From here on, it was quite straightforward: expand the circles to smooth spherical caps and smartly transform them into urban furniture.

The interstitial space that remained between the”bubbles” is packed with wood alongside the routes we discovered to be ideal using the circulation study. The rest is English lawn pure green smartly-cut grass.

This project is a collaboration between Veronica and me.

Tools we used include:
Processing was VERY important early on in the design process – we used it for the circulation study as well as for the early circle packing experiments (size, density, spread etc).

Rhino and RhinoScript: RhinoScript is great – we used it extensively (create spherical caps based on the generating circle’s radius, expand circles, contract circles, import circles which the processing applet generated, etc.). Rhino was used for everything else – 3D modelling and, of course, making valid STL files for the 3D printer and also exporting the right things for the laser printer.

Laser cutting was done here (as always).

3D wax models were kindly made for us by mazarom (at the moment the only 3D printing service in Bucharest). If you need a complicated model, don’t hesitate to contact them!

Plotting the final presentation was done at studio spot. They don’t have a webiste…

How things are going on:

Work done for the current school project by Veronica and me. This is just a snapshot, stay tuned for more. I’ve written 2 pretty interesting scripts for this one… so stay tuned.