Home / Posts tagged "Installation"

Touch table

I build this projection / touch table for a while now, never got a chance to write about it until now. I got this idea last year and in the time I need a working table for myself so I think : why don’t I just build one for both working and projection ? The idea is simple : Just make the top of the table removable and keep the width / height ratio to 16 : 9 which is the aspect ratio of my projector.



Building the table

For the frames I got some pieces of wood left from my ikea shelves and found a big and thick piece of wood in my backyard which is perfect for the top. It took me about 2 days to build and I don’t have proper tool for this, it will be much faster with the right tool. And of course the quality will be much better too 😀

Projection and Touch

When I want to project I just remove the top and cover with a sheet.  The way that the touch works is that I put a Kinect under the table and facing straight up. So when I press on the sheet the Kinect can capture the depth difference of the press point. It’s not a complicate concept but just a lot of tweaking and calibre, e.g find the right distance range to detect, ignore the frames, noise reduction … etc. However there’s one thing does matter a lot, which is the sheet. I was using the bed sheet, it works but it’s not very flexible so when you press you also pull down quite a big area, therefore it’s not very accurate. Then later I found a really flexible piece of cloth that when you press it can create a small point which is perfect for position detection.


And then the next just use this point as a virtual mouse. Theoretically it could detect multitouch as long as the sheet can shows different points you press, but also need an algorithm to find all the different points. I haven’t tried openCV yet, maybe there’s some thing to use.



It’s a simple and silly idea, also the table is really shaky, but I really enjoy it.  I especially like the touch feeling, it’s very satisfying. And also building the table itself is a lot of fun too, I really enjoy building real stuff that I can actually touch it, it’s very different from code but both very interesting to me.

Star Canvas – Case study

Last thursday it was the 5th anniversary of B-reel London, we had a party on thursday night along with couple of our R&D projects. And I am working on this one : “Five”, it was based on one of my old prototype :


The idea is to show all the constellations on the sky, and also we created a drawing tool so people can create their own constellation and send it to the sky. We find a way to build a dome and project it inside and we did it. It’s my first installation, really fun and a lot of learning, and I need to say thanks for my colleagues who makes this possible. I want to write a little bit more about this project, I think b-reel will put together a beautifully edited making-of video, so in this article I will be focusing on the tech/dev part.

The installation

Here is a diagram of the basic structure :



Our developer Liam build the communication using node.js as the backend server for this project. Apart from the projection, we have 2 drawing tool running on iPad allow user to create their own constellation, and also a virtual keyboard on the iPad for people to use it to search the constellation they created. We don’t want to use a real keyboard because people can miss-trigger other applications, therefore we create this keyboard and limit the usage only to this project.

The projection is built using WebGL, at the beginning I was considering using Cinder to build this project, however due to the limit time and also because I had built a working prototype in webgl, I choose to use webgl in the end. I wasn’t too sure about it when I choose it, I didn’t know if the performance will be good enough and is it stable enough to run through all night with a lot of communications, but it turns out work quite well, I was thinking that i might need to restart the whole thing couple of times during the party but in the end i didn’t have to. I’m really happy with it and feel more security now with using WebGL for installations. And also i’m not using any webgl libraries but just some tool class i create, this is another big achievement for me as well.

The sound is another big part, done by our sound master Owen. All I did is send out the event base on the hand gesture and the constellation selected to the node server, then it gets re-send to pure data to generate the sound effect. When a constellation is selected, I calculate the number of lines in this constellation and max/min/avg distance of these lines and then send it out, and Owen create different melody based on these informations, so each constellation has its unique melody. I would really like to bring this feature to the web version, but i need to learn from master Owen first 😀

For the drawing tool we capture user’s strokes and then simplify them to fewer dots to create the “constellation like” kind of effect, then we insert this generated constellation to the database and in the same time send out a signal via node server to tell the projection to update the display.

And the keyboard is just simply transfer the event of which key being pressed to the projection, very straight forward.


The dome

We discover this article teaching people how to build a dome just using card boards, and the best is that it already has all the detail of triangles you need including the plan and sizes. Once I saw this i couldn’t help but want to build one myself. Me and my wife start right away to build a small scale one to test, here are some pictures :

It’s really easy to build, even with the actual dome we build is not that complicated either, however the problem is how to support it, the geometry itself is flexible, in order to make it stay in a perfect shape we use a lot of fish strings attached to almost every corner to hold the shape. This is actually the most difficult part, building the dome actually is much easier than this.



The other challenge is how to project on to this dome, in the link they are using a hemispherical mirror which we did try, however the result is really not acceptable. The first problem is when reflected to the dome, it loses a lot of details and become very pixelated. The second one is the distortion is really hard to correct. Due to this 2 reasons we give up using the hemispherical mirror. Then I tried to project it directly on the dome and then correct the distortion in the code, however we find actually it looks better just leave the way it is without doing any correction. Maybe this is because the nature of this project that everything is already on a sphere so there is no need to do more to correct it. All I need to do is just create a gradient circle mask to mask out the part outside the dome.



The constellations

This project is not actually super 3D heavy, the only 3D part of it is everything is on a sphere and that’s all, the stars are flat, the constellations are flat, almost everything is flat. This is the nature of this project, we don’t need super complicated 3D models for this. On the other hand, we want to push a little bit more to the visual parts, to push as many layers as possible, as we all know the more layer you have, the more detailed / beautiful it’s going to get. So here is a little demo of the project and you can see all the layers being rendered  :


We started from a basic skybox, however soon after we build it we think it will feel much better if we can have an animated background, therefore our motion designer create an animate background for me. At the beginning we just put it as a plane texture on top of the skybox, but then we discover it will look better to map it onto a sphere that moves with the skybox, this gives the animation a feeling that it’s the real sky instead of just an overlay.

I’ve already blogged how i make the stars and lines facing to center/screen in my previous post, it was in stage3D but it works in the same way.


I’ve put a fake depth of field effect on the name of the constellation and the stars so it gets a little bit blurry/transparent when it gets closer to the edge. It’s a fake depth of field because i didn’t use the depth buffer, it’s just a simple calculation base on its x and y, however this is very suitable for this project.


For the appearing of the constellation, I had a little fun with the fragment shader. I want to make a simple particle effect transition to show the constellation drawings. I found this useful glsl random function on shader toy :

float rand(vec2 co){
    return fract(sin(dot(co.xy ,vec2(12.9898,78.233))) * 43758.5453);

to create this effect is actually quite simple : When getting the texture coordinate, add this random number to the desired coordinate, and this will result getting a random position therefore it looks like particles. And to create the animation is just to tweak the amount of this offset :

targetPos = orgPos + offset * randomPos;

so you can see it here : the bigger the offset is, the more random it gets( more particle like), and if the the offset is 0 then we get the original position which will generate a clean drawing. So basically the animation is just to tween offset from some big value back to 0. Voila, that’s how simple it is. You can add more to this random position such as scale or rotation to give a more dramatic effect.

And in this project we used a lot of video textures, some of them we need transparency, here is a easy way to do it : using the blend mode, before you render it, set the blend mode to this :

this.gl.blendFunc(this.gl.SRC_ALPHA, this.gl.ONE);

and make sure the transparent part is pure black, then it will get removed when being rendered. This trick is not only for video but for all the textures, so you can save some file size if you use this wisely.

The video texture is quite heavy for the loading, some of them can be done in code but will be very difficult to get the detail as rendered video. I think this is the choice based on the project, in our case we are building an installation so we don’t need to care about the loading, and we use super powerful machine to run it, so the file size is not a problem. In this case I will choose to use videos to get the best detail and also easier to modify. However if we are building an online experience then we need to do more tests on the performance and loading. Anyway my point is : Choose the best solution for the project, I know i’ll have a lot of fun playing with shaders if I were building it in code, but it will be very time consuming and hard to change.


The navigation

This is the first time we use leap motion for a real project, it turns out working quite well. I won’t say that it’s going to replace the mouse but definitely it can provide an extra way to navigate. The part I like about leap motion is it’s really sensitive and responsive,  you can create a really good control with it. However some gesture are still very hard to use, especially everyone has its way to do the gesture. At the beginning or you can see my prototype video, i created this “Grab” gesture to navigate, to be honest i quite like it, it gives me the control of holding something, however some people find it difficult to use, and it’s really hard for me to improve the gesture because people have different ways of “grabbing”, it sounds a little bit funny but it’s what I encounter during this project. So in the end i have to remove this grab gesture and goes for full hand. If you have a leap motion you can have a play with the link i mentioned before. We have 3 gestures : full hand open to move around, 1 finger pointing to select, and clap, this one i’ll leave for you to discover 😀

There an interesting part of navigation : How I select the constellation i want ? do I need to compare the distance of every constellation to my mouse’s position ? this sounds quite heavy and need a lot of math. Luckily our ancestor has already solved this problem, basic astronomy : there are in total 88 constellations on the sky and have taken all the space. There’s a way to determine the boundary of each constellation using Right ascension andDeclination, which is the IAU constellation boundariesSo basically these scientists has already create the boundaries of all the constellations, and you can find a map like this (of course without the name, i put the names on just easier to see)


when you map it to a sphere you can see it fits perfectly with all the constellations. So what i did is to paint each region with different color, and then when the mouse event triggered ( mouse move or click), i will perform a gl.readPixels to get the pixel value of the mouse position, and because each region has an unique value, therefore i can know which one i just selected ( or roll over ). Just couple things to pay attention : when you doing the readPixels you don’t need to read from the whole render, you just need that 1 pixel under your mouse, this will save some performance. However the readPixels call is still heavy so make sure to skip it whenever you can ( e.g. when the mouse is not moved, so it’s the same result as last frame), secondly when you export this map make sure you export a png so it won’t get compressed and loose the value you set.


This is about the longest post i have ever made but i am really proud of this project. I am still working on the last part of this project with my friend, there’s something we want to do with this project and would like to bring the beauty of the constellations to everybody. We have discovered the beautiful drawings of  Johannes Hevelius and we want to show it to everybody. So stay tuned for the updates!

Since I started working on this project i fall in love with astronomy. Our sky is amazingly beautiful, every night if the sky is clear i will look up and try to find the constellation i know. And I realise no matter how much I do I cannot compete with the beauty of the nature, however I can get inspired by it. There is a lot of inspirations you can find just by looking at the nature around you.

Some pictures of the night