This weekend I am participating in a hackathon at Startup Hub Timișoara where we will code on Project Tango devices. These are Android prototypes which track the surrounding and create a 3D model of the environment around.


I have no experience of developing on mobile devices, let alone working with computer vision. But with a book on Unity and a 10 hour train travel, I will study tonight.

 


The Train

I got myself the Unity Cookbook an worked some chapters from it during the night. I arrived at 7 a.m. in the morning in Timișoara, opened up Foursquare and went to the first place recommended by my friends here. I had to wait 30 minutes for the sandwich restaurant to open. But, I have the book "The Hard Thing About Hard Things" by Ben Horowitz, the venture capitalist from a16z, on my iPod. Listening to his business story really pumps me up.

Arrival

Startup Hub Timișoara is big. Being just an open space, I feel it's bigger than TechHub Bucharest. I've met with Radu Ticiu and the Project Tango devices have arrived.

The Device

iFixit has disassembled one for more details. It feels like a Kinect camera strapped on the back of a military radio station prototype. But the demos are awesome. The infrared camera is a depth sensor, so it knows if in an object is in front of a device. So you could build a robot that can avoid obstacles. 
example built by a Google engineer
The devices also know where you are in the space by using the gyroscope and a special kind of algebra. The problem with the units is that they are still beta and get hot quite fast.
cooling a device using a radiator
The hackathon will take 48 hours and will finish on Sunday afternoon.

opening ceremony

SketchDown

Their name comes from the opposite of SketchUp. They want you to take a picture and later select the focus. In general, you get the depth of field by using many pictures. But the Tango saves a JPG image with an RGBZ format, where the last 2 bytes come from the value of the z-axis. Currently they are still tinkering with the SDK and testing the device capabilities.

Dynamic ventures

These are 4 hardcore programmers that know what they are doing. Their background is in hardcore vision stuff. They want to reconstruct 3D objects, measure them and finally recognize them. The problem is that the data they need is not exported on the flash memory. So, they are trying to get it the data right from the device's memory through the Java API. Also, they have strip down the data, because they get it on grayscale, but the API returns it in RGB.

I've asked them what does it take to be a pro in this domain. Their answer was to learn your mathematics (OCW linear algebra anyone?) and to steal knowledge from an expert.
Mihai helping one of the engineers from Dynamic Ventures

Three pillar global

The guys from 3PG have a lot of ideas, but they haven't made much progress. They made the Unity demo to run on the Tango devices and know they are trying to customize it. Little step by little step is their motto. Without any prior experience they just want a ball to float and see it from all the possible angles.

A Google engineer

Mihai Pora is a Google engineer in Krakow, Poland. We talked about the differences between the Romanian and the Polish cultures, about how universities shape young students and about life decisions in general. He'll also be in the jury for this year's InfoEducatie contest, preparing some surprises for the hardware category.

CoderDojo Timișoara

This morning, in the open space we are currently coding there should have been a CoderDojo. More than 50 kids come each weekend to learn how to program. It felt for me like a chaos, but everybody seemed to have fun. About 10 mentors help each week the students, and CoderDojo has extended in more than 5 cities from Banat.
CoderDojo kids

Dancing snails

Their idea is to use the depth information from the photo and extract an object from the image. You can already do that with tools like Photoshop, but only if that object has a different color than the background. Their only experience is in Android development. So, they feel like the Tango documentation is not quite as descriptive as they would want it to be. But, the demos are some pretty strong starting points.
team Dancing Snails

Legendary Coders

The 3 guys from Legendary Coders want you to scan objects with the Peanut (secret codename of the device) and reconstruct them after. You can then use the model in other tools like OpenGL. They have a tangential experience with computer vision. Their problem is that the device gets hot fast. Also, the battery gets depleted after 10 minutes of using the camera.
team Legendary Coders

Tango360

These guys have the first working demo I've seen. They are scanning the room and reconstructing a 3D model of it in Blender. They showed me a nice model of our kitchen. It seemed like abstract art. The problem is for the Tango scan is not pretty accurate. And a table might result in having more than 4 legs. You have to be steady and straight when you record a room. Or else the phone might get dizzy and your model would become gibberish.


360 World

Ladies and gentleman, I've seen the virtual reality future, and it looks as real as it can get.



These is my first contact with a virtual reality world. And it's astonishing. The guys from Airglass took what looked like a helmet with two eye contact lenses, started an app that was running the same image split screen, attached the mobile to the helmet at the level of my eyes and put it on my head. And low on behold I was on a roller coaster. Everywhere I looked, there was another part of the world. The smoothness of the image was unbelievably real.

The guys told me that using mobile phones gives them a better pixel density then the Oculus Rift. And it's cheaper because you can attach almost any phone. They then demoed me a HUD. You would see an image taken by the mobile camera, and over that you had information displayed. Durovis Dive is the name of their full fledged product.

I talked business related stuff to them. They said that their niche is in aerial sports. In the video above, a  famous air-glider is exercising a pass between one of the most narrow hills in the world. Everything feels surreal.

the Durovis Dive using a Project Tango
But using a normal camera phone does not cover the whole field of view of a human. So, they want to use the Tango's wide camera for a demo.

ARDU

My team and I are trying to mount the Tango camera on top of an Arduino - RaspberryPi car, that gets around in a room. It tries to avoid obstacles by using the depth camera.



I didn't know these guys before I arrived here. We all mistakenly arrived 3 hours before the contest had started. So, we had time to talk and get to know each other. They were awesome and we worked a lot helping each other.
from left to right: Daniel,  Bogdan,  me, Vlad and Viorel who is at the laptop

Organization

Startup Hub Timisoara is an amazing place. The big open space encourages every the team to talk to each other. Also, we have plenty of food and it's delicious. A nice suprize was to see that we were 3 vegetarians and had separate tasty meals. The accommodation was in a cozy student's dorm.


Tangoception

This morning we had a surprise prepared by the organizers. Three people came in the middle of room and started to dance Tango. Some geeks also went out of their comfort zones and started learning how to dance Tango.

The demos

About 45 people have participated in 11 teams. We are now starting the demos.

Our demo was quite unsuccessful. We decided to go first, but one of our pins that transmitted power to the engine failed. So we left the guys from 360World go first. After we managed to solder the pin we went to do the demo in front of the other people. It worked, but too much power got transmitted to the other pin, so instead of going straight our robot was deviating a little.

our robot, preparing to avoid the chair
I'm sorry I missed the Cloudbase team, but now I see their demo. It's a game in Unity named Egg Hunt. Eggs appear random in the room and you must go and pick them. It was funny because eggs spanned all over the room and they ran to people to pick up the eggs. They also used the occlusion effect and if a wall was in front of you, you had to go to the next room to see it and pick it up. Or downstairs sometimes. It seemed like a fun indoor version of Ingress.

an egg appearing near the public
Also missing from my interviews above is the Rombots Team. Personal Space Invaders is the name of their game. The game uses the 3D depth sensor to scan the environment to find where are the resources and generates a level. It's like a tower defense game using the room.

Aliens attacking in Personal Space Invaders
The Legendary Coders use face detection to extract 3D humans from the camera. They put a green rectangle over the face with the distance to that human. You take the photo and then you get a 3D model.

They generated a 3D model of a human on the spot from a picture
Now, Tango360 is presenting. They told us that the road from what you want to do to what you do is difficult. They've made a map from the kitchen in StartupHub, put some flies on it which you have to kill. You can recognize the chairs, but there is a lot of noise due to the low resolution of the infrared sensors.

Dancing Snails wants to replace the ugly green background in TV that get replaced with a weather map in post production TV. They on the fly extract a human from the photo and put a background like an underwater scenery.


3PG has worked on 2 projects. They put a ball in the room and when you kick it, it goes further away. They would have want to have collisions with the walls, but could not reconstruct the model. In their second project, they get a frame every two seconds and reconstruct a 3D model on the fly. They only keep 6 frames of it and display cubes for every point of the depth sensor.

the ball from 3PG
Unfortunately, I could not stay until every team presents their project.

Update 2: you can see in this video a compilation of interviews and footage taken from the hackathon.

Winners

The two winners who each get a Alcatel OneTouch 8008D are 360 World and SBRL. The jury was impressed that they not only used the SDKs provided, but also crunched their own data. You can watch the full presentation of one of the winners in the video below.


Wrap up

At the end of the weekend, everybody was happy with the results. We had, unfortunately, to return the devices, but if they will ever become a mass product, then a big part of this world will change. Some nice articles have appeared in the local Adevarul newspaper (only in Romanian).

Update: Radu from Legendary Coders has written about his experience at the hackathon. You should read it because he shares some nice bits of code

So Long, and Thanks for All the Fish
Some photos are taken from the Startup Hub, TiMoDev and Mihai Pora's G+ album.