Nitin has posted 391 posts at DZone. View Full User Profile

Simon Free: Merapi Robots and the World of Augmented Reality

  • submit to reddit

In this interview, recorded at CFinNC 2009, Dan Wilson speaks with Adobe Community Expert Simon Free about Merapi, a bridge between applications written in Java and those running in and created for Adobe AIR. Simon describes how he built a desktop AIR application that talks, through Merapi, to a Java library that controls a Lego Mindstorm robot. Simon also describes some of the practical use cases for Augmented Reality (AR) -- applications which involve the overlay of virtual imagery on the real world -- and FLARToolkit,  a software library for building AR applications.

The complete transcript of the interview has been provided below. 

Dan Wilson:  Hey, folks. Welcome to another interview series with DZone. My name is Dan Wilson, and I'm here with Simon Free, ColdFusion personality, Flex personality, generally nice guy, and he loves to play with robots in his spare time. Simon, how are you?

Simon Free:  I am good. How are you, sir?

Dan:  Oh, doing well. Thanks for coming out.

Simon:  My pleasure. My pleasure.

Dan:  So, for the benefit of the viewership, tell us a little bit about yourself.

Simon:  Sure, no problems. My name is Simon Free, Adobe Community Expert, Adobe Certified Instructor, been in the field for about 11, 12 years. Originally from England, so hence why a little bit of an accent in there. But yeah, been stateside for eight years or so, based out of New York, in Manhattan, and kind of one of those guys who goes around on the conference circuit doing presentations.

Dan:  You have a very busy conference schedule this year.

Simon:  Yes.

Dan:  Tell us some of the stuff you've been working on. Where's your background?

Simon:  My background, mainly, professionally, a lot of ColdFusion, a lot of Flex. Been working with some very large companies, working with advertising media and things like that. And then, outside of work is I've been really playing around with some cool things such as Flex applications that are working with augmented reality or AIR applications that are using Merapi, so I'm able to control robots and have them run around the room and things like that.

Dan:  So, for those who aren't familiar with Merapi, can you talk about where that fits into the stack?

Simon:  Sure. So Merapi is a bridge between AIR and Java. Now, we all know that an AIR application is in a sandbox. There's a lot of things we can't really do with AIR, for security. And rightly so. I mean, we don't want people to be able to make AIR applications that can just wreak havoc on your computer.

Dan:  Right.

Simon:  But with Merapi, you can set it up so it'll be a bridge between AIR and Java. Now, a lot of things in everything are based on Java. We've got APIs for different pieces of external hardware, scanners or RFID tags and things like that. So we have these APIs we can talk to. Now, through AIR, using the bridge, we can send the communications to Java to do all the communication, and then Java will send the messages back to AIR, and then we can do whatever we want with it. So, with that, I was able to actually use the API that was created for Lego Mindstorms, and I was able to create the AIR application that would control a little robot that I made.

Dan:  So the Lego Mindstorm is a teaching robot.

Simon:  Yes. Lego marketed the robot to go out to a lot of schools, colleges, and things like that, to show a lot of robotics stuff. But they also allow people to look at the programming aspect of it as well. So you can actually build programs, put them onto the robot and have the robot do things. There's a lot of sensors that you can use as well, such as light, color, distance, so you're able to make some really great technologies. And they also packaged it with an API for those of us who want to do it over Bluetooth so we can do some actual live communication with the robot.

Dan:  So you have an AIR application on a desktop that talks, through this Merapi library, to a Java library that communicates over Bluetooth that then directs a robot.

Simon:  Yes. And then, in my playing around which I've been doing, I actually added a step further. So we have the AIR application which then actually communicates with ColdFusion, and we use ColdFusion as a brain which is able to work out all the available commands that we actually want to do with the robot. That sends that back to the AIR application, which then sends it over Merapi to Java to then control the robot.

Dan:  So now you have a ColdFusion back end communicating with AIR that communicates with Merapi that communicates with Java that communicates with Bluetooth that communicates with a robot.

Simon:  Yeah.

Dan:  Sounds complicated.

Simon:  Oh, it's totally simple.

Dan:  [laughs] So, what sort of business case would you recommend this technology stack for? It sounds like a fun experiment to waste away hours on end, but is there a business need? Is there a way that companies can use this for fun and profit?

Simon:  I mean, technically, I would say yes. With the robot, for example, there's lots of sensors, such as distance, color, sound, and things like that. Now, if you imagine being able to build a robot that can sense where walls are, you could create a robot that's going to go, and you put it into a room and, just like the Roomba vacuum does, goes all around the room trying to find where the walls are. You could set it off in the morning and come back at night. That robot could have sent all its findings up to an AIR application that could have mapped out the entire room. If you put that into a real‑estate market, you're going to give these real‑estate agents a robot they can put into the room, and it will create the entire floor plan for the house for them while they're off showing houses elsewhere. You've got a really useful product right there for them. It saves time, makes things more efficient, and also is cool.

Dan:  Cool is definitely important.

Simon:  Oh, I love cool.

Dan:  So where does AIR fit into this? Surely you could have used a different platform, like Java Swing, to control the robot. What does AIR bring to the table?

Simon:  AIR itself, especially when you're making Flex‑based AIR applications, has got a very fast learning curve. A lot of the things which you've got available for Flex are actually potentially drag‑and‑drop items. The Flex library itself has a lot of predefined and preset pieces of UI, and also, even when it comes to some of the functionality as well. It's very simple to actually create an interface and have it interact with ActionScript code, which can then communicate with everything else which you need to add, whereas, if you're looking for some of the other Java‑based things, there's more of a harder, harsher learning curve. If we use AIR, we're making it simpler.

Plus, AIR itself is extremely simple and easy to install on other machines as well. You don't have to worry about the different operating systems that you've got available out there: PC, Mac, or even Linux. AIR will work across all of them.

And all the libraries will take into account that you just make a call, and the AIR itself will decide what it should do underlying, based on the operating system. So you've now got that decision process taken away from you. You don't have to worry about it.

Dan:  So now you can actually write one client and deploy that client across multiple platforms without having to manage inconsistencies.

Simon:  Exactly.

Dan:  And what about auto‑updating? Does it have any capability for keeping itself synced?

Simon:  Oh, that is so smooth, so smooth. You can actually just use the simple auto‑updating library which you've got with it. When you're building your application, you say which version you're currently on, and what it will do, whenever it starts, it'll check a URL to see if there's a new update. If there is, it'll say, "Hey, there's a new update, " and it'll install it. That's all. You don't have to have people download anything directly from your website. It'll just do auto‑updates.

Dan:  And these are self‑contained packages?

Simon:  Yes.

Dan:  That's really nice. I'm currently dealing with a classpath problem in Java where I have to keep stitching together weird open‑source libraries, and it's a bit of a pain.

Simon:  Oh, yeah. You won't have that problem.

Dan:  Now, you're also involved in some other cutting edge technology, augmented realty. Talk a little bit about that. First of all, define it for the benefit if the views. Then talk about some of your experimentation.

Simon:  Sure. Augmented realty is one of the new things. Well it's not too new, but it's really hitting the market recently. There are a number of organizations who've done some augmented realty stuff, which is really where you take a web cam, a video feed, and the video feed itself will look for a kind of pattern. It's normally a box with a thick, black border, with some kind of patter in it. What it will do is is it'll find that within the video feed, and it will overlay on top of it some kind of design element. May it's a flat box, or a circle or something like that. Or maybe it's a whole 3D person. So you can actually take that, and on the person's screen, even though they are holding a piece of paper in front of them, if they're actually look at themselves there will be a 3D object on that piece of paper.

Now as you move the piece of paper around, where the actual pattern is moving around, the 3D object will be tied to that and will move around as well. So even if you've got a 3D cube, as you're moving it around the shadowing and shading, which is in the cube, will as changed based off the position.

Playing around with it is a little ‑‑ you can't really see too much from it because it's just a box. The business use cases for it isn't that great. But USPS did something where they took, through the augmented reality, they project a box on a piece of paper.

This matches the packing boxes they have. So you can see exactly how big the box will be, and you can start placing items around to see if they are going to fit inside the box. Which I think is a really good business use case.

Dan:  So to summarize, you have some sort of pattern, and there's a web cam recording that recognizes that pattern, and allows you to programatically overlay content that's spatially aware on the video feed that's being played back.

Simon:  Exactly. It doesn't have to be just a webcam, it can be an video source.

Dan:  OK. The USPS thing, what it does is it allows you to see what will fit in boxes.

Simon:  That's useful because who wants to take your items to the post office to see what's going to fit.? But are there other use cases for this?

Dan:  Well if you think about it, the webcam itself is looking ‑ whenever the feed is on, it's always looking for new patterns. You can add as many patterns as you want. So you could have whole library of available patterns, where it will go and it will see if it can match those patterns. You don't actually have to overlay something, that's a nice graphical thing which we have going on for users to see what's going of. But if matches these patterns, and you know that pattern 485 was matched. Then you can send that to the system saying "Hey, I found this, " it's found that. So if you had in a warehouse, where you have a little machine with pattern on that which is scanning similar to bar code, it can look and it can find it's patter. Then it can be "OK, we found this box." The commands can go top the different commands to pick out the box.

Or just having a robot that's going around searching for a certain location, so it knows which site it is in the room, or things like that. It can see that pattern, and it will register "Oh, OK. That pattern's there, which means I must be in this quadrant." So there's a number of potential features.

Simon:  So a box with a square, blackboard is a pattern. A bar code is pattern. Is a face a pattern?

Dan:  No. When it come to faces for this augmented reality stuff, it's harder for it to tell. Because if you think of the video feed coming in, it's just a series of images. We've got frames‑per‑second, it's really just how many images we're getting per second. It scans that entire image trying to pick out pieces of information to see if it matches or not. A face itself has got to many complex features for it. There's going to be shadowing issues, there's going to be the shape of the face and things like that. So trying to find something within a face is going to bit a little difficult.

Simon:  A bit tricky.

Dan:  Yeah. I'm not saying it's not going to happen...

Simon:  Of course not.

Dan:  ..but just not quite now.

Simon:  OK, well that's very interesting. I appreciate you sharing a little bit about augmented reality. It certainly has a lot of buzz these days. That seems like it's sort of on the cutting edge of where things are going. I have a Blackberry that has a video recorder in it. So image that could be used in some sort of way to provide an augmented reality experience, possibly in a store or some other way. Since some of you background is in advertising, talk a little bit about how you see augmented reality fitting into that channel.

Dan:  There are many potential things which you can have with augmented reality. Especially if you're doing something with a cell phone. Imagine in a store, having patterns by clothing items. Where if you were able to look at it, it would show different sales which were going on for those jeans. Or you could have it or additional information. Because if you look at the back of you're jeans, you have a label which tells you what the jeans are made of, how you should wash them. It's not exactly that readable. So maybe if you had that, you could see additional care instructions for those products.

Or you could actually use it from a game standpoint. Imagine if you inside you your box of Wheaties, you had a little pattern in there. And you had to go to the Wheaties website and show that in front of the web cam to see if you've won. You can kind of use it for those kind of purposes as well.

Getting back to the clothes thing, I absolutely hate trying on clothes. I would almost rather buy the clothes, and then if they don't fit, take them back, than sit in the dressing room. So if I could somehow cam up and see what the clothes would look like, I would know whether or not it's worth even bothering trying it on.

"Does my butt look fat in these jeans?" kind of a thing. I'm sure that's a useful item.

Simon:  It is, you know there, the only thing which is limiting what is created with augments reality is just people's imagination. There's so many things that it can do, we just need to think of what we need to do.

Dan:  Video is the next thing, at that seems like a great way to fit in there. So where would you want to send folk for more information about augmented reality?

Simon:  For augmented reality, there are a number of websites out there with demonstration code and things like that. Such as the FLAR Toolkit, F‑L‑A‑R Toolkit. There's locations there. Really just do some Google searches. There are a lot of researchers out there, there are a lot of places out there with information on them, blogs and things like that.

Dan:  FLAR is a Flash‑based toolkit.

Simon:  Yes, but there are a lot of people who've ported it over to Flex as well. So you've got flash where you can use it, and you can use it in Flex as well. Don't forget, Flash and Flex both use ActionScript.

Dan:  And they both do video.

Simon:  Yes.

Dan:  What about your own resources for yourself? Do you have a website of a blog that you'd like to mention?

Simon:  Sure. I'll be posting up some examples and things like that on So they can go there. I'll also have a nice list of resources there as well. I send people to that for AIR applications, and augmented reality.

Dan:  I would love to see this robot of your controlled by ColdFusion, AIR, Java, Bluetooth, Lego stack. That seems complicated.

Simon:  Oh, definitely. We're going to be posting a video of that real soon as well. So definitely come check it out.

Dan:  Simon, thank you so much for coming out.

Simon:  Thank you very much for having me.

Dan:  This is Dan Wilson for DZone, thanks for tuning in.

Published at DZone with permission of its author, Nitin Bharti.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)