This week Winslow and I made some great progress on our ICM/PComp finals, tentatively titled "Sharp Shooters." There's probably a better name that we'll end up with, but it's working for now. I'll update our progress in three areas: filming, coding and construction. FILMING We made the most progress this week in developing our storyline and actually shooting all of the scenes. Both of us agree that it doesn't matter how interactive the technology is if people aren't interested in the story, so we're working hard on that. The first scene we shot was with Michelle's dog, Saul. He was very cute, but we probably shouldn't have started by working with animals. It was tough because we couldn't keep him framed against the wall, and we also weren't able to get the exact shot with the green ball that we needed for the shot to work. After that experience, we moved on to a much more agreeable actor, Dimitrius. He played the role of the doorman, and he did a great job. As we were shooting this scene, we realized that our original idea - to have the celebrity visit all of the floors and get objects to use on other floors - was a bit more complicated than necessary. It made more sense for the celebrity to receive different parts of the disguise as the user moves between the floors. With that in mind, we made a few changes in our scenes so we could get back on track. CODING This week, we received some valuable feedback from our PComp classes after we presented our project. The main critique was that the story, while interesting, was a lot of work for very little reward. The opinion was that we needed to simplify the construction and add more interaction to make this a fun experience. So while we were filming, we worked on some code ideas. We want to incorporate two different games that users will play in order to clear their floors. Here's a list of our code challenges:

• Serial communication between the force sensor representing the elevator buttons and Processing that will allow us to simulate movement between floors;
• Serial communication between the LEDs and Processing that will indicate when the floor has been cleared;
• Code to capture an image of the user when s/he begins the game that will be integrated into a layer later on;
• Code to layer a captured image with a pre-exisiting graphic;

CONSTRUCTION We made some good progress on building the tower this week. I went down to Canal Plastics and purchased a piece of two-way mirror ($40!) that we can use for the front of the door. We also need to buy hinges for the doors and start thinking about how we want to wire it. CONCLUSION Good progress this week, but we have Thanksgiving next week and only one week after that to get it all done. I know we will complete the project, but we have a lot of different tasks to work on at the same time.

It's pretty interesting to watch how my final project is evolving week over week. Last week, Sharp Shooters was only going to be my ICM final, and this week, it has transformed to both a group project (working with Winslow Porter) and also a combined final for both my ICM and Physical Computing classes.  In last week's post, I described the background inspiration and research for this interactive video/game. This week, Winslow and I made a lot of progress in bringing this idea to fruition. First, we simplified. The success of this final is not about using lots of different types of technology so much as it's about having a simple, fun narrative that engages the user. To that end, we really want a story that is easy to tell and to follow and that the user can experience within 2-3 minutes. I think we're almost there, but it's a lot of work getting the story to that place. We've done a lot of brainstorming over the last week, as you can see. Here is one of our early attempts at organizing our game:   which then turned into: which is now: THE STORY On the table in front of you (in real life) will be a video camera, an elevator control panel and a small scale replica of the building in which you, as the main character, are located. The main character will be you, which means that everyone will talk directly to the camera. As it starts out, your assistant throws a newspaper down on the table with your picture and says that an embarassing story has been published about you. The paparazzi have staked out the building and you have to get out without them seeing you. He/she says that the residents of the building are pretty loyal and will help you to escape. Your assistant then puts on a disguise and leaves the room, giving you time to get to the elevator.  You run to the elevator and start going floor to floor. When the doors to the elevators open, some of the building's "odder" residents are there. In their own special way, they try to help facilitate your escape. After you complete the task on each level, you unlock that level of the building. Only after each level is unlocked will you be able to escape unnoticed. Our work is really now divided into three parts: filming, coding and construction. FILMING This has proved a huge challenge thus far. There is an ample supply of willing actors, but finding a location to nail down has been difficult. It's amazing to me that ITP has so much camera equipment yet no agreement with the film school or any other department to secure space for us to film our projects. We have asked friends and put out notes on the list serve to find a location, but we're still waiting on a lot of places. Since Thanksgiving is coming up, we have a pretty tight deadline for filming. The goal is to shoot on Friday and Saturday so that we can move on to the more crucial part of this assignment, namely the coding and the construction. CODING Our initial idea is pretty ambitious, and the coding challenge right now is mostly organizational. There are a series of options on each level of the building, and we have to organize the logic for that. In addition, each level has a different challenge the user must overcome in order to move forward, so we're working with several different libraries (minim, to start). CONSTRUCTION There are two objects that need to be built: the elevator control panel and the small scale replica of the building. We are going to start with the building replica. The idea here is that the plexiglass structure will have the same number of floors as there are floors in the game. In each floor, there will be an object that is unlocked after you complete the level. The object is initially hidden behind two way glass, and then revealed with a bright LED upon "winning" the level. Here are some initial images from constructing the tower: and and GOALS FOR THIS WEEK Complete construction of the tower, complete filming and make significant progress on code.

Last week, I had an entirely different idea for my physical computing final. The feedback from class is that in order to make it work, it needed to be a lot more interactive. I had been excited about it because it was really more art for the sake of art and trying to build something that was both beyond my current feedback and pointed for no other purpose than to create something beautiful. While the idea is still of great interest, it might not be interactive enough for this particular assignment, so I am going to go another direction. Instead, what I'd like to do is try and combine my ICM final with my PComp final to create an interactive game experience that relies on both sensors and software programming, which means I will be working a lot with serial communication. BACKGROUND For my ICM midterm, I created an interactive video where the main character was trying to get from his home to the subway. When he would reach certain points in the journey, he would hold up two different colored balls. The video would then pause and the video camera on the user's computer would turn on. The user would then have to hold up a ball corresponding to the direction that the character should move. If it was the wrong way, the character would get beat up and the game would reset. I had a lot of fun with this project, and I would like to take it to the next level for my final. I actually think it's a good fit for both classes because a story/game like this involves the user in a very interactive way. They are not just using a joystick or wiimote to navigate through the world that we create. Our goal is to make the player part of the story and equip them with the tools to win the game, which I'll explain in more detail below. RESEARCH I'm going to work on this project with Winslow Porter from the afternoon class. The goal is to first come up with a compelling narrative and then determine the best ways to apply some of the different technical executions we have in a meaningful way.  There are a few examples of interactive games that struck us as fascinating in the way they took an existing technology and repurposed it in a new way: In "A Car’s Life," participants watch a brief video of a car meandering over some difficult terrain. At some point towards the end of the piece, a button appears for viewers to click. If you do so in time, you advance to the next level and watch the car overcome even more obstacles. If you don’t click the button in time, the car crashes and the video ends.   Hexolabs used YouTube’s annotation feature to insert links into each video connecting them to other YouTube videos they created for the various levels. In doing so, Hexolabs created what is probably the first interactive video game on the popular video sharing site. Twix created what they call an "interactive love story, (sort of)" where the user watches a clip of a movie and then checks a box to determine which way the story goes next. Here's a sample clip: While both of these are innovative uses of video, we felt they could still be more interactive. NEXT STEPS There are a couple of things we need to do next. We first need to come up with the story that we want to tell, and then we need to determine what types of physical computing and processing codes we need to write in order for the user to make choices and progress through the game. THE STORY We came up with a storyline that we like, although it still needs a little more work to get it fleshed out. The premise is that you are a celebrity. The camera will act as the main character to make this feel as personal as possible. Your assistant comes to you with a valise that includes a number of items, including:  

• cell phone
• subway card
• green apple
• sunglasses
• street map

THE SYSTEM So once the story is ironed out, the next thing we have to do is shoot the video. There will be somewhere around 16 separate scenes that will need to be shot, edited, mixed with music and the fused together into one long .MOV file. The rest of the work (and the most challenging) will then be to determine the order in which movie is played, thus the "choose your own adventure" style. This is decided by the various programming and physical computing processes we put into place. I am estimating there will be either five or six choices that will need to be made, thus giving us five or six places where we will need the user to choose between two options. Depending on their choice, the video will either continue or a final scene will play out indicating that the game has ended. We want to leverage the items in the valise to make the choices more interactive. Here are some of the options we've discussed so far:

• User takes a picture when s/he starts the game, which is then incorporated later in the story line;
• User is asked to send a text message to a specific number, which will auto-return instructions for making the next choice;
• User will have to choose between three options on a subway map, one of which will progress the story while the other two will result in game over sequences;
• User receives an email upon ending the game (contents TBD);
• Color tracking apple;
• Voice recognition (clap once to go left, twice to go right);

For my ICM midterm, I had a lot of different ideas. I thought about building a teleprompter for bloggers and I considered building a video program that would track colors on the screen and create a disguise based on some really colorful things people could put on their faces. The problem was that I just wasn't too passionate about either of these ideas. So I thought some more and decided that I wanted to try and use Processing's brightness tracking video feature as a detector that, based on the colors read, would facilitate some other kind of action. Premise I created a video around the premise, "Help Me Get to the Subway." As the character tries to get to the subway, he reaches three places where he can choose between going two directions. He holds up a brightly colored ball in each hand, each representing one of the choices. The user then holds a ball up to the video camera to designate which way the character should go. The camera reads the color of the ball and the video then jumps to that point to continue the story. If the user chooses the wrong direction, the character gets beaten up and the game restarts First Step: Filming I started by filming all the video. I had the actor walk to the subway and when he got to three separate points, he held up the two balls. Then we had to film him going the right way and going the wrong way. When he went the wrong way, I had two actors beat him up. It took about an hour to film. Second Step: Coding Once I had the video, there were two different pieces of code that I needed. The first was to jump between video segments. After reviewing the available code online, there were two choices. I could either have different videos start and stop or I could make one long video with all of the choices and program Processing to jump to different points. The latter seemed to make more sense. The next piece of code I needed was for the video camera on my computer (or an external video camera) to register the different color balls and jump to the different parts of the video based on it. This was the most frustrating and most difficult to figure out, but Shawn was a huge help during his office hours. Once I had the code, I had to test it. Obviously the hardest part was the different lighting conditions, which affected how bright the pixels were on the screen. Here's how I set up the balls so that I could find the average brightness:

I really liked this project, and I'm thinking that I will continue using this system as I think about my final project. The hardest thing to control of course is the lighting. I had it working perfectly in the Japanese room the night before I demo'd it in class. The next morning, I got to class early in order to try and recalibrate it for the brightness of the room. That didn't end up working too well and I could only demo one of the three threads. The feedback was that I needed to change the way that I found the average brightness by putting more limits on it, rather than just using greather than and less than.

Here's the code.

Too dorky to actually upload verbatim, but the assingment was to write a story about a day in my life 10 years from now. I wasn't sure what to write and went through a myriad of drafts. I ended up writing a keynote address that I imagined giving sometime in the future. Just to give you a sense as to what level of geekiness I achieved, here's a few paragraphs: Things have changed a lot over the years. When I came out of graduate school, I was inspired to make a difference. I thought about the normal, everyday experiences and interactions that I wanted to change. For example, when I wonder in to the kitchen. Wouldn’t it be cool if the room greeted me and adjusted the lighting based on the time of day? What about when I grab the handle of the refrigerator and pull, wouldn’t it be cool if it read my pulse and rotated my favorite foods to the front. I would want it to know when I was coming for a midnight snack so that it would whisper its greeting and I would want it to talk to the kitchen so that it knew not to tell my wife I was even there. You probably take many of these things for granted now, but they were just prototypes when you were born. Along the way, though, many things went wrong. In hind site, I probably shouldn’t have started with the interactive toilet. That was my first project that was hacked. I felt horrible when those old ladies were sucked in by the undertow created by reversing the flow of water and significantly increasing the power of the vacuum. That being said, the video, which you can find by a quick Ask.com search, (ha, ha…you guys probably don’t use .com, huh?) launched my career. Can't wait to see Shawn's comments. :-)

As with most ICM assignments, the basic functions are relatively easy to understand. The challenge is applying them to a more complex project for the homework assignments. I battle with this each week, trying to find an interesting way to implement some of the new things I've learned. This week I was obsessed with watching CSI: Miami marathons, and so I decided to use this project to pay homage to David Caruso. I created a little program that lets you say any of the pithy lines Horatio Caine is famous for: Once you do, I set up an audio cue when the mouse is pressed that rewards you with the audio sting that kicks off the opening credits! This wasn't easy. I spent a ton of time trying to figure out how to crop the video, and I was never actually able to get it how I wanted. My goal was to crop, rotate it and feather the edges so that it better simulated his mouth, but I just couldn't find the resources online to make that happen. I did use the blend feature to try and get the tone of the video to better resemble his skin. Not sure if it worked or not. Finally, I couldn't integrate the mirror code with the crop code, which would have made it easier for users to line their mouths up with the camera. Here's the code: import processing.video.*; // get library PImage img; // delcare image Capture video; // get video Movie sound; void setup() { size (265,238); img = loadImage("david.jpg"); // load image of David Caruso image (img,0,0,265,238); // set image size sound = new Movie (this, "sound.wav"); video = new Capture(this,350,350,30); // capture video video.crop(115,151,70,15); // crop size } void captureEvent(Capture video){ video.read(); } void draw(){ if (video.available()){ video.read(); } image(video,115,151,70,15); // view cropped video position blend (img,120,156,75,20,115,151,70,15,ADD); // Blend image and video colors } void mousePressed(){ sound.stop(); sound.play(); }

I should have posted this link last week, but I had some trouble with the code and out of frustration, decided to keep it private. This was the XML assignment for ICM. My original idea was to pull the XML feed from the blog Overheard in New York and project it over an image of New York I found and edited from Flickr. However, the blog didn't update enough to make this very compelling. I met with Sean and he suggested Twitter, but I thought it might be more interesting to pull from FriendFeed since it has even more diverse sources. I used a comic font I found because I imagined these conversation snippets coming up like comic book speech bubbles. Here's what it looked like: and I really had the hardest time wrapping the text so that it would remain on the page. In fact, it was something that I was never totally able to solve. I'm not able to post the applet on the NYU server for obvious reasons without the proxy, so I'll include the code here and if there's time, perhaps I'll return to it and post in its entirety. Code: // Aaron Uhrmacher // XML Assignment import processing.xml.*; // import library XMLElement[] links; Word[] words; String var; int statsNum = 5; void setup() { size(800, 472); PImage b; b = loadImage("nyc.jpg"); // import image image(b, 0, 0); String url = "http://friendfeed.com/?auth=oLnvsSZuMNC5ksRd&format=atom"; // import feed XMLElement rss = new XMLElement(this, url); words = new Word[20]; // new object array XMLElement[] links = rss.getChildren("entry/title"); // parse feed for (int i = 0; i < statsNum; i++) { String var = links[i].getContent(); // status text variable println(var); words[i] = new Word(var); //     //instantiate all the status objects } } void draw() { PImage b; b = loadImage("nyc.jpg"); // background image(b, 0, 0); for (int i = 0; i < statsNum; i++) { words[i].display(mouseX, mouseY+i*30, 800,400); // location of XML data moves } } class Word { PFont font; String statusText; float x; float y; float w; float h; Word(String tStatusText) { statusText = tStatusText; font = loadFont("ComicPro.vlw"); // font textFont(font, 10); } void display(float tx, float ty, float tw, float th) { x = tx; y = ty; w = tw; h = th; // The font must be located in the sketch's // "data" directory to load successfully text(statusText,x,y,400,200); fill(0); } }

Finally, things are starting to make sense. I wasn't sure if that was going to happen, or when it might happen, but it did. This week I worked with Li on a couple of our projects and having someone to go over and review each step with certainly was helpful. Temp Converter This was pretty simple after the reading. I wrote out my version of the program and then checked it against the example on the website, only to see that there was a discrepency. I posted a comment for Dan Shiffman and he corrected it. Bouncing Balls We learned how to use gravity on two bouncing balls. Once I figured out how to do it, I started playing around with colors and such just to make it a bit more interesting. I noticed that if I set the speed of the bounce to -0.99, it dramatically shortened how long the balls bounced (as well as the height of each bounce). This had a bit of math to it, but once I figured out how to make them, it was fun playing with them and trying to manipulate the action. Robot This is the one I spent the most time on this week. Li and I worked on it for a few hours. We couldn't figure out how to make stars into a class, so we ended up reading about how to import an image instead. Then, once we did that, we tried to create a flickering effect on the some of the stars by adding an ellipse with some transparency over the brightest ones and then setting the parameters so it would grow to a certain size and then shrink again. You can few the final result and the code here. Anyway, I felt a lot better after this week. My understanding is improving and I'm overall starting to grasp the key concepts!

This week was much harder than last week for a lot of reasons. In particular, I had trouble understanding how to write the code using the mousePressed variable, but I figured that out with Li's help. It was frustrating trying to read the book and then discern the best way to write the code. What I learned in particular was that there's more than one way to skin a cat, and you have to be open to trying a whole bunch of things until one works. Four Quadrants The first assignment was an individual one, and this is the result of about seven hours of reading, referencing, frantic IMs to resident researchers, etc. Ball Click It sounds easy, right? You click the mouse and the ball starts moving across the screen. When it gets to the end, it starts over. I still think it sounds easy. It wasn't. Check it out. Ball Moving Around Randomly I'm so glad I had Li to work on this one with me, because he understands all this stuff a lot better than I do. It was useful to be able to watch over his shoulder to look for coding errors but also to be able to ask questions as we wrote the program. Here it is. Paint Program This is my favorite for a ton of reasons. One, I can't believe that after only using Processing for one week, we were able to create a painting program. Holy Crap! Our program has a couple of basic commands. The screen starts out white, and as you click and drag the mouse you're able to paint in an array of blues. To erase, all you have to do is hold down any of the keys on the keyboard and drag the mouse. Finally, you can erase the entire canvas and start over by pushing 'E' or 'e' to erase. Try it out and share your feedback!

Wow, this was a lot tougher than I thought, particularly because I wasn't able to find the solutions to the exercises yet on Dan Schiffman's site. The part that gave me the most trouble (and which I still don't really understand) is the third chapter on mouse interaction. Anyway, here's what I did. I took my original drawing from class: and recreated it in Processing. Then, just for fun, I played around and created this. It ain't beautiful, but it's a start!