Outcome

Introduction

In this phase of the project, we aimed at creating a design system that incorporates the act of punching with the act of 3D mesh modeling. In that, the user would then be able to export the mesh to have it fabricated through some means of 3D printing. Skeleton data from an Xbox Kinect was used to track hand positions in a 3D space and measure when, where, and to what degree a fist intersected a mesh plane. The Kinect data was captured using Firefly in Grasshopper, then transformed and stored to produce a transformation in the mesh.

The Theory

The theory behind our conceit was to take a completely immaterial process and bring some of it into the real world. This involved processing raw hand location data and augmenting that with the 3D environment in (for our case) Rhino and Grasshopper. Though our original idea took from the idea of punching and "denting" a virtual object, the outcome of our project focused more on shaping the mesh simply using your hand's, which turned out to be a much less aggressive process.

Below: Motion Tracking Portion of the Grasshopper Patch

Below: Point Cloud Calculation Portion of the Grasshopper Patch

The Result

When creating our final artifact, we ran into a number of issues relating more to the chosen medium.  We decided to attempt to use our system to create an ice sculpture that would produce caustic effects when a light was shown through it.  A number of issues prevented us from completing this goal.  First, the foam we chose for our ice molds turned out to not be water-tight.  This led us to pursue a gesso treatment of the foam to keep water from leaking through.  In the end, the Gesso ended up dissolving into the water when added to the mold, leading to fogged ice.  Second, the lack of a powerful enough freezer prevented us from freezing water in one of our larger molds.  The combination of the amount of water in the large mold as well as a questionable freezer led to the failure to produce an ice sculpture.

As far as the creation of the molds go, however, we were definitely able to take kinect data, deform a point cloud, bake the mesh, and CNC the mesh into a mold.

Below: The Mesh That was Created and Routed Into a Mold in Foam

Moving Forward

Similar to the previous phase of this project, it became clear that we need to incorporate some form of physical feedback to the user. Providing the user with a physical boundary helps improve the user's mental model. On the other hand, this would also limit the scope and possibilities for a user, which may or may not be a good thing. Limiting our scope will allow us to create an easier workflow from beginning with a flat mesh to a fabricated object, however, in the spirit of creating a versatile design system, choices for the user are an important consideration.

We attempted to create a form of user feedback by projecting onto a stretchy fabric that the user would then punch, depress, or pull.  Limitations to the the kinect's mesh reconstruction in firefly prevented us from using the cloth as a physical medium and we had to resort back to using the kinect's skeleton tracking.  Another attempt to use the kinect's mesh reconstruction should be made, however, this will require quite a bit of data processing and a much faster computing environment to make it usable in realtime.

Improvement

Our system could greatly improve from better sensing of the dexterous skills of the user.  Currently with our system, you can only push a mesh in.  Adding other features such as mesh grabbing (closing your hands to either push/pull the mesh) or adding brush hardness/size could really increase the diversity of the artifacts created.  One consistent feature of all of the artifacts we created was that each one was very abstract.  To diversify the system,