Kinect Design System

Introduction

In this phase of the project, we aimed at creating a design system that incorporates the act of punching with the act of 3D mesh modeling. In that, the user would then be able to export the mesh to have it fabricated through some means of 3D printing. Skeleton data from an Xbox Kinect was used to track hand positions in a 3D space and measure when, where, and to what degree a fist intersected a mesh plane. The Kinect data was captured using Firefly in Grasshopper, then transformed and stored to produce a transformation in the mesh.

The Theory

The theory behind our conceit was to take a completely immaterial process and bring some of it into the real world. This involved processing raw hand location data and augmenting that with the 3D environment in (for our case) Rhino and Grasshopper. Though our original idea took from the idea of punching and "denting" a virtual object, the outcome of our project focused more on shaping the mesh simply using your hand's, which turned out to be a much less aggressive process.

Below: Motion Tracking Portion of the Grasshopper Patch

Below: Point Cloud Calculation Portion of the Grasshopper Patch

### Python Code For Retrieving Left and Right Hand Positions ###


import rhinoscriptsyntax as rs
import scriptcontext as sc

if refresh:
    leftHand = getLeftHand()
    rightHand = getRightHand()

def getRightHand():
    hand = input[11]
    return hand
    
def getLeftHand():
    hand = input[7]
    return hand

Click to Expand

### Python Code For Recording Deformations in the Point Cloud ###


import rhinoscriptsyntax as rs
import scriptcontext as sc

def accumulate_displacement( accum, punch ):
    """Given two lists of Point3D objects, return a list with the minimal Y displacement."""
    return [ a if a.Y < p.Y else p for a,p in zip(accum, punch) ]
    
if clear or sc.sticky.get('pointShift') is None:
    accum = pointCloud
else:
    accum = sc.sticky['pointShift']
    

def compare_pts(p1,p2):
    """Sort function for x,z points.  Returns -1, 0, or 1."""
    if p1.X < p2.X:
        return -1
    elif p1.X > p2.X:
        return 1
    elif p1.Z < p2.Z:
        return -1
    elif p1.Z > p2.Z:
        return 1
    else:
        return 0
        
pointCloud.sort(compare_pts)
deformed = accumulate_displacement( accum, pointCloud )
sc.sticky['pointShift'] = deformed

Click to Expand

The Result

When creating our final artifact, we ran into a number of issues relating more to the chosen medium. We decided to attempt to use our system to create an ice sculpture that would produce caustic effects when a light was shown through it. A number of issues prevented us from completing this goal. First, the foam we chose for our ice molds turned out to not be water-tight. This led us to pursue a gesso treatment of the foam to keep water from leaking through. In the end, the Gesso ended up dissolving into the water when added to the mold, leading to fogged ice. Second, the lack of a powerful enough freezer prevented us from freezing water in one of our larger molds. The combination of the amount of water in the large mold as well as a questionable freezer led to the failure to produce an ice sculpture.

As far as the creation of the molds go, however, we were definitely able to take kinect data, deform a point cloud, bake the mesh, and CNC the mesh into a mold.

Below: The Mesh That was Created and Routed Into a Mold in Foam

Moving Forward

Similar to the previous phase of this project, it became clear that we need to incorporate some form of physical feedback to the user. Providing the user with a physical boundary helps improve the user's mental model. On the other hand, this would also limit the scope and possibilities for a user, which may or may not be a good thing. Limiting our scope will allow us to create an easier workflow from beginning with a flat mesh to a fabricated object, however, in the spirit of creating a versatile design system, choices for the user are an important consideration.

We attempted to create a form of user feedback by projecting onto a stretchy fabric that the user would then punch, depress, or pull. Limitations to the the kinect's mesh reconstruction in firefly prevented us from using the cloth as a physical medium and we had to resort back to using the kinect's skeleton tracking. Another attempt to use the kinect's mesh reconstruction should be made, however, this will require quite a bit of data processing and a much faster computing environment to make it usable in realtime.

Improvement

Our system could greatly improve from better sensing of the dexterous skills of the user. Currently with our system, you can only push a mesh in. Adding other features such as mesh grabbing (closing your hands to either push/pull the mesh) or adding brush hardness/size could really increase the diversity of the artifacts created. One consistent feature of all of the artifacts we created was that each one was very abstract. To diversify the system,

Start the Conversation

Share this Project

Found In

HMV-S15:Project Two: Digital-Physical Transformations

Courses

16-455 Human-Machine Virtuosity

· 0 members
Human dexterous skill embodies a wealth of physical understanding which complements computer-based design and machine fabrication. This project-oriented course explores the duality between hand and...more

About

In this phase of the project, we aimed at creating a design system that allows you to shape a mesh through whole body interactions. By taking skeleton data from an Xbox Kinect, we were able transform body movements into mesh shaping movements in Rhino through Grasshopper.

We focused on improving the user interaction with the system over our previous iteration by incorporating various suggestions we received from users