Outcome

Move up Delete Move down

Talk to the MeBot about a movie or TV show you like, and it can tell you if the film was filmed in San Francisco! Or ask the MeBot about the movies that I like :).

The MeBot is chat bot with which you can text to learn more about me and get movie fun facts about San Francisco – all by using San Francisco's open source API.

Previously practiced as a planner, this chat bot project is a fun marriage of my interest in civic engagement and interaction design. While this MeBot take on a more casual topic of city culture. It lays the ground work for more professional uses such as looking up ongoing development projects in cities or learning about the public space maintenance.

Youtube link to video (in case it doesn't show up below: https://youtu.be/wou1skeBnhQ)

Move up Delete Move down

Show Advanced Options

Move up Delete Move down

Show Advanced Options

Move up Delete Move down

Choosing Databases

When I started this project I knew I wanted to use government data – it's what I use a lot previously as a planner for technical purposes. I looked into San Francisco's data base and found a lot of them with open API! I eventually narrowed down to two very different types of data that I really liked, weighed the pros and cons, and chose the filming location data that may attract larger audience.

Move up Delete Move down

01 choosing database.thumb

Show Advanced Options

Move up Delete Move down

Designing Conversation Structure

In choosing the database I actually sketched the conversation structure for both – it definitely helped me see how one is more challenging than the other. The final structure as shown below was very similar to the one I had when I started coding, but the coding process definitely also help clarify the routes.

Move up Delete Move down

02 design conversation structure.thumb

Show Advanced Options

Move up Delete Move down

Composing Speech with Available Data

The Film Location In San Francisco database (https://data.sfgov.org/Culture-and-Recreation/Film-Locations-in-San-Francisco/yitu-d5am) has a lot of information – from title, release, locations, fun facts, distributors, directors, writers, and actors (actor_1, actor_2, and actor_3). I started with wanting to cram in everything but quickly realized that it would make the speech bubble too long.

I decided to only use 4 data points for my chatbot and quickly realized that I have to make quite a few pieces of speech to accommodate different data quantity and make the bot sound natural.

Move up Delete Move down

03 composing speech with available data.thumb

Show Advanced Options

Move up Delete Move down

Constructing Code in Local Environment

DataSF's API is powered by SODA (https://github.com/socrata/soda-ruby), which is a gem that let you access the data simply with a weblink to the JSON file. The output of the data is stored as Hashie (https://github.com/intridea/hashie), which is a growing collection of tools that extend Hashes.

I spent the longest time trying to figure out how to get data from hashie. It was especially tricky because hashie doesn't store empty value, so the location of each item can be different depending on how complete each data point is.

For instance, this is the output for the movie "A Smile Like Yours":

Move up Delete Move down

Screen shot 2018 09 22 at 7.53.40 pm.thumb

Show Advanced Options

Move up Delete Move down

You'll see that it skipped the field actor_3 and fun_facts because there was no value for those fields.

Eventually I figured out a way to simply add each unique value to the another array. (note: I'd add locations.uniq to delete duplicated values, but I know this dataset doesn't have any.)

Move up Delete Move down

locations = []
fun_facts = []
 

output.each do |row|
      if row["locations"]
            locations.push(row["locations"])
      end
      if row["fun_facts"].to_s != ""
            fun_facts.push(row["fun_facts"])                    
      end 
end

Show Advanced Options

Move up Delete Move down

I coded everything locally and tested it in the local HTML as well as terminals (puts "#"*90 FTW!)

Move up Delete Move down

get "/apitest" do
    
    film = params[:film]
    output = client.get("https://data.sfgov.org/resource/wwmu-gmzc.json", {"$where" => "title like '#{film}%'"})
    
    
    # ---------- if the response does not match with any film title, ask to try again ----------
    if output == []
        message = "I don't recall and film or TV series that match what you said. Maybe try again? (Note: case and space sensitive)"
   
        
    # ---------- if the response match with any film, move forward ----------
    else
        titles = []
        release_year = output.first["release_year"]
        locations = []
        fun_facts = []
        director = output.first["director"]
        writer = output.first["writer"]
        actors_1 = output.first["actor_1"].to_s + output.first["actor_2"].to_s + output.first["actor_3"].to_s


        output.each do |row|
            if row["title"]
                titles.push(row["title"])
            end  
        end

        titles = titles.uniq

        # ---------- if the response matches with more then 2 film titles ----------
        if titles.length >=2

            message = "There are actually quite a few. It'd really help if you can be more specific! For example: 'Star Trek IV' or 'Change Seasion 1'."

        # ---------- if the response matches with 2 film titles, ask ----------    
        elsif titles.length == 2
            message = "Do you mean '" + titles[0].to_s + "' or '" + titles[1].to_s + "'?"

        # ---------- if the response matches with 1 film title, produce answer ----------    
        elsif titles.length == 1

            output.each do |row|
                if row["locations"]
                    locations.push(row["locations"])
                end
                if row["fun_facts"].to_s != ""
                    fun_facts.push(row["fun_facts"])                    
                end    
            end

            # ---------- edit message about location ----------

            locations = locations.uniq
            
            if locations.length >=3
                location_list = locations.sample(2)
                location_1 = location_list[0].to_s
                location_2 = location_list[1].to_s
                message_location = output.length.to_s + " locations, including " + location_1 + ", " + location_2 + ", and more places in SF."
            elsif locations.length == 2
                location_list = locations.sample(2)
                location_1 = location_list[0].to_s
                location_2 = location_list[1].to_s
                message_location = output.length.to_s + " locations, including " + location_1 + " and " + location_2 + " in SF."
            elsif locations.length == 1
                location_1 = locations[0].to_s
                message_location = "one location – " + location_1 + " in SF."
            elsif locations.nil?
                message_location = "one location in SF." 
            end
            
            # ---------- edit message about fun fact ----------
            
            #fun_facts = fun_facts.uniq
            
            if fun_facts == []
                message_fun_facts = ""
            else
                message_fun_facts = " Also fun fact: " + fun_facts.sample.to_s
            end
            
            
            # ---------- print answer about the movie ----------
            message = "Bingo! " + titles.to_s + " was released in " + release_year + " and featured " + message_location + message_fun_facts
            
            
            puts "#"*90
            shuffle = IO.readlines("angela_film_pick.txt").sample.to_s.chomp
            puts determine_message shuffle
            puts determine_congrats shuffle
            puts "#"*90
            
            
        end

    end
    
    
end

Show Advanced Options

Move up Delete Move down

While I was able to work around most response variations, there remain quite a few issues that I didn't have time to get to. They are listed below:

Move up Delete Move down

04 composing speech with available data.thumb

Show Advanced Options

Move up Delete Move down

Adjusting Conversation in Context

Once I was confident with the code I pushed it to Heroku and started to test it on my phone. It allowed me to catch a few bugs and awkward dialogue flow.

Move up Delete Move down

Cleaning Up the Code

My code would be working if I just copy-and-paste the local code. However, I wanted to consolidate them into methods so my code can be cleaner. Here are my final two method.

Move up Delete Move down

def determine_message x
    
    client = SODA::Client.new({:domain => 'explore.data.gov', :app_token => '1ItWOJ1oGYyvN9Ajo5UvHCSaa'})
    output = client.get("https://data.sfgov.org/resource/wwmu-gmzc.json", {"$where" => "title like '#{x}%'"})
    
    
    # ---------- if the response does not match with any film title, ask to try again ----------
    if output == []
        "I don't recall and film or TV series that match what you said. Maybe try again? (Note: case and space sensitive)"
   
        
    # ---------- if the response match with any film, move forward ----------
    else
        titles = []
        release_year = output.first["release_year"]
        locations = []
        fun_facts = []
        director = output.first["director"]
        writer = output.first["writer"]
        actors_1 = output.first["actor_1"].to_s + output.first["actor_2"].to_s + output.first["actor_3"].to_s


        output.each do |row|
            if row["title"]
                titles.push(row["title"])
            end  
        end

        titles = titles.uniq

        # ---------- if the response matches with more then 2 film titles ----------
        if titles.length >=2

            "There are actually quite a few. It'd really help if you can be more specific! For example: 'Star Trek IV' or 'Change Seasion 1'."

        # ---------- if the response matches with 2 film titles, ask ----------    
        elsif titles.length == 2
            "Do you mean '" + titles[0].to_s + "' or '" + titles[1].to_s + "'?"

        # ---------- if the response matches with 1 film title, produce answer ----------    
        elsif titles.length == 1

            output.each do |row|
                if row["locations"]
                    locations.push(row["locations"])
                end
                if row["fun_facts"].to_s != ""
                    fun_facts.push(row["fun_facts"])                    
                end    
            end

            # ---------- edit message about location ----------

            locations = locations.uniq
            
            if locations.length >=3
                location_list = locations.sample(2)
                location_1 = location_list[0].to_s
                location_2 = location_list[1].to_s
                message_location = output.length.to_s + " locations, including " + location_1 + ", " + location_2 + ", and more places in SF."
            elsif locations.length == 2
                location_list = locations.sample(2)
                location_1 = location_list[0].to_s
                location_2 = location_list[1].to_s
                message_location = output.length.to_s + " locations, including " + location_1 + " and " + location_2 + " in SF."
            elsif locations.length == 1
                location_1 = locations[0].to_s
                message_location = "one location – " + location_1 + " in SF."
            elsif locations.nil?
                message_location = "one location in SF." 
            end
            
            # ---------- edit message about fun fact ----------
            
            #fun_facts = fun_facts.uniq
            
            if fun_facts == []
                message_fun_facts = ""
            else
                message_fun_facts = " Also fun fact: " + fun_facts.sample.to_s
            end
            
            
            # ---------- print answer about the movie ----------
            titles[0].to_s + " was released in " + release_year + " and featured " + message_location + message_fun_facts
            
            
        end

    end
    
    
end

Show Advanced Options

Move up Delete Move down

def determine_congrats x
    
    client = SODA::Client.new({:domain => 'explore.data.gov', :app_token => '1ItWOJ1oGYyvN9Ajo5UvHCSaa'})
    output = client.get("https://data.sfgov.org/resource/wwmu-gmzc.json", {"$where" => "title like '#{x}%'"})
    
    if output == []
        ""
    else
        titles = []

        output.each do |row|
            if row["title"]
                titles.push(row["title"])
            end  
        end
        titles = titles.uniq
        if titles.length == 1
            "Bingo! "
        else
            ""
        end

    end
    
    
end

Show Advanced Options

Move up Delete Move down

Final Reflection

While we have been building the foundation for this chatbot in weekly exercises, it was REALLY CHALLENGING for me to finally getting it to work the way I wanted (with major room for improvement). Here are some of my final thoughts:

Things I was proud of:

Learning how to use government open data
Designing conversation pieces that together sounds natural
Solving some of the edge scenarios

Things I could do better

Incorporating natural language processes capability in my Chat Bot
Allowing for more back and forth in the conversation design

That's it :) it was a fun project and I learned a lot :)

You can upload files of up to 20MB using this form.

Add Rich Text Add Code Block Add Embeddable Content