Data mining with Ruby and Twitter
The interesting side to a Twitter API
In October 2008, like many others, I created a Twitter account out of curiosity. Like most people, I connected with friends and did some random searching to better understand the service. Communicating at 140 characters didn't seem like an idea that would be popular. An unrelated event helped me understand Twitter's real value.
In early July 2009, my web-hosting provider went dark. After random web searching, I found information pointing to a fire in Seattle's Fisher Plaza as the culprit. Information from traditional web-based sources was slow and gave no indication of when the service might return. However, after searching Twitter, I found personal accounts of the incident, including real-time information on what was happening at the scene. For example, shortly before my hosting service returned, there was a tweet indicating that diesel power generators were outside the building.
This was when I realized that the true power of Twitter is open and real-time communication of information among individuals and groups. Yet, under the surface, it is a treasure trove of information about behaviors of the users, and trends at the local and global levels. I explore this realization in the context of simple scripts using the Ruby language and the Twitter gem, an API wrapper for Twitter. I also demonstrate how to build simple mashups for data visualization using other web services and applications.
Twitter and APIs
Although the early web was about human-machine interaction, today's web is about machine-machine interaction, enabled using web services. These services exist for most popular websites—from various Google services to LinkedIn, Facebook, and Twitter. Web services create APIs through which external applications can query or manipulate content on websites.
Web services are implemented using a number of styles. Today, one of the
most popular is Representational State Transfer, or REST. One
implementation of REST is over the well-known HTTP protocol, allowing HTTP
to exist as a medium for a RESTful architecture (using standard HTTP
operations like GET, PUT, POST, and
DELETE). The API for Twitter is developed as an abstraction
over this medium. In this way, there's no knowledge of REST, HTTP, or data
formats like XML or JSON, but instead an object-based interface that
integrates cleanly into the Ruby language.
A quick tour of Ruby and Twitter
Let's explore how you can use the Twitter API with Ruby. First, we need to
get the necessary resources. If like me you're using Ubuntu Linux®,
you use the apt framework.
To get the latest full Ruby distribution (approximately a 13MB download), use this command line:
$ sudo apt-get install ruby1.9.1-full
Next, grab the Twitter gem using the gem utility:
$ sudo gem install twitter
You now have everything you need for this step, so let's continue with a test of the Twitter wrapper. For this demonstration, use a shell called the Interactive Ruby Shell (IRB). This shell allows you to execute Ruby commands and experiment with the language in real time. IRB has a large number of capabilities, but we'll use it for simple experimentation.
Listing 1 shows a session with IRB that has been
broken into three sections to aid readability. The first section (lines
001 and 002) simply prepares the environment by importing the necessary
run time elements (the require method loads and executes the
named library). The next line (003) demonstrates the use of the Twitter
gem to display the most recent tweet from IBM® developerWorks®.
As shown, you use the user_timeline method of the
Client::Timeline module to display a tweet. This first
example demonstrates the "chain methods" capability of Ruby. The
user_timeline method returns an array of 20 tweets that you
chain into the first method. Doing so extracts the first
tweet from the array (first is a method of the
Array class). From this single tweet, you extract the text
field emitted to output via puts.
The next section (line 004) uses the user-defined location field, a
free-form field into which the user can provide both useful and non-useful
location information. In this example, the User module grabs
user information, constrained with the location field.
The final section (from line 005) explores the Twitter::Search
module. The search module provides an extremely rich interface with which
to search Twitter. In this example, you first create a search instance
(line 005), then specify a search at line 006. You're searching for the
most recent tweets containing the word why that are directed to
the LulzSec user. The resulting list has been reduced and edited. Searches
are sticky in that the search instance maintains the defined filters. You
can clear these filters by executing search.clear.
Listing 1. Experimenting with the Twitter API through IRB
$ irb
irb(main):001:0> require "rubygems"
=> true
irb(main):002:0> require "twitter"
=> true
irb(main):003:0> puts Twitter.user_timeline("developerworks").first.text
dW Twitter is saving #IBM over $600K per month: will #Google+ add to that? >
http://t.co/HiRwir7 #Tech #webdesign #Socialmedia #webapp #app
=> nil
irb(main):004:0> puts Twitter.user("MTimJones").location
Colorado, USA
=> nil
irb(main):005:0> search = Twitter::Search.new
=> #<Twitter::Search:0xb7437e04 @oauth_token_secret=nil,
@endpoint="https://api.twitter.com/1/",
@user_agent="Twitter Ruby Gem 1.6.0",
@oauth_token=nil, @consumer_secret=nil,
@search_endpoint="https://search.twitter.com/",
@query={:tude=>[], :q=>[]}, @cache=nil, @gateway=nil, @consumer_key=nil,
@proxy=nil, @format=:json, @adapter=:net_http<
irb(main):006:0> search.containing("why").to("LulzSec").
result_type("recent").each do |r| puts r.text end
@LulzSec why not stop posting <bleep> and get a full time job! MYSQLi isn't
hacking you <bleep>.
...
irb(main):007:0>Next, let's look at the schema for a user in Twitter. You can also do this
through IRB, but I'll reformat the result to illustrate more simply the
anatomy of a Twitter user. Listing 2 shows the result
of printing the user structure, which in Ruby is a
Hashie::Mash. This structure is useful, because it permits an
object to have method-like accessors for hash keys (an open object). As
you can see from Listing 2, this object contains a
wealth of information (user-specific and rendering information), including
current user status (with geocode information). A tweet also contains a
large amount of information, and you can easily visualize generating this
information using the user_timeline class.
Listing 2. Anatomy of a Twitter user (Ruby perspective)
irb(main):007:0> puts Twitter.user("MTimJones")
<#Hashie::Mash
contributors_enabled=false
created_at="Wed Oct 08 20:40:53 +0000 2008"
default_profile=false default_profile_image=false
description="Platform Architect and author (Linux, Embedded, Networking, AI)."
favourites_count=1
follow_request_sent=nil
followers_count=148
following=nil
friends_count=96
geo_enabled=true
id=16655901 id_str="16655901"
is_translator=false
lang="en"
listed_count=10
location="Colorado, USA"
name="M. Tim Jones"
notifications=nil
profile_background_color="1A1B1F"
profile_background_image_url="..."
profile_background_image_url_https="..."
profile_background_tile=false
profile_image_url="http://a0.twimg.com/profile_images/851508584/bio_mtjones_normal.JPG"
profile_image_url_https="..."
profile_link_color="2FC2EF"
profile_sidebar_border_color="181A1E" profile_sidebar_fill_color="252429"
profile_text_color="666666"
profile_use_background_image=true
protected=false
screen_name="MTimJones"
show_all_inline_media=false
status=<#Hashie::Mash
contributors=nil coordinates=nil
created_at="Sat Jul 02 02:03:24 +0000 2011"
favorited=false
geo=nil
id=86978247602094080 id_str="86978247602094080"
in_reply_to_screen_name="AnonymousIRC"
in_reply_to_status_id=nil in_reply_to_status_id_str=nil
in_reply_to_user_id=225663702 in_reply_to_user_id_str="225663702"
place=<#Hashie::Mash
attributes=<#Hashie::Mash>
bounding_box=<#Hashie::Mash
coordinates=[[[-105.178387, 40.12596],
[-105.034397, 40.12596],
[-105.034397, 40.203495],
[-105.178387, 40.203495]]]
type="Polygon"
>
country="United States" country_code="US"
full_name="Longmont, CO"
id="2736a5db074e8201"
name="Longmont" place_type="city"
url="http://api.twitter.com/1/geo/id/2736a5db074e8201.json"
>
retweet_count=0
retweeted=false
source="web"
text="@AnonymousIRC @anonymouSabu @LulzSec @atopiary @Anonakomis Practical reading
for future reference... LULZ \"Prison 101\" http://t.co/sf8jIH9" truncated=false
>
statuses_count=79
time_zone="Mountain Time (US & Canada)"
url="http://www.mtjones.com"
utc_offset=-25200
verified=false
>
=> nil
irb(main):008:0>That's it for the quick tour. Now, let's explore some simple scripts that you can use to collect and visualize data using Ruby and the Twitter API. Along the way, you'll get to know some of the concepts of Twitter, such as authentication and rate limiting.
Mining Twitter data
The following sections present several scripts for collecting and presenting data available through the Twitter API. These scripts focus on simplicity, but you can extend and combine them to create new capabilities. Further, this section touches the surface of the Twitter gem API, where many more capabilities are available.
It's important to note that the Twitter API only allows clients to make a limited number of calls in a given hour, that is, Twitter rate-limits requests (currently no more than 150 per hour), which means that after some amount of use, you'll get an error message and be required to wait before submitting new requests.
User information
Recall from Listing 2 that a large amount of information is available about each Twitter user. This information is only accessible if the user isn't protected. Let's look at how you can extract a user's data and present it in a more convenient way.
Listing 3 presents a simple Ruby script to retrieve a
user's information (based on his or her screen name), and then emit some
of the more useful elements. You use the to_s Ruby method to
convert the value to a string as needed. Note that you first ensure that
the user isn't protected; otherwise, this data wouldn't be accessible.
Listing 3. Simple script to extract Twitter user data (user.rb)
#!/usr/bin/env ruby
require "rubygems"
require "twitter"
screen_name = String.new ARGV[0]
a_user = Twitter.user(screen_name)
if a_user.protected != true
puts "Username : " + a_user.screen_name.to_s
puts "Name : " + a_user.name
puts "Id : " + a_user.id_str
puts "Location : " + a_user.location
puts "User since : " + a_user.created_at.to_s
puts "Bio : " + a_user.description.to_s
puts "Followers : " + a_user.followers_count.to_s
puts "Friends : " + a_user.friends_count.to_s
puts "Listed Cnt : " + a_user.listed_count.to_s
puts "Tweet Cnt : " + a_user.statuses_count.to_s
puts "Geocoded : " + a_user.geo_enabled.to_s
puts "Language : " + a_user.lang
if (a_user.url != nil)
puts "URL : " + a_user.url.to_s
end
if (a_user.time_zone != nil)
puts "Time Zone : " + a_user.time_zone
end
puts "Verified : " + a_user.verified.to_s
puts
tweet = Twitter.user_timeline(screen_name).first
puts "Tweet time : " + tweet.created_at
puts "Tweet ID : " + tweet.id.to_s
puts "Tweet text : " + tweet.text
endTo invoke this script, ensuring that it's executable
(chmod +x user.rb), you invoke it with a user. The result is
shown in Listing 4 for the developerworks
user, showing the user information and current status (last tweet
information). Note here that Twitter defines followers as people
who follow you; but people that you follow are called
friends.
Listing 4. Sample output from user.rb
$ ./user.rb developerworks Username : developerworks Name : developerworks Id : 16362921 Location : User since : Fri Sep 19 13:10:39 +0000 2008 Bio : IBM's premier Web site for Java, Android, Linux, Open Source, PHP, Social, Cloud Computing, Google, jQuery, and Web developer educational resources Followers : 48439 Friends : 46299 Listed Cnt : 3801 Tweet Cnt : 9831 Geocoded : false Language : en URL : http://bit.ly/EQ7te Time Zone : Pacific Time (US & Canada) Verified : false Tweet time : Sun Jul 17 01:04:46 +0000 2011 Tweet ID : 92399309022167040 Tweet text : dW Twitter is saving #IBM over $600K per month: will #Google+ add to that? > http://t.co/HiRwir7 #Tech #webdesign #Socialmedia #webapp #app
Friends popularity
Look at your friends (people you follow), and gather data to understand their popularity. In this case, you gather your friends and sort them in the order of their followers count. This simple script is shown in Listing 5.
In this script, after you understand the user you want to analyze (based on their screen name), you create a user hash. A Ruby hash (or associative array) is a data structure that allows you to define the key for storage (instead of a simple numerical index). Your hash is then indexed by Twitter screen name, and the associated value is the user's follower count. The process is simply to iterate your friends and hash their followers count. Sort your hash (in descending order), and emit it as output.
Listing 5. Friend's popularity script (friends.rb)
#!/usr/bin/env ruby
require "rubygems"
require "twitter"
require 'google_chart'
name = String.new ARGV[0]
user = Hash.new
# Iterate friends, hash their followers
friends = Twitter.friend_ids(name)
friends.ids.each do |fid|
f = Twitter.user(fid)
# Only iterate if we can see their followers
if (f.protected.to_s != "true")
user[f.screen_name.to_s] = f.followers_count
end
end
user.sort_by {|k,v| -v}.each { |user, count| puts "#{user}, #{count}" }Sample output from the friends script in Listing 5 is shown in Listing 6. I've clipped the output to conserve space, but as you can see, ReadWriteWeb (RWW) and Playstation are popular Twitter users in my direct network.
Listing 6. Screen output from the friends script in Listing 5
$ ./friends.rb MTimJones RWW, 1096862 PlayStation, 1026634 HarvardBiz, 541139 tedtalks, 526886 lifehacker, 146162 wandfc, 121683 AnonymousIRC, 117896 iTunesPodcasts, 82581 adultswim, 76188 forrester, 72945 googleresearch, 66318 Gartner_inc, 57468 developerworks, 48518
Where are my followers?
Recall from Listing 2 that Twitter provides a wealth of location information. There's a location field that is free form, user defined, and optional geocoding data. However, a user-defined time zone can also provide a hint as to the follower's actual location.
In this example, you build a mash-up that extracts time zone data from your Twitter followers, and then visualize this data using Google Charts. Google Charts is an interesting project that allows you to build a variety of different chart types over the web; defining the chart type and data as an HTTP request, where the result is rendered directly in the browser as the response. To install the Ruby gem for Google Charts, use the following command line:
$ gem install gchartrb
Listing 7 provides the script for extracting time zone data, then building the Google Charts request. First, unlike previous scripts, this script requires that you be authenticated with Twitter. To do this, you need to register an application with Twitter, which provides you with a set of keys and tokens. Those tokens can be applied to the script in Listing 7 to successfully extract the data. See Related topics for details on this easy process.
Following a similar pattern, this script accepts a screen name, and then
iterates the followers of that user. The time zone is extracted for the
current follower and stored in the tweetlocation hash. Note,
you first test whether this key is in the hash and, if so, increment the
counter for that key. You also keep a tab on the number of total time
zones for the later construction of percentages.
The last portion of the script is the construction of the Google Pie Chart
URL. You create a new PieChart and specify some options
(size, title, and whether it's 3D). Then, you iterate your time zone hash,
emitting data for the chart for the time zone string (removing the
& symbol) and the percentage of the time zone from the
total.
Listing 7. Building a pie chart from Twitter followers' time zones (followers-location.rb)
#!/usr/bin/env ruby
require "rubygems"
require "twitter"
require 'google_chart'
screen_name = String.new ARGV[0]
tweetlocation = Hash.new
timezones = 0.0
# Authenticate
Twitter.configure do |config|
config.consumer_key = '<consumer_key>'
config.consumer_secret = '<consumer_secret>'
config.oauth_token = '<oath_token>'
config.oauth_token_secret = '<oath_token_secret>'
end
cursor = "-1"
# Loop through all pages
while cursor != 0 do
# Iterate followers, hash their location
followers = Twitter.follower_ids(screen_name, :cursor=>cursor)
followers.ids.each do |fid|
f = Twitter.user(fid)
loc = f.time_zone.to_s
if (loc.length > 0)
if tweetlocation.has_key?(loc)
tweetlocation[loc] = tweetlocation[loc] + 1
else
tweetlocation[loc] = 1
end
timezones = timezones + 1.0
end
end
cursor = followers.next_cursor
end
# Create a pie chart
GoogleChart::PieChart.new('650x350', "Time Zones", false ) do |pc|
tweetlocation.each do |loc,count|
pc.data loc.to_s.delete("&"), (count/timezones*100).round
end
puts pc.to_url
endTo execute the script from Listing 7, provide it with a Twitter screen name, and then copy and paste the resulting URL into a browser. Listing 8 shows this process with the resulting generated URL.
Listing 8. Invoking the followers-location script (result is a single line)
$ ./followers-location.rb MTimJones http://chart.apis.google.com/chart?chl=Seoul|Santiago|Paris|Mountain+Time+(US++Canada)| Madrid|Central+Time+(US++Canada)|Warsaw|Kolkata|London|Pacific+Time+(US++Canada)| New+Delhi|Pretoria|Quito|Dublin|Moscow|Istanbul|Taipei|Casablanca|Hawaii|Mumbai| International+Date+Line+West|Tokyo|Ulaan+Bataar|Vienna|Osaka|Alaska|Chennai|Bern| Brasilia|Eastern+Time+(US++Canada)|Rome|Perth|La+Paz &chs=650x350&chtt=Time+Zones&chd=s:KDDyKcKDOcKDKDDDDDKDDKDDDDOKK9DDD&cht=p $
When you paste the URL from Listing 8 into a browser, you get the result shown in Figure 1.
Figure 1. Pie chart of Twitter followers' locations
Twitter user behavior
Twitter contains a large amount of data that you can mine to understand some elements of user behavior. Two simple examples are to analyze when a Twitter user tweets and from what application the user tweets. You can use the following two simple scripts to extract and visualize this information.
Listing 9 presents a script that iterates the tweets
from a particular user (using the user_timeline method), and
then for each tweet, extracts the particular day on which the tweet
originated. You use a simple hash again to accumulate your weekday counts,
then generate a bar chart using Google Charts in a similar fashion to the
previous time zone example. Note also the use of default for the hash,
which specifies the value to return for undefined hashes.
Listing 9. Building a bar chart of tweet days (tweet-days.rb)
#!/usr/bin/env ruby
require "rubygems"
require "twitter"
require "google_chart"
screen_name = String.new ARGV[0]
dayhash = Hash.new
# Initialize to avoid a nil error with GoogleCharts (undefined is zero)
dayhash.default = 0
timeline = Twitter.user_timeline(screen_name, :count => 200 )
timeline.each do |t|
tweetday = t.created_at.to_s[0..2]
if dayhash.has_key?(tweetday)
dayhash[tweetday] = dayhash[tweetday] + 1
else
dayhash[tweetday] = 1
end
end
GoogleChart::BarChart.new('300x200', screen_name, :vertical, false) do |bc|
bc.data "Sunday", [dayhash["Sun"]], '00000f'
bc.data "Monday", [dayhash["Mon"]], '0000ff'
bc.data "Tuesday", [dayhash["Tue"]], '00ff00'
bc.data "Wednesday", [dayhash["Wed"]], '00ffff'
bc.data "Thursday", [dayhash["Thu"]], 'ff0000'
bc.data "Friday", [dayhash["Fri"]], 'ff00ff'
bc.data "Saturday", [dayhash["Sat"]], 'ffff00'
puts bc.to_url
endFigure 2 provides the result of the execution of the tweet-days script in Listing 9 for the developerWorks account. As shown, Wednesday tends to be the most active tweet day, with Saturday and Sunday the least active.
Figure 2. Relative bar chart of per-day tweet activity
The next script determines from which source a particular user tweets. There are several ways you can tweet, and this script doesn't encode them all. As shown in Listing 10, you use a similar pattern to extract the user timeline for a given user, and then attempt to decode the source of the tweet in a hash. You use the hash later to create a simple pie chart using Google Charts to visualize the data.
Listing 10. Building a pie chart of a user's tweet sources (tweet-source.rb)
#!/usr/bin/env ruby
require "rubygems"
require "twitter"
require 'google_chart'
screen_name = String.new ARGV[0]
tweetsource = Hash.new
timeline = Twitter.user_timeline(screen_name, :count => 200 )
timeline.each do |t|
if (t.source.rindex('blackberry')) then
src = 'Blackberry'
elsif (t.source.rindex('snaptu')) then
src = 'Snaptu'
elsif (t.source.rindex('tweetmeme')) then
src = 'Tweetmeme'
elsif (t.source.rindex('android')) then
src = 'Android'
elsif (t.source.rindex('LinkedIn')) then
src = 'LinkedIn'
elsif (t.source.rindex('twitterfeed')) then
src = 'Twitterfeed'
elsif (t.source.rindex('twitter.com')) then
src = 'Twitter.com'
else
src = t.source
end
if tweetsource.has_key?(src)
tweetsource[src] = tweetsource[src] + 1
else
tweetsource[src] = 1
end
end
GoogleChart::PieChart.new('320x200', "Tweet Source", false) do |pc|
tweetsource.each do|source,count|
pc.data source.to_s, count
end
puts "\nPie Chart"
puts pc.to_url
endFigure 3 provides a visualization of a user on Twitter who has an interesting set of tweet sources. The traditional Twitter website is used most often, along with a mobile phone application next.
Figure 3. Pie chart of a Twitter user's tweet sources
Followers graph
Twitter is a massive network of users that forms a graph. As you've seen from the scripts, it's easy to iterate your contacts, and then iterate their contacts. Doing so forms the basis for a large graph, even at this level.
To visualize a graph, I've chosen to use the graph visualization software GraphViz. On Ubuntu, you can easily install this tool using the following command line:
$ sudo apt-get install graphviz
The script shown in Listing 11 iterates a user's followers, and then iterates their followers. The only real difference in this pattern is the construction of a GraphViz dot-formatted file. GraphViz uses a simple script format to define graphs, which you'll emit as part of your enumeration of the Twitter users. As shown, you define a graph simply by specifying the relationships of the nodes.
Listing 11. Visualizing a Twitter followers graph (followers-graph.rb)
#!/usr/bin/env ruby
require "rubygems"
require "twitter"
require 'google_chart'
screen_name = String.new ARGV[0]
tweetlocation = Hash.new
# Authenticate
Twitter.configure do |config|
config.consumer_key = '<consumer_key>'
config.consumer_secret = '<consumer_secret>'
config.oauth_token = '<oath_token>'
config.oauth_token_secret = '<oath_token_secret>'
end
my_file = File.new("graph.dot", "w")
my_file.puts "graph followers {"
my_file.puts " node [ fontname=Arial, fontsize=6, penwidth=4 ];"
# Get the first page of followers
followers = Twitter.follower_ids(screen_name, :cursor=> -1 )
# Iterate the followers returned in the Array (max 10).
followers.ids[0..[5,followers.ids.length].min].each do |fid|
f = Twitter.user(fid)
# Only iterate if we can see their followers
if (f.protected.to_s != "true")
my_file.puts " \"" + screen_name + "\" -- \"" + f.screen_name.to_s + "\""
# Get the first page of their followers
followers2 = Twitter.follower_ids(f.screen_name, :cursor => -1 )
# Iterate the followers returned in the Array (max 10).
followers2.ids[0..[5,followers2.ids.length].min].each do |fid2|
f2 = Twitter.user(fid2)
my_file.puts " \"" + f.screen_name.to_s + "\" -- \"" +
f2.screen_name.to_s + "\""
end
end
end
my_file.puts "}"Execute the script from Listing 11 on a user results in a dot file that you
then generate an image from using GraphViz. First, invoke the Ruby script
to gather the graph data (stored as graph.dot); then, use GraphViz to
generate the graph image (here, using circo, which specifies
a circular layout). The process of generating this image is defined as
follows:
$ ./followers-graph.rb MTimJones $ circo graph.dot -Tpng -o graph.png
The resulting image is shown in Figure 4. Note that the
Twitter graphs tend to be large, so I've constrained the graph by
minimizing the number of users and their followers to enumerate (per the
:count option in Listing 11).
Figure 4. Sample Twitter follower graph (extreme subset)
Location information
When enabled, Twitter collects geolocation data about you and your tweets. This data consists of latitude and longitude information that can be used to pinpoint a user or from where a tweet originates. Further, searches can incorporate this information so that you can identify places or people based on a defined location or your location.
Not all users or tweets are geo-enabled (for privacy reasons), but this information serves as an interesting dimension to the overall Twitter experience. Let's look at a script that allows you to visualize with geolocation data as well as another that allows you to search with this information.
In the first script (shown in Listing 12), you grab latitude and longitude data from a user (recall the bounding box from Listing 2). Although the bounding box is a polygon defining the area represented for the user, I simplify and use one point of this data. With this data, I generate a simple JavaScript function in a simple HTML file. This JavaScript code interfaces with Google Maps to present an overhead map of this location (given the latitude and longitude data extracted from the Twitter user).
Listing 12. Ruby script to construct a map of a user (where-am-i.rb)
#!/usr/bin/env ruby
require "rubygems"
require "twitter"
require 'google_chart'
Twitter.configure do |config|
config.consumer_key = '<consumer_key>'
config.consumer_secret = '<consumer_secret>'
config.oauth_token = '<oauth_token>'
config.oauth_token_secret = '<oauth_token_secret>'
end
screen_name = String.new ARGV[0]
a_user = Twitter.user(screen_name)
if a_user.geo_enabled == true
long = a_user.status.place.bounding_box.coordinates[0][0][0];
lat = a_user.status.place.bounding_box.coordinates[0][0][1];
my_file = File.new("test.html", "w")
my_file.puts "<!DOCTYPE html>"
my_file.puts "<html><head>"
my_file.puts "<meta name=\"viewport\" content=\"initial-scale=1.0, "
my_file.puts "user-scalable=no\" />"
my_file.puts "<style type=\"text/css\">"
my_file.puts "html { height: 100% }"
my_file.puts "body { height: 100%; margin: 0px; padding: 0px }"
my_file.puts "#map_canvas { height: 100% }"
my_file.puts "<style>"
my_file.puts "<script type=\"text/javascript\""
my_file.puts "src=\"http://maps.google.com/maps/api/js?sensor=false\">"
my_file.puts "<script>"
my_file.puts "<script type=\"text/javascript\">"
my_file.puts "function initialize() {"
my_file.puts "var latlng = new google.maps.LatLng(" + lat.to_s + ", " + long.to_s + ");"
my_file.puts "var myOptions = {"
my_file.puts "zoom: 12,"
my_file.puts "center: latlng,"
my_file.puts "mapTypeId: google.maps.MapTypeId.HYBRID"
my_file.puts "};"
my_file.puts "var map = new google.maps.Map(document.getElementById(\"map_canvas\"),"
my_file.puts "myOptions);"
my_file.puts "}"
my_file.puts "<script>"
my_file.puts "<head>"
my_file.puts "<body onload=\"initialize()\">"
my_file.puts "<div id=\"map_canvas\" style=\"width:100%; height:100%\"<>/div>"
my_file.puts "<body>"
my_file.puts "<html>"
else
puts "no geolocation data available."
endThe script in Listing 12 is executed simply as:
$ ./where-am-i.rb MTimJones
The resulting HTML file is rendered through a browser, such as:
$ firefox test.html
This script can fail if no location information is available; but if it succeeds, an HTML file is generated that a browser can read to render the map. Figure 5 presents the resulting map image, which shows a portion of the Front Range of northern Colorado, USA.
Figure 5. Sample image rendered from the script in Listing 12
With the geolocation, you can also search Twitter to identify Twitter users and tweets related to a particular location. The Twitter Search API allows geocoding information to restrict its results. The following example shown in Listing 13 extracts latitude and longitude data for a user, then uses this data to fetch tweets within a radius of 5 miles of that location.
Listing 13. Search for local tweets with latitude and longitude data (tweets-local.rb)
#!/usr/bin/env ruby
require "rubygems"
require "twitter"
Twitter.configure do |config|
config.consumer_key = '<consumer_key>'
config.consumer_secret = '<consumer_secret>'
config.oauth_token = '<oauth_token>'
config.oauth_token_secret = '<oauth_token_secret>'
end
screen_name = String.new ARGV[0]
a_user = Twitter.user(screen_name)
if a_user.geo_enabled == true
long = a_user.status.place.bounding_box.coordinates[0][0][0]
lat = a_user.status.place.bounding_box.coordinates[0][0][1]
Array tweets = Twitter::Search.new.geocode(lat, long, "5mi").fetch
tweets.each do |t|
puts t.from_user + " | " + t.text
end
endThe result of the script in Listing 13 is shown in Listing 14. This is a subset of the tweets given the frequency of tweeters out there.
Listing 14. Viewing local tweets within 5 miles of my location
$ ./tweets-local.rb MTimJones Breesesummer | @DaltonOls did he answer u LongmontRadMon | 60 CPM, 0.4872 uSv/h, 0.6368 uSv/h, 2 time(s) over natural radiation graelston | on every street there is a memory; a time and place we can never be again. Breesesummer | #I'minafight with @DaltonOls to see who will marry @TheCodySimpson I will marry him!!! :/ _JennieJune_ | ok I'm done, goodnight everyone! Breesesummer | @DaltonOls same _JennieJune_ | @sylquejr sleep well! Breesesummer | @DaltonOls ok let's see what he says LongmontRadMon | 90 CPM, 0.7308 uSv/h, 0.7864 uSv/h, 2 time(s) over natural radiation Breesesummer | @TheCodySimpson would u marry me or @DaltonOls natcapsolutions | RT hlovins: The scientific rebuttal to the silly Forbes release this morning: Misdiagnosis of Surface Temperatu... http://bit.ly/nRpLJl $
Going further
This article presented a number of simple scripts for extracting data from Twitter using the Ruby language. The emphasis was on the development and presentation of simple scripts to illustrate the fundamental ideas, but much more is possible. For example, you can also use the API to explore your friends networks and identify the most popular Twitter users of interest to you. Another interesting area is the mining of tweets themselves, using geolocation data to understand location-based behaviors or events (such as flu outbreaks). This article only scratched the surface, but feel free to comment below with your own mash-ups. Ruby and the Twitter gem make it simple to develop useful mash-ups or dashboards for your data-mining needs.
Downloadable resources
Related topics
- Ruby's official language website is the single source for Ruby news, information, releases, documentation, and community support for the Ruby language. Given Ruby's growing use in web frameworks (such as Ruby on Rails), you can also learn the most recent security vulnerabilities and their solutions.
- The Github social coding site provides the official source for the Twitter gem. At this site, you can get access to the source, documentation, and mailing list for the Ruby Twitter gem.
- Registering a Twitter application is necessary to use certain elements of the Twitter API. The process is free and allows you to access some of the more useful elements of the API.
- Google Maps JavaScript API tutorial shows how to use Google Maps to render maps of various types using user-provided geolocation data. The JavaScript used in this article was based on the "Hello World" example code provided within.
- developerWorks on Twitter: Follow us and follow this author at M. Tim Jones.
- The Twitter Ruby gem, developed by John Nunemaker, provides a useful interface to the Twitter service that cleanly integrates into the Ruby language.
- The Google Chart API is a useful service that provides the ability to construct complex and rich graphics using a variety of styles and options. This service provides an API through which a URL results that is rendered at the Google site.
- The Google Chart API Ruby wrapper provides a Ruby interface to the Google Charts API for the construction of useful charts within Ruby.
- Start developing with product trials, free downloads, and IBM Bluemix services.