The Setup

Data science is a buzzword these days and everyone is asking how to be a data scientist. A data scientist's job is definitely awesome, because if anyone is close to predicting the future, they are the ones who probably do it. But what does a data scientist actually do? The process of "data science" roughly consists of three things: mining, analysis and visualization. Each one of these three segments is interesting in it's own right. In data mining you get to collect data from large data sources, mostly from the Internet using clever techniques like scraping.

Many websites provide API so that other people can collect data from those sites to use in their work. For example you might want to know a Facebook page's growth over a certain period of time. Facebook made an API just so that you can get the data to do the analysis needed to answer your question. This sort of API is provided by Github and Twitter too. You can not imagine how vastly useful this public data is, they even came in handy as much as detecting earthquakes faster than the U.S. Geological Survey!

Enough talk, let's have some action! In this tutorial, I will get you started with scraping data from API using Python. We will use Github's API as data source and keep the scraped data in a database called RethinkDB. Don't worry if you have never heard of it before, it's pretty cool. If you want to try out other databases, you will just have to tweak a few lines of code. Let's set up the necessary softwares.

You will need to install a few softwares, preferably in a Linux workstation. I am going to use Ubuntu. If you are using Windows, then switch to Ubuntu or any other *nix. There is no way around it. You will need-

Virtualenv and virtualenvwrapper

You probably have already used these if you tried to solve version dependency in your code. These are handy tools that create Python virtual environments that are separate and isolated from eachother. Follow this link to read more and install them. Create and activate a virtual environment for this project:

mkvirtualenv scraping
workon scraping

RethingDB

If you are new to Ubuntu or Linux, installing RethinkDB might be a little bit complicated for you. Nothing you can't do though, just a few lines of extra commands to copy and paste. RethinkDB has really nice docs starting with installation and trying it for the first time.

TIP: RethinkDB server won't start automatically everytime you log on to your PC, so you might want to set it up for autostart.

You will also need the Python driver to connect to the database using your code. Start up the virtual environment you created earlier and install the driver:

pip install rethinkdb

Now you are worthy enough to move on to next stage :D

Fetching Data

API, or Application Programming Interface was created to make things easier for developers who want to use anything made by other people. That's the watered down version of what an API is, you can(should) Google and learn more about it. There are tons of APIs out there that you can use to access anything starting from social media posts to map location data. A site called Programmable Web keeps track of over 13,000 APIs like Google Maps, Twitter, Youtube, Flickr etc.

Are you ready to begin scraping? Definitely you are, you have set up all the tools needed. But the question is, what are you going to scrape? Remember the API I talked about last time?

Exploring Github's API

Websites like Github and Facebook have API or Application Programming Interface so that you can interact with their services and data through your programs. They usually have extensive documentation on how to do this. In Github's API documentation you will see how to send requests to Github, what you will get in reply and how you can filter the reply for your specific needs. How about you check it out? Open up a new tab in the browser and go to the address https://api.github.com/users/username with your own username in place of "username". You will see something similar to this:

{
  "login": "gvanrossum",
  "id": 2894642,
  "avatar_url": "https://avatars.githubusercontent.com/u/2894642?v=3",
  "gravatar_id": "",
  "url": "https://api.github.com/users/gvanrossum",
  "html_url": "https://github.com/gvanrossum",
  "type": "User",
  "site_admin": false,
  "name": "Guido van Rossum",
  "company": "Dropbox",
  "blog": "http://python.org/~guido/",
  "location": "San Francisco Bay Area",
  "email": "guido@python.org",
  "hireable": null,
  "bio": null,
  "public_repos": 4,
  "public_gists": 5,
  "followers": 911,
  "following": 0,
  "created_at": "2012-11-26T18:46:40Z",
  "updated_at": "2015-12-23T18:35:37Z"
}

This is what I see when I visit the address with Python's daddy Guido's username: https://api.github.com/users/gvanrossum. I left out a few lines so that it's easier to read. You will notice that we can know many things about Guido's Github account: when it was created, how many public repositories he has, even how many people he is following. This is how it works: Github takes a few parameters from you (eg. Guido's username) and looks up the related information in their database. Depending on your access permission, Github then shows you a webpage with the available info. The page is in JSON format, that is, list of data in key-value pairs. Not very unlike Python's dictionaries.

Let's start scraping!

Who and what to scrape

Now that we know what sort of data we can get, let's start coding. First of all we need to import the modules we are going to use in this project. We will also be defining a few variables:

import requests

def main():
    users = ['google','facebook','apache']
    baseurl = "https://api.github.com/users/"
    properties = ['login', 'id', 'html_url', 'public_repos', 'created_at']

You probably don't have requests module installed, so type pip install requests in your terminal for that. We can put any Github username in the users list, but for now let's keep Google, Facebook and Apache in it. baseurl is the URL to access Github's API. We will add usernames at the end of this URL to access that user's info. Finally, the properties list has a few properties of the JSON file that we are going to use.

Test run

We have three users in our users list, so we will have to loop through it. But at let's test with one user first. Add these lines to your code:

    # to grab and show info from Github
    # make API url with Github username
    current_page = baseurl + "facebook"  # "https://api.github.com/users/facebook"
    # take response from the url
    response = requests.get(current_page)
    # conversion to human readable format
    json = response.json()
    # print properties in json dictionary
    print(json["name"])
    print(json["id"])
    print(json["html_url"])


if __name__ == '__main__':
    main()

Now run the code. Did you see something like this?

Facebook
69631
https://github.com/facebook
147
2009-04-02T03:35:22Z

Now time for line by line explanation. After putting Facebook's API URL in current_page, we grab the page by this:

    response = requests.get(current_page)

response object now holds the exact response that Github's server replied us with when we try to access the address in current_page. It has got the JSON information of Facebook, along with some other HTTP metadata. We take the JSON info from that using json() method and convert it into Python's familiar dictionary:

    page_info = response.json()

We have all the necessary info in json, you can try testing that by printing it. But we only need the name, id and profile address this time, so:

    # print properties in json dictionary
    print(page_info["name"])
    print(page_info["id"])
    print(page_info["html_url"])

Looping through

We already completed our main objective, that is, scrap info from APIs. Congratulations to you! But we are not done yet. This is not an efficient way to get information of multiple users. So we need to loop through the users list as well as the properties list to get all the user's info. Let's rewrite our test code with the following loop like this:

    # to grab and show info from Github
    for username in users:
        # make API url with Github username
        current_page = baseurl + username
        # take response from the url
        response = requests.get(current_page)
        # conversion to human readable format
        page_info = response.json()
        for property in properties:
            print("{} : {}".format(property, page_info[property]))
        print('')

This time when we run the code, it will grab info of all the usernames in users list. Then print the properties of them too. We can move on to next stage: saving the info in to RethinkDB's database. Here is the full code for further use.

Storing in RethinkDB

RethinkDB is a cool database system that is used for realtime apps. It has a flexible query language, monitoring APIs and is really easy to set up. It does not use SQL to run queries, so you can use your knowledge of Python to do all the stuff you would do with SQL in other databases like MySQL or PostgreSQL. Interestingly data is stored in RethinkDB in the format of JSON. And Python + JSON = <3