Contents


Build and deploy a World Bank charting app on Bluemix using Python, Django, and MongoDB

Comments

This article was written using the Bluemix classic interface. Given the rapid evolution of technology, some steps and illustrations may have changed.

The World Bank provides free and open access to volumes of data about developmental indices such as economy, health, environment, energy, infrastructure, poverty, social development, education, science, and technology for countries across the globe. Being able to visualize World Bank data in chart and diagrammatic forms can help data analysts understand the current evaluation of a country's growth in these various areas.

In this article, I will show you how to develop a web-based charting application that pulls data from World Bank APIs and renders it in comprehensible charts.

For developers, this presents an opportunity to build applications using World Bank data that can help governments take preventive and corrective measures so that their country's resources can be best managed for rapid development. In this article, I will show you how to develop a web-based charting application that pulls data from World Bank APIs and renders it in comprehensible charts.

For simplicity of code and explanation, my charting app focuses on showing energy consumption trends across various countries. The energy consumption dashboard provides quick insight into the consumption trends of various energy sources like solar, biomass, hydro, wind, and geothermal energy in different countries.

About the application

This application is developed on IBM Bluemix using Python, Django, and MongoDB (as data cache). The front end is designed using charting APIs exposed by the JavaScript charting engine Highcharts. Highcharts also allows users to print the charts or download them as images or PDFs. Django is a Python-based open source web application framework that encourages rapid development of web applications and allows users to write clean, easily maintainable code. Some well-known sites that use Django include Pinterest, Instagram, Mozilla, The Washington Times, Disqus, and the Public Broadcasting Service.

Whenever the application is queried for any statistics or metric, such as the solar energy consumption for the United States, it fetches the JSON data associated with that metric from World Bank REST APIs and stores it in MongoDB collections using a PyMongo session. This data is cached in MongoDB for one day. Queries seeking the same statistics are then served by pulling the data from the MongoDB cache instead of querying the World Bank API. The data cache is refreshed every day and is populated again with the first call to the application.

In the final part of this article, I show you how easy it is to deploy the application to Bluemix, the IBM open-standard, cloud-based platform for building, managing, and running apps and services.

Okay, lets get started building the charting app!

What you'll need to build this application

If you want to deploy the application directly to Bluemix and do not want to develop locally, you can skip Steps 1-3 of this article and jump to Step 4. Deploy the application to Bluemix. You can clone the source code from the IBM DevOps Services repository with the command:

git clone https://git.ng.bluemix.net/IDS_Samples/world-bank-trends.git

You can also click the Get the code button above and browse the complete code online using IBM DevOps Services. For a detailed explanation of the code, see Step 3. Create and configure the charting application of this article.

Step 1. Set up a Django project

  1. Install Python and Django. For the local setup, make sure that you have Python installed before installing Django.
  2. Create a Django project. A project in Django is a collection of settings for an instance of Django, including database configuration, Django-specific options, and application-specific settings. To create a Django project, run the following command:
    django-admin.py startproject trends_app

    The startproject command creates a directory named trends_app containing five files. These files, listed below, already constitute a working Django application.

    trends_app/
        manage.py
        trends_app/
            __init__.py
            settings.py
            urls.py
            wsgi.py
  3. If you are already familiar with Django, you can skip the following details on these files. If not, the following short descriptions should help you with the basic understanding of what these files do.
    • trends_app/: The outer trends_app/ directory acts like a container for the Django project.
    • manage.py: A command-line utility that lets you interact with your Django project. Typing python manage.py gives you a list of available commands.
    • trends_app/trends_app/: The inner trends_app/trends_app directory is the actual Python package for your project. Its name is the Python package name you'll need to use to import anything inside it, for example, import trends_app.settings.
    • __init__.py: This file is required for Python to treat the trends_app/ directory as a package (a group of Python modules).
    • settings.py: The settings/configuration for this Django project.
    • urls.py: The URLs for this Django project. Think of this as the "table of contents" of your Django-powered site.
    • wsgi.py: An entry-point for WSGI-compatible web servers to serve your project.

Step 2. Start the development server

Django has a built-in lightweight development web server that automatically reloads when any changes are made to the source code.

  1. To start the server, change to your project container directory (cd trends_app) and run this command:
    python manage.py runserver

    If you see the following message it means the server is launched locally on port 8000.

    Validating models...
    
    0 errors found
    August 25, 2014 - 21:51:25
    Django version 1.6.5, using settings 'trends_app.settings'
    Starting development server at http://127.0.0.1:8000/
    Quit the server with CTRL-BREAK.
  2. Visit http://127.0.0.1:8000/ with your web browser and you'll see your first Django-powered page. First django-powered page
    First django-powered page

Step 3. Create and configure the charting application

  1. Create a sample Django application with the following command, which creates a Django app called trends.
    django-admin.py startapp trends
  2. Modify the settings.py file to use the following code:
    INSTALLED_APPS = (
        'django.contrib.auth',
        'django.contrib.contenttypes',
        'django.contrib.sessions',
        'django.contrib.sites',
        'django.contrib.messages',
        trends,
    )
  3. Now, configure the URL mapping. The mapping dictates how you access your app. The following code listing shows how the trends_app/urls.py file should look:
    from django.conf.urls import include, url
    
    urlpatterns = [
        url(r'^trends/', include('trends.urls'))
    ]
  4. Change the trends/urls.py file to include code that maps the URL fetch_and_draw_data/ to the fetch_and_draw_data() view function, as shown in the following code listing. The trends/urls.py file maps a URL to a view function so that Django knows where to find the view functions.
    urlpatterns = [
        url(r'^$', views.index, name='index'),
        url(r'^fetch_and_draw_data/', views.fetch_and_draw_data, name='fetch_and_draw_data'),
    ]
  5. Include the following view function named fetch_and_draw_data() to the trends/views.py file. The trends/views.py file contains Python functions that accept a request parameter and return a response. The request is typically a request from the web server, and the view takes any parameters passed along with this request. The view then performs the logic required to determine the appropriate response.
    def fetch_and_draw_data(request):
        connect.initialize()
        country = request.GET.get('country', 'USA')
        metric = request.GET.get('metric', 'Total')
    
        country_code = countries_mapping[country]
        indicator_code = indicators_mapping[metric]
        data_url = get_url(country_code, indicator_code)
        json_from_cache = connect.get_json_data(data_url)
        ...
        //create a data list
        data_list = {"graphTitle": graphTitle ,"xAxisLabels" : xAxisLabels,
        "xAxisTitle" : xAxisTitle, "yAxisTitle" : yAxisTitle, "yAxisValues" : yAxisValues}
        return HttpResponse(simplejson.dumps(data_list))
  6. Create a new file called trends/connect.py and add the initialize() function to it. The trends/connect.py file is responsible for the connection to MongoDB. The connection is established using PyMongo, which is a Python interface to MongoDB, and is initialized in the MongoDB by calling the initialize() function.

    The following code snippet shows how a PyMongo session is established using connection credentials obtained from the Bluemix VCAP_SERVICE variables (assuming you already have an instance of the MongoDB service running on Bluemix).

    # Initialize MongoDB connection with credentials from vcap config
    def initialize():
      # If connection is not initialized, initialize it.
      if not config.initialized:
        #Get the connection credentials from VCAP variables
        vcap_config = os.environ.get('VCAP_SERVICES')
        decoded_config = json.loads(vcap_config)
        for key, value in decoded_config.iteritems():
             if key.startswith('mongodb'):
                 mongo_creds = decoded_config[key][0]['credentials']
        mongo_url = str(mongo_creds['url'])
    
        client = pymongo.MongoClient(mongo_url)
        config.db = config.client['db']
        config.collection = config.db['mycollection']
        config.collection.remove({})
        config.initialized=True
        refresh()
  7. Add the refresh() function to the trends/connect.py file. The refresh() function is called when the initialize() function is executed. It is a periodic task that kicks in every 24 hours and cleans up the existing data in MongoDB collections so that the next call to the application fetches data from the World Bank REST URL.
    # Periodic task to purge data from MongoDB collection
    def refresh():
      config.collection.remove({})
      threading.Timer(86400, refresh).start()
  8. Add the get_url() function to the trends/views.py file. The get_url() function takes the country code and metric name (for example, solar energy consumption) as a request parameter and constructs an API URL, as shown the following code listing:
    def get_url(country_code, indicators):
      return ("http://api.worldbank.org/countries/%s/indicators/%s?per_page=10&date=2000:2010&format=json" % (country_code, indicators)) ;
  9. Add the get_json_data() function to the trends/connect.py file. This function checks to see if the URL is already present in the MongoDB collection. If the entry for the URL is not found, it retrieves the data in JSON format by making a call to the API URL, then caches it in the MongoDB collection. If the entry is found, it returns the cached data corresponding to the existing URL.

    The following code listing shows how fetching and caching of data happens through a call to the get_json_data() function.

    # Hit the World Bank url to retrieve data
    def get_json_data(data_url):
      # If data is in cache, return it
      if(config.collection.find_one({"url": data_url}) is not None):
        return config.collection.find_one({"url": data_url})["jsonData"]
      # Fetch it from MongoDB otherwise
      else:
        response = urllib2.urlopen (data_url)
        jsonData = simplejson.load(response)
        config.collection.insert({'url': data_url, 'jsonData' : jsonData})
        return jsonData
  10. Add a directory called templates in the trends_app directory. Then, in the templates directory, add a file called index.html (the only template in the project), and import the Highcharts JS library into it. Highcharts JS is a JavaScript charting engine popular for its flexible and simple approach toward rendering data in the form of interactive charts.

    The following code listing shows the Highcharts import:

     # Import highchart javascript
    <script src="//code.highcharts.com/highcharts.js"></script>
    <script src="//code.highcharts.com/modules/exporting.js"></script>
  11. Add the JavaScript fetch_and_draw() function to the index.html file. This function makes an Ajax call to the back-end fetch_and_draw_data view function, as shown in the following code listing.
     $.ajax({
                    url : "/trends/fetch_and_draw_data",
                    type : "GET",
                    dataType: "json",
                    data : {
                        csrfmiddlewaretoken: '{{ csrf_token }}',
                        country : country,
                        metric : metric
                    },

    When a metric is selected from the tab menu or a country is selected from the drop-down menu, tab_id is retrieved and the fetch_and_draw() function is called, which, in turn, gets the data from the back end by making an Ajax call to the fetch_and_draw_data URL. Finally, the bar chart is rendered using Highcharts. The following code snippet shows the call to the JavaScript fetch_and_draw() function:

    $('#tabs').tabs({
            activate: function(event ,ui){
              var tab_id = ($("#tabs").tabs('option', 'active'));
              tab_id += 1;
              fetch_and_draw(tab_id);
                
            }
        });
    
    
    $('#countries_id').change(function()
            {
              var tab_id = ($("#tabs").tabs('option', 'active'));
              tab_id += 1;
              fetch_and_draw(tab_id);
            
        });

    You can see the complete code listing of the views.py and index.html files on IBM DevOps Services.

Step 4. Deploy the application to Bluemix

After the application is coded, it's time to deploy it to Bluemix.

  1. Add the requirement.txt file in the root of your project. The requirement.txt file should contain all the external dependencies (like Django and PyMongo) that are needed to run the charting application. When running the application, Bluemix makes sure that this file is read and that the dependencies listed are installed.

    The requirement.txt file needed for this project is:

    Django==1.6.5
    pymongo==2.7.1
  2. Create the run.sh file in the root of your project. The run.sh file is the shell script that starts the web app after the application is deployed on Bluemix. Make sure you get the port value from the environment variable VCAP_APP_PORT when starting the web server.

    Add the following lines to the run.sh file.

    #!/bin/bash
    if [ -z "$VCAP_APP_PORT" ];
    then SERVER_PORT=5000;
    else SERVER_PORT="$VCAP_APP_PORT";
    fi
    echo port is $SERVER_PORT
    python manage.py runserver --noreload 0.0.0.0:$SERVER_PORT
  3. Download and install the Cloud Foundry (cf) command-line tool that will be used to upload and manage the app. To validate if Cloud Foundry is installed successfully, you can run the following command to return the version:
    cf --version
  4. Log in to Bluemix and set the end point. You can log in to Bluemix from the CLI by running the command:
    cf login -a https://console.bluemix.net/login
  5. Push the application to Bluemix. Go to the root of your project (trends_app/) and upload the charting application to Bluemix by running the following command:
    cf push  trendsapp001 --no-manifest --no-start -b https://github.com/cloudfoundry/python-buildpack  -c "sh run.sh"

    The --no-start option indicates that the application should not be started unless it is bound to the required services. This ensures the MongoDB service is bound to the application before the app is started.

    The -c option is used to specify the start command that should be executed by Bluemix while starting the app.

    The -b flag indicates the buildpack to be used. For this charting app, we need a Python runtime, which is obtained by passing the Git repository URL of the Python buildpack (https://github.com/cloudfoundry/python-buildpack) with the -b option.

    You can also provide the application name, buildpack information, and sh run.sh command in the manifest.yml configuration file, as follows:

    ---
    applications:
    - name: trendsapp001
      memory: 256M
      command: sh run.sh
      buildpack: https://github.com/cloudfoundry/python-buildpack
  6. Include the manifest.yml file in the project root. When using the manifest.yml file, push the application to Bluemix using the following command:
    cf push  --no-start

    Remember to choose a different host name so that your resulting URL route doesn't conflict with the demo URL. You should see a screen similar to the one below when pushing the charting app to Bluemix.

    Screen capture of output of 'cf push' command
    Screen capture of output of 'cf push' command
  7. Create an instance of the MongoDB service. In the terminal, go to the project root and create a MongoDB service instance with a unique name by typing the following command:
    cf create-service mongodb 100 mongodb001
  8. Bind the MongoDB service instance to the new app, trendsapp001 in our case, by typing the following command:
    cf bind-service trendsapp001 mongodb001

    Alternatively, you can also create a MongoDB service instance by logging into the Bluemix administration dashboard and locating the charting app under the APPS menu.

    Bluemix dashboard - application listing
  9. Select your application, and on the resulting page use the Add Service option to add the MongoDB service to your application. Screen capture of Bluemix dashboard - adding MongoDB as a service
    Screen capture of Bluemix dashboard - adding MongoDB as a service

    You should now see a MongoDB service instance that is bound to your application in the Bluemix administration dashboard.

    Screen capture of Bluemix dashboard - listing bound services
    Screen capture of Bluemix dashboard - listing bound services

    After you bind a MongoDB service instance to the application, the following configuration is added to your VCAP_SERVICES environment variable:

    Bluemix dashboard - environment variables
    Bluemix dashboard - environment variables
  10. Run the following command to start the app.
    cf start trendsapp001
  11. Access your charting application by visiting trendsapp001.mybluemix.net/trends (replacing trendsapp001 with the unique host name you provided in your manifest.yml file).

    The following screen capture shows the solar energy consumption chart for the United States as displayed in the app. You can print the chart or download it in image or PDF format by clicking on the menu on the right of the screen.

    Screen capture - Energy Consumption Dashboard
    Screen capture - Energy Consumption Dashboard

Conclusion

In this article, I have demonstrated how Django applications can be created and deployed on the Bluemix platform. I have also included details on how to use the Bluemix MongoDB service as a data cache and how to integrate it with the Django framework. The source code for the charting application presented in this article can be downloaded from its IBM DevOps Services repository, along with all the configuration files needed to get it up and running. After building this charting app, you should now be able to apply the process in your own projects. The possibilities are endless!


Downloadable resources


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Cloud computing
ArticleID=985855
ArticleTitle=Build and deploy a World Bank charting app on Bluemix using Python, Django, and MongoDB
publish-date=10142014