Visualizing price data with the eBay API

By Peter Georgeson

Overview

The eBay developer API is powerful and comprehensive. It is a window to an amazing source of data. This tutorial describes one possible use case where eBay's data can be utilized, and explains how to configure your application to retrieve eBay's data. In this instance, the Finding API is used.

The Problem

Suppose you are about to list an item for sale on your website shopping cart - what is a competitive price? One way to get an idea of the competition is to look at what an item is selling for on eBay.

Although you can physically go to eBay and look at prices yourself, looking over eBay listings is not a particularly efficient use of your time. Much better would be for the shopping cart to generate a summary of pricing data and display this in-page to allow for a realistic price to be set with minimal user effort.

Fortunately, eBay's Finding API is an excellent option for retrieving prices for items currently for sale on eBay. In fact, eBay data can even be used to set an initial price for an item without any user interaction.

Setting up

This article extends a simple Google App Engine shopping cart, available on GitHub. This article assumes that you have the appropriate accounts configured. More detailed instructions are available elsewhere, but briefly:

You should now have a locally running app engine application.

Connecting to eBay

In this example, we want to retrieve prices for a specified item from eBay, so as to come up with a reasonable price on our own shopping site. The finding API is well suited to this.

First, a few constants are added to settings.py. The finding API has a different endpoint to the trading API, so the new endpoint is added here:


EBAY_FIND_ENDPOINT = 'http://svcs.sandbox.ebay.com/services/search/FindingService/v1' # sandbox
#EBAY_FIND_ENDPOINT = 'http://svcs.ebay.com/services/search/FindingService/v1' # live
USE_EBAY_PRICER = True

This example uses the findItemsAdvanced call, which is the most flexible of the find methods. This method allows a full search on the item's title, subtitle and description. Using the appropriate documentation, a template for the call is defined in templates/ebay_find.xml:


<findItemsAdvancedRequest xmlns="http://www.ebay.com/marketplace/search/v1/services">
  <keywords>"{{ keywords }}"</keywords>
  <descriptionSearch>true</descriptionSearch>
  <itemFilter>
    <name>ListingType</name>
    <value>FixedPrice</value>
    <value>AuctionWithBIN</value>
  </itemFilter>
</findItemsAdvancedRequest>

The findItemsAdvanced is particularly powerful because of its filtering capabilities, however, it is still quite simple to use for our basic requirements. In this case, we ask only for items that are marked as fixed price or buy it now. This is so that the returned prices are an accurate indication of what the purchase price of the item would be. This often results in a limited result set - however, we get a guaranteed price, rather than a much lower price that would be given by an auction in progress.

Some other points about the above template:

With the template written, a new eBay class is created to build the XML template, make the HTTP request and parse the results. The class is added to ebay.py:


class Find(object):
  '''find items by keyword'''
  def __init__(self, keywords):
    headers = {
      'X-EBAY-SOA-SERVICE-NAME': 'FindingService',
      'X-EBAY-SOA-OPERATION-NAME': 'findItemsAdvanced',
      'X-EBAY-SOA-REQUEST-DATA-FORMAT': 'XML',
      'X-EBAY-SOA-RESPONSE-DATA-FORMAT': 'XML',
      'X-EBAY-SOA-SECURITY-APPNAME': settings.EBAY_APPID,
      'X-EBAY-SOA-SERVICE-VERSION': '1.0.0',
      'X-EBAY-SOA-GLOBAL-ID': 'EBAY-US'
    }

    data = {
      'keywords': keywords
    }
    template_path = os.path.join(os.path.dirname(__file__), 'templates/ebay_find.xml')
    self.raw_request = template.render(template_path, data)
    logging.info( 'ebay request: ' + self.raw_request )
    self.raw_response = url_request( settings.EBAY_FIND_ENDPOINT, data=self.raw_request, headers=headers ).content()
    logging.info( 'ebay response: ' + self.raw_response )
    # extract list of prices
    fixed_prices = re.findall( "<convertedCurrentPrice[^>]*>([0-9.]*)</convertedCurrentPrice>", self.raw_response )
    buynow_prices = re.findall( "<convertedBuyItNowPrice[^>]*>([0-9.]*)</convertedBuyItNowPrice>", self.raw_response )
    self.prices = fixed_prices + buynow_prices
    self.success = self.raw_response.find( "<ack>Success</ack>" )

The overall format of the class mirrors those that access the trading API, however, the HTTP headers that are required for this call are quite different. In this instance, only an APPID is required; the API version is completely different; and a GLOBAL-ID defines which eBay market to use. As mentioned earlier, the endpoint is also different.

A regular expression is used to extract the prices for fixed price items (convertedCurrentPrice) and Buy It Now items (convertedBuyItNowPrice). Whether or not the call succeeded is also determined by searching for a success string in the returned XML.

For anything even slightly more complicated than extracting prices, the response should be parsed. For instance, when returning price data, you may also want to return a link to the associated item. The response will need to be parsed to do this.

The easiest way to parse the response is to take advantage of eBay's support for multiple formats. Change the HTTP header X-EBAY-SOA-RESPONSE-DATA-FORMAT to JSON and eBay will respond with a JSON formatted string. Now Python's simplejson module can be utilized and the response can be parsed into an easy-to-query object tree with one line:


self.response = json.loads( self.raw_response )

Exposing price data to the client

In this example, we want to dynamically retrieve and display price data on the sell page. This is achieved by adding a simple AJAX endpoint to main.py:


application = webapp.WSGIApplication( [
  ...
  ('/api/(.*)/(.*)', API),

This endpoint accepts two parameters: a command and data. For this example, the only command accepted is "find". The implementation of the API class is defined in main.py:


class API(RequestHandler):
  @login_required
  def get(self, cmd, data):
    if cmd == 'find':
      find = ebay.Find( urllib.unquote( data ) )
      if find.success:
        self.response.out.write( simplejson.dumps( find.prices ) )
      else:
        self.error(501)
    else:
      self.error(501)

The handler implements a simple wrapper to the eBay interface. An error is raised if something goes wrong; otherwise prices are wrapped in a JSON object to be parsed by the client.

With this code in place, the API can easily be tested directly with a web browser. For instance, try browsing to http://localhost:8080/api/find/catch+22 and a JSON encoded list of prices should display in the browser.

The front end

With the API implemented, it can now be utilized by the front-end. Some simple JavaScript is added to provide the dynamic content.

In sell.htm, the appropriate JavaScript libraries are first loaded:


{% block head %}
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js"></script>
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
<script src="/static/main.js"></script>
<script type="text/javascript">
  supernifty.init();
</script>
{% endblock %}
...
{% if use_ebay_pricer %}
  <p><a onclick="supernifty.prices(); return false" href="/nojs">Generate Price Data</a></p>
{% endif %}
...
<div id="ebay_data">
</div>

jQuery is included for the AJAX capability; the Google library is used for visualization. main.js contains the application functionality.

static/main.js contains the client-side implementation. When the user clicks "Generate Price Data", supernifty.prices() is invoked:


prices: function() {
  var title = document.getElementById("title").value,
    url = '/api/find/' + encodeURIComponent(title),
    target = document.getElementById("ebay_data");
  target.innerHTML = 'Please wait...';
  $.get( url, success );
}

The URL to call the AJAX endpoint is generated, then jQuery is used to make the request. All going well, the success function is called. This function first does some work on the returned data - jQuery is used to parse the JSON, then the prices are sorted.

To prepare the data for visualization as a histogram, the price data is divided into ten buckets. We track how many data points are in each bucket, and the minimum and maximum price of the items in a bucket:


bucketSize = ( prices[len-1] - prices[0] ) / 10;
for ( i = 0; i < prices.length; i++ ) {
  bucket = Math.floor( ( prices[i] - prices[0] ) / bucketSize );
  if ( histogram[bucket] == undefined ) {
    histogram[bucket] = { frequency: 1, min: prices[i], max: prices[i] };
  }
  else {
    histogram[bucket].frequency += 1;
    if ( histogram[bucket].min > prices[i] ) {
      histogram[bucket].min = prices[i];
    }
    if ( histogram[bucket].max < prices[i] ) {
      histogram[bucket].max = prices[i];
    }
  }
}
After executing this code, we now have an array of items, with each element containing a frequency, min and max.

The next step is to turn this into a chart to give the user an idea of the distribution of prices for the specified item. Google Chart Tools are powerful, easy and free. The library provides a simple interface and deals with all those nasty browser compatibility issues.

Setting up a visualization with Google Chart Tools typically involves first populating a data table, then specifying how it should be displayed to the user.


var data = new google.visualization.DataTable();
data.addColumn('string', 'Range');
data.addColumn('number', 'Frequency');
data.addRows( histogram.length );
for ( i = 0; i < histogram.length; i++ ) {
  if ( histogram[i] == undefined ) {
    data.setValue( i, 0, 'none' );
    data.setValue( i, 1, 0 );
  }
  else {
    if ( histogram[i].min == histogram[i].max ) {
      data.setValue( i, 0, '$' + histogram[i].min );
    }
    else {
      data.setValue( i, 0, '$' + histogram[i].min + '-' + histogram[i].max );
    }
    data.setValue( i, 1, histogram[i].frequency );
  }
}
In the above code, we iterate over the generated histogram and add it to the data table. The title for each row is based on the minimum and maximum prices in that bucket.

Once this is done, the visualization is easy:


var chart = new google.visualization.BarChart( target );
chart.draw( data, {width: 600, height: 400, title: 'Price Histogram', vAxis: {title: 'Price', titleTextStyle: {color: 'red'}} });
The chart is rendered in the target element. The image below shows the returned price data for the classic book, Catch-22.

ebay price chart

Since the Google Charts Library separates the data from the view, it is easy to render the same data in a different type of chart.

As part of the calculation, the code also calculates the median price and automatically populates the price element in the form. According to this formula, a fair price for Catch-22 is $7.07.

Conclusion

This tutorial demonstrated how to extract and display eBay price data in a relevant manner. This is a simple example which hopefully highlights the many exciting possibilities for using the eBay API, and how easy it is to integrate eBay data into your own application.

This example could easily be extended to be more robust and useful. All source code for this example is available on GitHub - if you decide to extend this idea, I would love to hear about it!

Links and Further Information