Using PayPal's Adaptive Payments and Google App Engine to build an online market with Python - Part 3

By Peter Georgeson

Overview

This multipart tutorial demonstrates how to build an online market application with Google App Engine (GAE), PayPal's Adaptive Payments and Python.

To learn about configuring, installing and deploying the basic application, refer to Part 1 of this series. To learn about chained payments, instant payment notification and embedded payments, refer to Part 2 of this series.

This is Part 3 and covers some considerations, caveats and gotchas for running this application in a live, production environment.

The source code for the example application developed for this tutorial is available at GitHub.

SSL Certificate Validation

By default, Python's urllib2 library does not validate the server certificate over an SSL connection. This means that an application relying on urllib2 is vulnerable to man-in-the-middle attacks, which HTTPS was designed to prevent.

In practice this would be a reasonably difficult attack to pull off, however, there's little point in using the HTTPS protocol if the client doesn't first validate who it is talking to.

Strangely, the OSX development installation of Google App Engine does implement host validation, whereas the live application does not implement host validation. Google state that "The urllib module currently provides no methods to validate hosts, but will default to host validation in the near future."

Instead Google provide an alternative class, urlfetch. urlfetch provides the same functionality as urllib2, but with host validation. It's easy to use and was implemented in the sample application within a wrapper class:


class url_request( object ):
  '''wrapper for urlfetch'''
  def __init__( self, url, data=None, headers={} ):
    # urlfetch - validated
    self.response = urlfetch.fetch( url, payload=data, headers=headers, method=urlfetch.POST, validate_certificate=True )

  def content( self ):
    return self.response.content

  def code( self ):
    return self.response.status_code

This code provides the appropriate security against man-in-the-middle attacks.

Note that it's easy to test the host validation: simply replace the host name with its equivalent IP address in settings.py.

Error Handling

When creating the main application object in main.py, it's normal to set debug=True. This sends the stack trace of any exception to the browser window. Useful for debugging, not for a live application. A stack trace not only looks bad, it provides useful information to an attacker. Set debug to False for a deployed application in main.py:


application = webapp.WSGIApplication( [
    ('/', Home),
...
  ],
  debug=False)

Custom Error Pages

It is increasingly common for applications to include user-friendly pages when something goes wrong, particularly for 404 (page not found) and 500 (server error) response types.

An easy way to do this with Google App Engine's web framework is to sub-class webapp.RequestHandler and override the error handler with your own implementation.


class RequestHandler(webapp.RequestHandler):
  def error( self, code ):
    webapp.RequestHandler.error( self, code )
    if code >= 500 and code <= 599:
      path = os.path.join(os.path.dirname(__file__), 'templates/50x.htm')
      self.response.out.write(template.render(path, {}))
    if code == 404:
      path = os.path.join(os.path.dirname(__file__), 'templates/404.htm')
      self.response.out.write(template.render(path, {}))

Now 404 and 50x errors will be directed to a template of your design.

To ensure the main application gets the opportunity to process 404 errors, a catch-all handler was added to the application definition:


    ('/.*', NotFound),

Its implementation simply raises a 404 error:


class NotFound (RequestHandler):
  def get(self):
    self.error(404)

Now the templates 404.htm and 50x.htm need to be created to render these new outcomes. Simple examples are included in the sample application.

This approach only covers errors that occur within the application. It is also possible to catch exceptions raised by the App Engine architecture, such as quota exceptions. This is not covered here - for details refer to the app engine documentation.

Quotas

Google App Engine offers a generous free tier which should cover many applications, however, if there is the potential that your application could make the mainstream media, or the front page of digg, it is a good idea to be aware of what Google App Engine does once you reach your free quota limits, and what these limits are.

For instance, the outgoing daily bandwidth limit is 1Gb. Excess is charged at $0.12/Gb, which compares favourably with other elastic providers such as the Amazon cloud ($0.15/Gb) but unfavourably with traditional web hosting service providers. Similarly, storage is limited to 1Gb, with a pricing at $0.15/Gb for excess storage requirements.

For details of resource pricing refer to App Engine Pricing. Once you have enabled billing on Google App Engine you can increase your daily resource limits based on your budget.

Limitations

App Engine imposes some hard limits to keep their architecture scalable. For instance, all requests must complete within 30 seconds. urlfetch by default must complete within 5 seconds. If PayPal is having a slow day, it may be worth increasing the urlfetch deadline to the maximum - 10 seconds - to avoid reaching this limit.

An alternative to dealing with latency issues is to use urlfetch's asynchronous option, described in the documentation, but not implemented in the example application.

Monitoring and Reliabiity

App Engine provides a useful dashboard for each of your applications, which gives a general idea of the health of your application. Google also provide details of App Engine's overall system status.

As with any live service, an external monitoring service is highly recommended. App Engine has historically experienced numerous outages and do not provide any uptime guarantee or SLA with their base offering. This may soon change with the release of App Engine for Business.

App Engine for business is initially targetted at internal business applications, but Google state that "You can also use App Engine for Business to build external applications. We’re still working out the details on pricing so stay tuned".

Profiling

Google App Engine provide a profiling tool called AppStats which is simple to install and configure. It's extremely useful for diagnosing problems. The file appengine_config.py must be added to the project, and app.yaml updated:

builtins:
- appstats: on

Once done, all API calls are logged. Profiling details can then be analyzed for your development application at http://localhost:8080/_ah/stats/. Similarly, you can also run profiling on your deployed application.

By clicking "Request History" and then selecting a request, full details of the API calls making up a request are listed. In the image above, the urlfetch call took over 3 seconds, indicating that the default timeout value of 5 seconds for this call may not be high enough.

This demonstrates how useful AppStats can be in tracking down issues with your app engine application.

Seller Administration

Although the data store interface provided by Google is extremely useful, sellers may want a more user-friendly web-based interface. A very simple example was added to the sample application:


class SellHistory (RequestHandler):
  def get(self):
    data = {
      'items': model.Purchase.all().filter( 'owner =', users.get_current_user() ).order('-created').fetch(100),
    }
    util.add_user( self.request.uri, data )
    path = os.path.join(os.path.dirname(__file__), 'templates/sellhistory.htm')
    self.response.out.write(template.render(path, data))

The page can be accessed via http://localhost:8080/sellhistory.

A trivial extension would be to enable sellers to update a purchase to fulfilled and to add notes for themselves. A similar page could be added for buyers to track their purchases through the site.

Shipping Physical Goods

One aspect of an online store that hasn't been covered is the shipping of physical goods. By default PayPal doesn't collect a shipping address. This is fine if you are selling digital goods, but if you are selling physical goods a little more work is required.

The process for collecting shipping information is:

The SetPaymentOptions request was implemented in paypal.py:


options_raw_request = json.dumps( {
  'payKey': self.paykey(),
  'senderOptions': { 'requireShippingAddressSelection': 'true', 'shareAddress': 'true' },
  'requestEnvelope': { 'errorLanguage': 'en_US' }
} )
options_raw_response = url_request( "%s%s" % ( settings.PAYPAL_ENDPOINT, "SetPaymentOptions" ), data=options_raw_request, headers=headers ).content()

GetShippingAddresses was implemented similarly:

class ShippingAddress( object ):
  def __init__( self, paykey, remote_address ):
    headers = {
    ...
    }

    data = {
      'key': paykey,
      'requestEnvelope': { 'errorLanguage': 'en_US' },
    }

    self.raw_request = json.dumps(data)
    self.raw_response = url_request( "%s%s" % ( settings.PAYPAL_ENDPOINT, "GetShippingAddresses" ), data=self.raw_request, headers=headers ).content()
    logging.debug( "response was: %s" % self.raw_response )
    self.response = json.loads( self.raw_response )

Unfortunately, although this follows the process recommended by PayPal's documentation, it doesn't work. There's currently a bug in PayPal's adaptive payments implementation. For a traditional payment, the shipping address is not collected; for embedded payments, the GetShippingAddresses call does not return the shipping address.

PayPal have acknowledged the bug and indicated that they expect to have this fixed April 2011. If you need shipping addresses right now, the recommendation is to collect them on your site rather than relying on the PayPal API.

Going Live with PayPal

An application that uses the Adaptive Payments API must first be approved by PayPal. Submit your application to PayPal and expect a wait of 5-15 days while they evaluate the application. If the application passes PayPal's scrutiny then you will be provided with a Live ID, which should then be entered into settings.py. At this point, your application will be taking live payments and real money. Congratulations!

Conclusion

This final part of the series has concentrated on some issues associated with taking an application from basic sample to real application. It has also outlined some of the issues specific to Google App Engine. Hopefully this series has explained and demonstrated some of the fundamental ideas and inspired you to take the tools and services discussed to new and exciting levels. Good luck!

Links and Further Information