Archive for the ‘python’ Category

Book Review – Python for Data Analysis

Sunday, January 20th, 2013


Python for Data Analysis is primarily a reference for Pandas. Pandas is a Data Analysis library for Python.

Also covered in less depth are some other components in Python’s data analysis ecosystem. There are chapters on iPython and NumPy. A chapter on plotting and visualization provides a great rundown of matplotlib, along with mention of alternatives like chaco and mayavi.

Pandas is then presented in significant depth, with sections on data storage, data transformation, data aggregation and time series analysis. This forms the bulk of the book.

This is a well-written book that provides a good summary of Python’s data analysis capabilities, however, it will not teach you how to do data analysis. This book will show you how to use the Pandas library.

Note: This book was provided by O’Reilly Media as part of their blogger review program.

I review for the O'Reilly Blogger Review Program

Book Review – Programming Computer Vision with Python

Tuesday, September 18th, 2012


Programming Computer Vision with Python covers the foundations of computer vision along with numerous interesting and practical examples.

Topics covered include:

  • Basic image handling and processing in Python
  • Image descriptors and points of interest
  • Image mapping and homographies
  • Augmented reality
  • 3D scene reconstruction
  • Clustering, searching and classifying images
  • Image segmentation and
  • Interfacing to OpenCV

Python forms an integral part of this book and is used throughout the book. It is an ideal language for this purpose, being easy to understand and with excellent libraries. The scientific library numpy is used extensively.

This book is well-written, easy to understand and a lot of fun. There was a heavy emphasis on practicality which I appreciated.

Typically the theory would first be explained in text, then implemented in code. Finally a practical example would demonstrate how to apply the theory.

There are some great examples. This is where the book shines.

For instance, in the chapter on image classification, we are given an image of a Sudoku. The grid location is first determined, then the contents of each cell in the grid are classified into digits.

It was great to see some machine learning algorithms applied to real problems. PCA (Principal Component Analysis) is used extensively. SVM (Support Vector Machines) and the Naive Bayes Classifier are also used to solve real computer vision problems.

The author clearly knows his stuff, often pointing out pitfalls and demonstrating many handy tricks.

A very enjoyable and recommended introduction to the world of computer vision.

Note: This book was provided by O’Reilly Media as part of their blogger review program.

I review for the O'Reilly Blogger Review Program

Python localization made easy

Friday, September 16th, 2011

Python
Here’s a set of simple steps to localize a Python application for different translations. This tutorial provides a clear set of steps with sample code.

Step 1: Initialize your application

Here is the code to initialize your application with localization enabled:

If you can’t see the source code try here.

This snippet looks for a resource file based on the users locale. For instance, “res/messages_en.mo” for English. If it fails to open the appropriate translation file, it falls back to NullTranslations, which simply performs no translation.

Step 2: Prepare your application for translation

trans.install() generates a global function available to all modules in your application: _().

Find all the strings in your application that you wish to translate, and wrap them with the _() function.

i.e. “Hello” becomes _(“Hello”).

This applies to parameterized strings as well. e.g. “Hello %s” % name becomes _(“Hello %s”) % name.

Step 3: Generate the pot

That’s messages.pot.

Run the command xgettext *.py or pygettext *.py. Under Windows, you might have to look for this tool. Under the Python installation directory, try Tools/i18n.

This command looks for all strings inside the _() function, and generates the file messages.pot.

Step 4: Translate

Send your generated pot file to your translator. They will replace the empty strings with the appropriate translations and return the file to you.

Step 5: Generate the mo

Save the returned file to reflect the new language that your application has been translated to. e.g. messages_De.po

Run the command msgfmt -o res/messages_De.mo messages_De.po to generate the required .mo file.

As with pygettext, if your system doesn’t find this command, look in Tools/i18n under the Python installation directory.

After running this command, the translation file required by the application will be in the res directory. When you’re distributing the application, make sure the res directory goes too.

Step 6: Test

On Windows XP, you can change your locale with the following steps:

  • Start->Control Panel->Regional and Language Options;
  • Under “Regional Options”, choose the locale for the translation file you have created and click “Apply”;
  • Start the Python application;
  • (Hopefully) enjoy your translated application!

In summary…

It’s straightforward to setup localization with Python once you know how.

Real-Time Auctions with HTML5, PayPal, and Google App Engine

Tuesday, June 28th, 2011

Google App EngineSupernifty’s latest series of technical articles demonstrates the use of Google App Engine and PayPal to build a real-time auction site.

We also show off some HTML5 and discuss some of the issues associated with designing a site suitable for mobile devices.

All pretty interesting if you’re into this kind of stuff. Check it out:

Or go straight to the source code.

Google App Engine, PayPal, Adaptive Payments – The Tutorial

Wednesday, April 13th, 2011

Google App EngineThe final part of my tutorial on using Google App Engine with PayPal’s adaptive payments to create an online market is available.

It extends the example online market application developed in part two by covering some of the practical considerations and handy features of hosting on GAE.

In summary:

Google App Engine and PayPal Adaptive Payments with Python – part 2

Friday, March 18th, 2011

Google App EngineIf you’re interested in Google App Engine, online payments and Python, then check out part 2 of Supernifty’s tutorial.

It extends the example online market application developed in part one by demonstrating:

  • Chained payments;
  • Instant Payment Notification (IPN); and
  • Embedded Payments
  • All handy stuff if you want to learn how to use this technology.

    In summary:

Google App Engine, PayPal, Python – A Tutorial

Monday, February 28th, 2011

Google App EngineInterested in accepting payments with PayPal Adaptive Payments using Google App Engine? Check out this tutorial by Supernifty.

It covers the steps involved in getting set up with Google App Engine and building a simple store with PayPal as the payment provider.

The source code is available from GitHub.

In summary:

Getting started with PayPal on Django

Thursday, January 27th, 2011

DjangoIf you’re a Django (or Python) developer, and want to integrate a payment solution into your web app, check out this tutorial by Supernifty: Getting started with PayPal on Django.

It covers the steps involved in getting set up with PayPal and building a simple online store with Django.

The sample code is also available from GitHub.

In summary:

py2exe, python 2.6 and wxPython – hack it #7

Tuesday, July 6th, 2010

Python
If you’ve tried to build a client application with Python 2.6 and wxPython, you might hit a problem.

Python 2.6 no longer includes the Visual Studio dependencies required by wxPython. These dependencies may not be present, either on your build machine, or the target user’s machine.

When building with py2exe, you may encounter an error similar to:

error: MSVCP90.dll: No such file or directory

The solution!

To successfully build your executable, you need the Visual Studio 2008 redistributable package, which can be obtained from Microsoft. Run this on your build machine to install the appropriate DLLs.

If your build still fails, you may need to copy the DLLs into your build directory. Locate msvcp90.dll, msvcm90.dll and msvcr90.dll on your machine and copy them to the build directory.

Once you have a successful Python build, you also want your executable to run on the target machine. To do this, your install script needs to install the vcredist package as part of the installation.

The command you need to run as part of your client installation is:

vcredist_x86.exe /q:a

If you are using NSIS as your installer, the appropriate code is:

File /r vcredist_x86.exe ;to copy the file
ExecWait ‘”$INSTDIR\vcredist_x86.exe” /q:a”‘ ;to install

With the Visual Studio redistributable installed, it will now execute successfully on the target machine. Hurrah!

Backup multiple MySQL databases with Python – hack it #5

Wednesday, May 28th, 2008

Keep copies, or risk the great sadness

Backing up is important.

If you host a website which has a database, and you’re not backing it up, you should.

If your database was wiped, and you lost your entire blog, wiki, customer details, and the rest, if you’re like me, you’d be deeply unhappy.

Most webhosting companies do backups, but are somewhat vague on guarantees and process. They often charge to do restores as well.

Best bet is to do it yourself.

This set of instructions applies to MySQL, and can be applied to one database or many.

Step 1. Backup your databases on the server
Below is a python script that will backup multiple MySQL databases. Put this on your web server, preferably in a directory called backup, with the name backup.py.

import os
 
databases = dict()
databases['*** db1 ***'] = '*** password1 ***'
databases['*** db2 ***'] = '*** password2 ***'
databases['*** db3 ***'] = '*** password3 ***'
 
def set_pass( name ):
 file = open( '/*** your home directory ***/.my.cnf', 'w' )
 file.writelines( ( '[client]\n', 'password=%s\n' % databases[name] ) )
 file.flush()
 file.close
 
for key in databases.keys():
 set_pass( key )
 os.system( "mysqldump --user %s --opt %s > %s.sql" % ( key, key, key ) )
 
os.system( "zip backup.zip *.sql" )
os.system( "rm *.sql" )
os.system( "chmod 600 backup.zip" )

About this script:

  • A problem with backing up multiple databases without intervention is the entering of passwords in a secure way. This script uses .my.cnf to set the password securely.
  • Fill in the database usernames and passwords. This script assumes the database has the same name as the username.
  • Fill in your home directory e.g. /home/john
  • The result of this script is backup.zip, in the current directory.

Step 2. Copy backup to your machine
This script connects to your web server, runs the above backup script, then copies the result back to your local machine. Put this on your local machine.

ssh -i "your_key" you@yourhost.com "cd backup; python backup.py"
rsync -avz --rsh="ssh -i your_key" you@yourhost.com:~/backup/ backup

About this script:

  • This script assumes the backup script is in the “backup” directory on your web server.
  • You need ssh and rsync to use this script. On Windows, PuTTY provides a great ssh client, but I don’t know of a good rsync implementation, other than cygwin. Instead of rsync, you could use scp, which comes with PuTTY.
  • If you want to automate this script (see later), you’ll want passwordless ssh, which means setting up “your_key”. Google on how to set this up.

Step 3. Repeat periodically
You want to automate this process so you can forget all about it.

  • Unix: crontab. If you’re unfamiliar with crontab, try “man crontab”. The main commands to use are “crontab -e” and “crontab -l”.
  • Windows: scheduler. Check out Control Panel->Scheduled Tasks, and add your client script.
  • As mentioned above, you need passwordless ssh so your scheduled task can run unattended.

There’s your nutshell guide to backing up multiple MySQL databases with Python.