NetWorkSpaces for Python

NetWorkSpaces (NWS) is a new way to write parallel programs. It allows you to take advantage of multicore and multiprocessors computers, as well as clusters, using scripting languages such as Python, R, and Matlab. With NetWorkSpaces for Python, you can execute Python functions and methods in parallel using methods very much like the standard Python map function. In some cases, you may be able to parallelize your program in minutes, rather than months.

For example, here's a simple Python NWS script:

          from math import sqrt
          from nws.sleigh import Sleigh
          s = Sleigh()
          for x in s.imap(sqrt, xrange(10)):
              print x
          

It looks pretty simple, and if you're already familiar with the imap function in the standard itertools module, you'll have no trouble with NetWorkSpaces. And yet, it's doing something extremely useful and powerful. You can easily submit millions of tasks that will be executed in parallel, with the results returned to you for processing as they arrive. NetWorkSpaces leverages Python iterators to perform that trick, and by using other functions in the itertools module, you can do in a few lines what would take thousands of lines of C and MPI code to do.

The example above executes three local workers, which is great if you have a four core processor, but if you're using a cluster, all you have to do is construct your Sleigh as follows:

          from nws.sleigh import sshcmd
          s = Sleigh(launch=sshcmd, nodeList=['node1', 'node2'])
          

Now, your Sleigh will start workers on the specified list of machines using ssh. Submitting tasks and processing results remains the same. This makes it easy to develop your programs on your laptop, and deploy it later on a cluster.

NetWorkSpaces is also pretty easy to install, being written in pure Python, and installed using the standard Python Distutils. However, for Windows users, we now have an "all-in-one" NetWorkSpaces installer, available at http://www.lindaspaces.com/downloads/NWSSetup-1.5.1.exe.

So give it a try, and find out how easy it is make your Python programs run faster.

---

For more information on NetWorkSpaces, see the article Python NetWorkSpaces and Parallel Programs at the Dr. Dobb's web site.

---