• Code chunks can be run as a block (great for debugging, and checking out sensitivity to starting values
• The R magic tool lets you send objects to R for work there (I am still taking a long time to learn python – at the moment rather a lot of plotting functions are farmed out that way
• There is an online notebook viewing service available via the nbviewer website. The current version of my notebook can be viewed here: Gibbs Sampler Changepoint

One small issue I’m still struggling with in terms of the nbviewer is the ability to output directly to html – I can’t find a template file html-blogger for example. But still, this looks like a very promising tool indeed.

1. Create an IPython profile with a password for the Notebook server.
2. Figure out your IP address on the local network.
3. Launch IPython in broadcast + read-only mode using your new profile.
4. Have your students navigate to your Notebook server.

Make a New Profile

IPython has quite a number of configuration options that you might like to be set one way or another depending what you’re doing. To help make that configuration management easier IPython has profiles. We’re going to make a new profile in which we’ll set a password that must be entered to edit or make new notebooks. Here’s the command:

ipython profile create teaching

I’ll use a profile named “teaching” throughout these instructions, you can name it whatever you like.

Setting a Notebook Password

One of IPython’s configuration options is the ability to set a password on the Notebook server. This usually makes it so that only people with the password can access the server, but if you start the server with the --read-only flag anyone can access and see the notebooks, but only you with the password can can edit notebooks or make new ones.

Follow the IPython instructions on making a Notebook password to generate your password hash string. Once you’ve got your password hash string
(which should look something like sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed), you’re going to put that into the Notebook configuration file for you new teaching profile. To find out where that is you can run this command:

ipython locate profile teaching

It will probably tell you something like ~/.ipython/profile_teaching, which is the directory where the configuration files for that profile are stored. Open the file called ipython_notebook_config.py within that directory and find the line that has the string c.NotebookApp.password. Uncomment that line and paste your password hash string into the empty quotes there. When you’re done it should like this:

c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

And when you start the Notebook server with the teaching profile it will be password protected. You can change your password at any time be generating a new password hash string and pasting into ipython_notebook_config.py.

Find Out Your IP Address

Your students need to know where to point their browsers so they can see your notebooks and for that you need to know your IP address on the local network. On Macs you can get this from the Network pane of System Preferences, but I like to use this Python script from Software Carpentry, which should work on any system. On my home WiFi my IP is usually something like 10.0.1.9 but it will vary depending on where you are. Take note of your IP address, in a minute we’ll combine it with the port number used by the IPython Notebook server so your students can type something into their URL bars.

Start The Notebook Server and Note the Port Number

It’s time to start your Notebook server with some added options. You’re going to tell it:

• Use my teaching profile: --profile=teaching
• Broadcast yourself: --ip='*'
• Allow others to see my notebooks without a password: --read-only

Here’s what the whole command looks like:

ipython notebook --profile=teaching --read-only --ip='*'

When IPython starts up the Notebook server it will print a line containing the URL of the server, including the port number. It will look like this:

[NotebookApp] The IPython Notebook is running at: http://[all ip addresses on your system]:8888/

The last four numbers after the colon (in this case 8888) are the port number of the server. Take note of that number.

Tell Your Students Where to Go

Finally you can tell your students what to type into their browsers. Combine your IP address with the Notebook server port number (separated by a colon) and that will be the URL your students go to. For me that would be:

http://10.0.1.9:8888/

When your students land there they should see the usual IPython Notebook dashboard and they should be able to look at your existing notebooks, but they shouldn’t be able to create new notebooks, save changes to your notebooks, or execute code.

Some Notes

• The students’ views don’t update in real time. I find that in order for them to see my changes I must save my notebook and then the students have to manually reload. They won’t see unsaved changes and they won’t see any changes at all unless they reload.
• The IPython security docs recommend making an SSL certificate so you can connect to your server over HTTPS, which will encrypt your password. I generally don’t bother, but be advised that anyone who can execute code on your computer can do things like delete your entire hard drive. Either way, be sure not to use a password you use anywhere else.

• Install command line tools on Xcode (Preferences–>Downloads)
• Checked my version of Python (yes, I have Python >2.7)
• Installed SciPy superpack from this site (used instructions for OS X 10.8)
• Installed PySide pyside-1.1.0-qt47-py27apple.pkg from this site
• Installed readline, pyzmq, pygments, tornado using this command: sudo easy_install readline pyzmq pygments tornado
• Installed iPython from Github using this command: sudo easy_install https://github.com/ipython/ipython/tarball/master
• Tested iPython install success by typing this command: ipython notebook –pylab inline (it worked! HTML notebook launched!)
• Followed the manual install instructions for StarCluster (see quick start install guide below) – had to download tarball, unzip and run script.
• Followed the StarCluster config instructions (but not sure if I did this right…)

Questions I still have:

• Do I need to install QT? Seems like I don’t need it if I’m using iPython Notebook in a web browser, but can’t find much info about possible QT dependencies.
• Need some advice on the StarCluster config file – AWS settings and iPython plugin options
• Then I need to figure out how to actually link iPython to AWS and start doing my analyses…

Useful Websites:

• iPython Notebook homepage
• StarCluster quickstart install guide
• Configuring StarCluster for iPython Notebook plugin
• Titus Brown’s video for linking iPython Notebook to Amazon AWS

The IPython Notebook makes it possible to display rich representations of Python objects using HTML (among other things). That allowed me to make a Python object whose representation in the Notebook is a colored table. Students can index into the table to change the color properties of individual table cells and then immediately display their changes.

With ipythonblocks instructors can give coding problems like ‘turn every block in the third column red’ or ‘turn every blue block green’ and by displaying their blocks students can see right away whether their code is having the desired effect.

Check out the demo notebook to see ipythonblocks in action.

My IPython Notebook

I love IPython Notebook. It is an interactive web environment for executing code. You can also save and share your notebook. Don’t have IPython? you can view notebooks on the web using the website http://nbviewer.ipython.org. I have a notebook for my Pandas Bar Chart Example. On the viewer site, enter the URL:

http://educationalfacilityplanning.com/MySamplePandasBarChart.ipynb

Or, click http://nbviewer.ipython.org/url/educationalfacilityplanning.com/MySamplePandasBarChart.ipynb

An article I’d seen in the news lately piqued my interest about the number of PhDs by US State. It took a goodly amount of fiddling before I figured out I could get the answer by submitting this query:

‘How many phds in California?’

Take a look at what this gives you: http://www.wolframalpha.com/input/?i=how+many+phds+in+california%3F

Lots of great information, including the specific answer I want, 331,101.

If I want to do this for all fifty states, I’m going to need to use the API not the interface. I registered for an API account which was quick and easy and checked out the bindings available on their site. The bindings left a lot to be desired, so I decided to just use urllib2.

Lately I can’t abide dealing directly with XML, so I found this nice library xmltodict. I’m sure [insert xml parsing technique] would work just as well or better.

As I was working I realized I also wanted to pull in total population, total adult population, etc. so I could talk about percentages.

I did this all in an IPython Notebook (if you’re not using this, you need to start, it’s totally awesome. Check out http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html).

Here is my notebook (created using Python 2.7): https://gist.github.com/4052828

Here is a rendered version of the notebook: http://nbviewer.ipython.org/4052828/

All in all it was a fun exercise but it still felt like page scraping. The one advantage is that you can ask the same question with slight modifications easily (e.g., how many people in California vs how many adults in Idaho) and get back essentially the same response so that’s handy. And although Wolfram Alpha returns a machine readable data structure (XML), it’s not exactly richly semantically tagged. There are plain text bits that have to be parsed. For example, sometimes a population will be expressed as its raw number, sometimes as 1.2 million. So I had to put special handling in my code for that case. It would be nice if such quantities were available with no plain-text parsing required.

One other adjustment that would be good would be to thread-out the calls to the API. It takes a good amount of time to process all of these serially and there’s no reason they couldn’t be threaded. I’ll definitely take a look at doing this.

If you’re just interested in results, here you go. This is the percentage of adults with PhDs and percentage of adults with at least an associates degree ranked by state from highest to lowest.

EDIT: After writing this, I found http://pypi.python.org/pypi/wolframalpha/1.0. This looks to be a nicer wrapper, I’ll have to take a look and see if it works as advertised. Let me know if you’ve used it successfully.

----------------------------------------------------------------------------------------------------
phds
#1: Washington DC: 3.68%
#2: Maryland: 2.32%
#3: Massachusetts: 2.30%
#4: New Mexico: 1.75%
#5: Vermont: 1.66%
#6: Connecticut: 1.60%
#7: Delaware: 1.56%
#8: Virginia: 1.54%
#9: California: 1.42%
#10: New Jersey: 1.41%
#11: Rhode Island: 1.39%
#12: Colorado: 1.37%
#13: Oregon: 1.35%
#14: Washington State: 1.34%
#15: Pennsylvania: 1.31%
#16: New York: 1.30%
#17: Hawaii: 1.25%
#18: New Hampshire: 1.24%
#19: Arizona: 1.18%
#20: Utah: 1.17%
#21: Minnesota: 1.16%
#22: Montana: 1.14%
#23: North Carolina: 1.13%
#24: Illinois: 1.13%
#25: Nebraska: 1.12%
#26: Maine: 1.12%
#27: Florida: 1.10%
#28: Alaska: 1.10%
#29: Kansas: 1.08%
#30: Tennessee: 1.05%
#31: Missouri: 1.05%
#32: Georgia: 1.04%
#33: Idaho: 1.02%
#34: Wyoming: 1.02%
#35: Iowa: 1.02%
#36: Wisconsin: 0.99%
#37: Michigan: 0.98%
#38: South Dakota: 0.97%
#39: Ohio: 0.97%
#40: Texas: 0.94%
#41: South Carolina: 0.93%
#42: Alabama: 0.92%
#43: Indiana: 0.91%
#44: North Dakota: 0.87%
#45: Mississippi: 0.82%
#46: Oklahoma: 0.81%
#47: Louisiana: 0.81%
#48: Kentucky: 0.80%
#49: Nevada: 0.78%
#50: Arkansas: 0.77%
#51: West Virginia: 0.77%
----------------------------------------------------------------------------------------------------
college graduates
#1: Washington DC: 50.47%
#2: Massachusetts: 48.28%
#3: Connecticut: 45.71%
#4: New Hampshire: 44.77%
#5: Colorado: 44.07%
#6: Vermont: 43.98%
#7: New Jersey: 43.76%
#8: Maryland: 43.53%
#9: Minnesota: 42.94%
#10: Hawaii: 41.90%
#11: Washington State: 41.71%
#12: Virginia: 41.58%
#13: New York: 40.38%
#14: Rhode Island: 39.72%
#15: North Dakota: 39.42%
#16: Maine: 39.06%
#17: Illinois: 39.05%
#18: Oregon: 39.04%
#19: Florida: 38.78%
#20: Nebraska: 38.50%
#21: Montana: 38.27%
#22: Kansas: 38.26%
#23: California: 38.13%
#24: Delaware: 37.30%
#25: South Dakota: 37.15%
#26: Iowa: 36.75%
#27: Wisconsin: 36.73%
#28: Pennsylvania: 36.66%
#29: Utah: 36.56%
#30: Arizona: 36.26%
#31: North Carolina: 35.96%
#32: Michigan: 34.98%
#33: Wyoming: 34.34%
#34: New Mexico: 34.09%
#35: Georgia: 33.89%
#36: South Carolina: 33.78%
#37: Idaho: 33.74%
#38: Ohio: 33.65%
#39: Missouri: 33.54%
#40: Alaska: 33.04%
#41: Texas: 31.96%
#42: Indiana: 31.04%
#43: Oklahoma: 30.72%
#44: Tennessee: 30.29%
#45: Alabama: 30.18%
#46: Nevada: 30.18%
#47: Kentucky: 28.44%
#48: Mississippi: 27.99%
#49: Arkansas: 26.74%
#50: Louisiana: 26.32%
#51: West Virginia: 25.49%

Unfortunately for this idea these testing frameworks have for the most part been engineered for doing their test collection using the file system as the starting point. In a couple hours of fiddling I couldn’t figure out how to use either nose or pytest to do test collection in a notebook. I’m sure it could be done with enough hacking.

Just for kicks, though, I threw together a little IPython line magic that does its own limited test collection, running, and reporting. You can see it via nbviewer and grab it on GitHub. This magic only grabs functions that begin with “test” and the reporting doesn’t include tracebacks when there are failures or errors. But you do get the exceptions themselves.

(This whole experiment was inspired by a conversation on Twitter.)

Update: On the advice of @stefanvdwalt I put the %runtests magic into a .py file so it can be installed via %install_ext:

%install_ext https://raw.github.com/gist/4013594/runtests.py

Once that’s done you can run

%load_ext runtests
%runtests

to have it do its limited collection, running, and reporting as in this demo notebook.

Update 2: Someone has made this work with nose: https://github.com/taavi/ipython_nose!

The code is available on GitHub.

For example, if you are at https://gist.github.com/3778422 you can click the “Open in nbviewer” extension button and it will load http://nbviewer.ipython.org/3778422/ in a new tab. It also works for URLs ending in .ipynb.

You can download the extension from the Chrome Web Store and see the code on GitHub.

I took the extension icon from the nbviewer favicon, so thanks to them for that! Thanks also for making something as awesome as nbviewer, it’s getting so I couldn’t live without it!

The IPython developers have to be given a huge amount of credit for putting together the Notebook and the rest of IPython. The Notebook especially is quite a feat: a top notch research/engineering/teaching tool all in one. And they aren’t resting on their laurels, they have a ton of ideas in mind for the Notebook in the future, including a slide-show mode. I’m definitely looking forward to seeing what they’ve got!

As with many open source projects, the IPython developers struggle to find the time and funding to write their software. If any open source project is helping with your job or your research you can easily help by citing the software in your papers and in public on social media or blogs. This gives the developers more ammunition the next time they’re writing grants, so please make your support known!

sudo pip install ipython -U
sudo pip install tornado
sudo apt-get install libzmq-dev
sudo apt-get install python2.7-dev
sudo pip install pyzmq

Execution:

ipython notebook

Update (20120418)
To learn more about IPython, check out this monstre tutorial (more than 3 hours long :))