In a previous post, I complained about the DTCC’s CDS data website and the one week lifespan of the data published there. For those of you who don’t know, the DTCC clears and settles a massive number of transactions every day for multiple asset classes. It’s one of those financial institutions that doesn’t get much press but underpins the entire capital market.

Anyway, the recent crisis motivated the DTCC to publish weekly CDS (single name, index, and tranche) exposure data. A good idea, until one realizes the data goes up in smoke when the next week’s data arrives. Although DTCC recently added links to data for “a week ago”, “a month ago”, and “a year ago,” it’s still pretty inconvenient. So, if you want the data, you have to parse it yourself. I originally wanted to write a smart parser that would dynamically react to whatever format it encountered…I came to my senses and adopted a simpler approach.

The approach thus far:

• Download the raw html pages/files via “curl.” Urllib2 is the preferred method to pull web pages, but I didn’t have the patience to figure out how to handle redirects. Curl is a utility included with OS X that, for whatever reason, ignores redirects automatically. As such, I created a short python script to download the html for all the tables of interest weekly.
• Use BeautifulSoup to parse the html. Other libraries, such as html5lib and lxml seem to be gaining ground on BeautifulSoup, particularly as it’s author wants to get out of the parsing game altogether. Nevertheless, I couldn’t be bothered to figure out the unicode issues I experienced with html5lib or lxml’s logic. BeautifulSoup is straightforward and “gives you unicode, dammit!” (quoting the author).
• Use numpy for easier data manipulation. Since my html, css, DOM, etc. knowledge is basic, I thought it might be better to use numpy to manipulate the table data rather than rely solely on the parser. This meant vectorizing the html data into a 1D array, cleaning it up, and generally preparing it for future reshaping. Numpy, how did I ever live without you?

This would’ve been much easier if all the tables were exactly the same format. Unfortunately, that’s never the case. An extra cell here or there, or weird characters, can throw things off. This isn’t an issue if you are parsing individual pieces of data or a single table. But what if you need to parse ten, 20, 100, etc. tables? It can get ugly fast. The DTCC data is broken into 23 pages, some of which have multiple tables. Luckily, most of my pain was self-inflicted (hey, I’m a parsing virgin). I only had to account for a few different table formats in the end.

One downside to my approach is I do not dynamically produce headers for the data I’m pulling. I plan to manually set the headers for each table (the ultimate destination for the data right now are csv files). If there’s a better way, please let me know.

You can find the code here via pastebin (feedback is welcome).
You can find the DTCC tables here (if you want to view the html source).

Part 2 will cover the process of reformatting the data with numpy and perhaps feature some charts. I’m very curious to see what the numbers show!

Here are a few screenshots of a terminal session using the code so far:

See and download the full gallery on posterous

In a previous post, I complained about the DTCC’s CDS data website and the one week lifespan of the data published there. For those of you who don’t know, the DTCC clears and settles a massive number of transactions every day for multiple asset classes. It’s one of those financial institutions that doesn’t get much press but underpins the entire capital market.

Anyway, the recent crisis motivated the DTCC to publish weekly CDS (single name, index, and tranche) exposure data. A good idea, until one realizes the data goes up in smoke when the next week’s data arrives. Although DTCC recently added links to data for “a week ago”, “a month ago”, and “a year ago,” it’s still pretty inconvenient. So, if you want the data, you have to parse it yourself. I originally wanted to write a smart parser that would dynamically react to whatever format it encountered…I came to my senses and adopted a simpler approach.

The approach thus far:

• Download the raw html pages/files via “curl.” Urllib2 is the preferred method to pull web pages, but I didn’t have the patience to figure out how to handle redirects. Curl is a utility included with OS X that, for whatever reason, ignores redirects automatically. As such, I created a short python script to download the html for all the tables of interest weekly.
• Use BeautifulSoup to parse the html. Other libraries, such as html5lib and lxml seem to be gaining ground on BeautifulSoup, particularly as it’s author wants to get out of the parsing game altogether. Nevertheless, I couldn’t be bothered to figure out the unicode issues I experienced with html5lib or lxml’s logic. BeautifulSoup is straightforward and “gives you unicode, dammit!” (quoting the author).
• Use numpy for easier data manipulation. Since my html, css, DOM, etc. knowledge is basic, I thought it might be better to use numpy to manipulate the table data rather than rely solely on the parser. This meant vectorizing the html data into a 1D array, cleaning it up, and generally preparing it for future reshaping. Numpy, how did I ever live without you?

This would’ve been much easier if all the tables were exactly the same format. Unfortunately, that’s never the case. An extra cell here or there, or weird characters, can throw things off. This isn’t an issue if you are parsing individual pieces of data or a single table. But what if you need to parse ten, 20, 100, etc. tables? It can get ugly fast. The DTCC data is broken into 23 pages, some of which have multiple tables. Luckily, most of my pain was self-inflicted (hey, I’m a parsing virgin). I only had to account for a few different table formats in the end.

One downside to my approach is I do not dynamically produce headers for the data I’m pulling. I plan to manually set the headers for each table (the ultimate destination for the data right now are csv files). If there’s a better way, please let me know.

You can find the code here via pastebin (feedback is welcome).
You can find the DTCC tables here (if you want to view the html source).

Part 2 will cover the process of reformatting the data with numpy and perhaps feature some charts. I’m very curious to see what the numbers show!

Here are a few screenshots of a terminal session using the code so far:

See and download the full gallery on posterous

No mês passado escrevi um artigo com um programa para capturar todos os items da primeira página de cada categoria do MercadoLivre.

Lá, lidava com alguns problemas como:

• limite de concorrência no download das páginas
• processamento de html em thread, síncrono
• manter a maior parte do processo assíncrono, para ganhar tempo e CPU

Depois disso, precisei fazer umas alterações no código e acabei modificando um pouco programa, usando outras técnicas como:

• cooperação de tarefas para limitar a concorrência
• processamento de html inline, mais rápido (pois usa backend em C)
• encadear algumas funções, executando inline

Cada item desta lista corresponde aos itens da lista mais acima, respectivamente.

O esquema de cooperação do twisted é muito melhor que o Controller que havia criado anteriormente. Porém, muito mais complicado para jovens aprendizes. Recomendo este link para mais detalhes.

Sobre o processamento do html, vale a pena verificar o lxml. Antes, havia usado BeautifulSoup, que é muito bom, mas perde violentamente em desempenho e suporte a broken-html.

Por fim, o truque de usar generators para executar alguns callbacks inline é incrível, e absurdamente prático em casos como esse, do programa abaixo.

O resultado é final é o mesmo, mas a melhoria em desempenho é absurda. Fiz alguns testes na minha máquina e obtive o seguinte:

• esta versão consome, em média, 20% menos de CPU
• como não há necessidade de gravar os arquivos no disco, não consome disco
• o processo todo ficou 657% mais rápido, simplesmente
• ainda, o código é muito menor +_+

Veja ai:

#!/usr/bin/env python
# coding: utf-8

from lxml import html
from twisted.web import client
from twisted.python import log
from twisted.internet import task, defer, reactor

class MercadoLivre:
    def __str__(self):
        return 'http://www.mercadolivre.com.br/jm/ml.allcategs.AllCategsServlet'

    def parse_categories(self, content):
        category = subcategory = ''
        doc = html.fromstring(content)
        for link in doc.iterlinks():
            el, attr, href, offset = link
            try: category = el.find_class('categ')[0].text_content()
            except: pass
            else: continue
            if category:
                try: subcategory = el.find_class('seglnk')[0].text_content()
                except: continue
                else: yield (href, category, subcategory)

    def parse_subcategory(self, content):
        doc = html.fromstring(content)
        for element in doc.find_class('col_titulo'):
            yield element[0].text_content()

class Engine:
    def finish(self, result):
        reactor.stop()

    @defer.inlineCallbacks
    def fetch_categories(self, link, parser):
        try:
            doc = yield client.getPage(link)
            defer.returnValue(parser(doc))
        except Exception, e:
            print e

    def fetch_subcategory(self, links, parser, limit):
        coop = task.Cooperator()
        work = (client.getPage(link[0]).addCallback(parser).addCallback(self.page_items, *link) for link in links)
        result = defer.DeferredList([coop.coiterate(work) for x in xrange(limit)])
        result.addCallback(self.finish)
        result.addErrback(log.err)

    def page_items(self, items, href, category, subcategory):
        print 'Categoria: %s / %s' % (category.encode('utf-8'), subcategory.encode('utf-8'))
        for item in items: print ' -> %s' % item.encode('utf-8')
        print ''

def main(limit, *parsers):
    e = Engine()
    for parser in parsers:
        links = e.fetch_categories(str(parser), parser.parse_categories)
        links.addCallback(e.fetch_subcategory, parser.parse_subcategory, limit)

if __name__ == '__main__':
    reactor.callWhenRunning(main, 150, MercadoLivre())
    reactor.run()

If you are a .NET programmer, then you find Python a bit tough. Reason? Python does not include library for each and every operation possible in this world. You may have to work around to find the necessary packages, download them and continue with your development.

Python’s standard module list has a finite number of entries as opposed to .NET    ( I use .NET at my workplace). Here this is an attempt to collect all such libraries which are outside standard modules, which you might badly need for your development. Many of them are extensions or wrapper packages for already existing libraries.

1 ) scapy

This is a library for  TCP/IP stack wherein you can have full control over the lowest detail of the Packet that leaves your computer. It supports many protocols like ETH, IP, ARP, ICMP, TCP, UDP etc. You can create custom TCP/IP packet and send it to any host. Typical implementation is ARP Ping, ICMP Ping.

Experience: Tried. Works perfectly. Havn’t stumbled across any bugs as of now.

2) soaplib

Used for creating lightweight web services. As the page says, it comes with a client and server built in and on-demand WSDL generation.

Experience: Havn’t tried. Heard about it’s existence.

3) mysql

Uh? Do I really have to tell what this is actually. I hope everyone knows.

Documentation for python-mysql

Experience: Obviously! Obviously! I think I should remove this line.

4) aubio

Stating directly from it’s site – “aubio is a library for audio labelling. Its features include segmenting a sound file before each of its attacks, performing pitch detection, tapping the beat and producing midi streams from live audio. The name aubio comes from ‘audio’ with a typo”

Experience: None. Presently in To-Do List.

5) Beautiful Soup

BeautifulSoup is an SGML parser which is highly robust and doesn’t die straight-off even if you give it poorly formed data. To make it scream and die all you have to do is to give something that isn’t SGML at all. It even has a parser class named BeautifulSOAP which is used to parse SOAP message (as the name applies). It even has a class named ICantBelieveItsBeautifulSoup. Sounds stupid? Who cares as long as it does it work.

Experience: Tried when I saw Anomit using it. Need more experience as I have lost touch as of now. Never tried BeautifulSOAP.

6) python-clamav

It is pending in my To-Do list. Will start working as I get time.  Check a small tutorial

Experience: No! Read the line above.

7) python-crypto

Presently in #1 position of To-Do list. Sounds just too promising. Hope it is as I thought it to be.

Check the API and it’s general overview

Experience: No

8 ) django

Now if you don’t know django – Go shoot yourself or read about it here if you somehow survive.

[ As pointed by Anomit, it isnt a framework, but library is a general name I have used for the title ]

Experience: Obviously!

9) gd

I have used GD a lot in PHP, but hardly on Python. GD is simpler than ImageMagik (never used) as people told me. Hope to use this library if I ever require.

If you ever require the documentation, head yourself to this page.

Experience: Not used in Python, but in PHP

10) gmp

GMP stands for GMP Multi Precision and gmpy is a python wrapper over it. Though you might not need it in Python, but if you are coming from C background, this might be a familiar name.

Experience: Normal, not an expert

11) python-jabber

Python-Jabber is a python module which implements jabber instant messaging protocol. Check out the documentation and a funny example .

Experience: Little experience. Not much. After all it doesn’t look so tough, so will sit down for a hacking session,

12) python-irclib

I encountered this library when I was searching more on python-jabber library. This also falls in the category of real-time messaging. The problem I can see is that there is no documentation. How to proceed? Use dir() and inspect module extensively?

Experience: Kidding? Please show me the documentation. I don’t have more time for hacks as I did with scapy.

–

Till now ,I had kept this list for my own reference. Many more required libraries are missing. If you have any more in mind, please mention it. I would be glad to add it.

A biblioteca pyQuery promete. Eu, viciado que sou por screen scrapping, achei que os meus problemas tinham acabado quando baixei a versão 0.3.1 hoje. A idéia da biblioteca é ser para o Python o que o jQuery é para o JavaScript, uma excelente mão na roda para manipulação de HTML (e XML).

O problema atual dela é que a parte que mais me interessa, traversing, não foi completamente implementada (preciso, a partir de um elemento, pular para o próximo, por exemplo). O timeline do projeto mostra que o desenvolvimento é esporádico. Por isso estou tentando utilizar uma biblioteca mais tradicional, a BeautifulSoup. Não está sendo fácil.

I finaly started coding for my CMS. I’m not sure if I’m already up to the job, but I’ll find out soon enough. The first library is part of the template engine. I don’t want people to write crap HTML, so I wrote a library that can generate strict XHTML with python code that is as similar to html as possible, see below for an example.

The big question remains, what to do with user content, even valid XHTML theme break if someone pastes his Word HTML in the editor. I plan on using BeautifulSoup to parse the user code, strip off the ‘invalid’ tags like <i>, <u>, <iframe>, etc… and feed it to my XHTMLLib. The even cooler part is that you could use your old HTML theme, feed it to my lib and fill/replace the content. With the latest version this became even easyer because I added some sort of xpath support(absolute paths only, but with attribute selection).

I present to you, my first lib!

Link: http://www.box.net/shared/6×01eol1×5 Also available in the sidebar widget. You need xhtmlattr.py as well!

Example:

page = XHTMLLib().template() # generate a basic XHTML template
    page.xpath('/html/head/title')['child'] = "Hello world" # set the title

    page.function_factory(['div', 'img', 'fieldset', 'input', 'ul', 'li'], '', '') # register functions

    lists = [li(class_='xtra', child="Home"), li(class_="test", child="Links"), li(child="Contact"), li(class_="test", child="About"), li(child="Sitemap")] # create a list
    lists.sort() # sort elements

    page.xpath('/html/body')['child'] = [ # add to the body
        div(id_='header', child='test & more'), # Entities get converted
        img(src='image.jpg'),
        fieldset(class_='test', child=[
            input(type_='text', name='name'),
           input(type_='submit')
        ]),
        ul(child=lists) # Comppose pages of different object sets.
   ]

    page.xpath('/html/body/img').__setitem__('onclick', 'alert("!@#$^, stop clicking!")', True) # Voodoo magic to get evil js in, alsays use external js!
    page.xpath('/html/body/ul/li[class=test][4]')['style'] = 'color:red' # Some more 'advanced xpath' to get a specific list item

    print page.render();

result:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
	<head>
		<meta content="application/xhtml+xml;charset=utf-8" http-equiv="Content-Type"></meta>
		<title>Hello world</title>
	</head>
	<body>
		<div id="header">test &amp; more</div>
		<img src="image.jpg" onclick="<![CDATA[alert("!@#$^, stop clicking!")]]>"></img>
		<fieldset class="test">
			<input type="text" name="name"></input>
			<input type="submit"></input>
		</fieldset>
		<ul>
			<li>Contact</li>
			<li>Sitemap</li>
			<li class="test">About</li>
			<li class="xtra">Home</li>
			<li style="color:red" class="test">Links</li>
		</ul>
	</body>
</html>

Hi all,

So I started parsing out some of the data from the GEO records for the automated prediction.  NCBI has both made my job slightly easier and immeasurably harder at the same time.

First the good, in 2006 NCBI adopted the MIAME (Minimal Information About a Microarray Experiment).  Shortly afterwards it released MiNiML, an XML based representation of the MIAME information.  These XML files are available for every GSM record.  With only a few lines of python code I can effortlessly extract the relevant feilds from the data.  Although I have noticed that I need to use BeautifulSoup to clean up the errant missing closing tag.

Now the bad, most of the feilds in the MIAME are “comment” fields.  These are free text explanations of the “Data Source”, “Data Description”, “Source Description”, etc.  I’m finding two main difficulties:

1. When the information in the field references information from the reference paper.
2. When the experimental procedures reference things OUTSIDE the scope of a “disease”.

I really don’t have a good automated solution for either of these problems.  For problem #1 it would require a method to extract the relationship from the paper itself … its my understanding is that this level of extraction is not possible with NLP.  Problem #2 is a little more expansive.  Some datasets study gene deletions, expression vectors, transformed cell-lines, etc.  I’m not quite even sure how to classify these into  an ontology myself, so I have no idea how to get the computer to do it.

Ulltimately I just need the computer to recognize when its come across these sorts of records and just ignore them.  However, this is a more difficult problem in machine learning then it seems.  I’ll probably need to make a specific set of tags for “unclassifiable” sentences.

Any thoughts?

-Will

Hi all,

Sadly nobody gets to program in a bubble.  Everyone has deadlines, legacy code, legacy computers, legacy programmers, etc.  We also don’t have unlimited time to “re-invent the wheel” at each opportunity.  Assumming you’re doing an open source project (much like this one), you can incorporate other people’s tools into your own code.

If I need to fix some automatically generated XML or HTML, instead of writing my own code, I can just import BeautifulSoup.  If I need to render information into HTML or an e-mail, instead of writing my own code, I can just use jinja templates.  If I need to make a GUI, instead of complaining about how people have forgotten (or never learned) how to use command line, I can just use wxPython.

I’m new to the development world.  I’m usually the only user of my code, and it only needs to run on my computer.  When I submit code for publication I do a cursory check to make sure it runs on another computer, but ultimately I leave it up to the readers to figure out the dependencies.  However, this project is different.  I need to ensure that the code runs on any computer with only a minor amount of fiddling … especially since this may be used by technically unsavy users.

After plenty of googling and some half-brained ideas to use command-line calls to easy_install I found that setuptools allows a programmer to specify the dependencies.  Whenever I used easy_install I noticed that it would install dependencies, I just didn’t make the obvious leap that I could also define them in my own setup file.

In the next post I’ll discuss some of my trials and tribulations on ensuring that all of my code is packaged correctly.

-Will

Aproveitando que meu time está em primeiro, aproveitei pra colocar mais um crawler com BautifulSoup, só que desta vez usando twisted.

A fonte de informação é o Globo Esporte, e pela leve bagunça que há no HTML e CSS, o código ficou meio feioso. Além disso, consegui deixar ainda mais feio pra poder imprimir bonito no terminal.

#!/usr/bin/env python
# coding: utf-8
# 20090127 AF - paulistão 2009, crawler

import re
from sys import stdout
from twisted.internet import reactor
from twisted.web.client import getPage
from BeautifulSoup import BeautifulSoup

crlft = re.compile(r'[\r\n\t]*')
URL = 'http://globoesporte.globo.com/Esportes/Futebol/Classificacao/0,,ESP0-9839,00.html'

def failure(err):
    print(err)
    reactor.stop()

def parser(contents):
    current = 1
    columns = 10

    f = lambda m: crlft.sub('', m.contents[0]+' ')
    soup = BeautifulSoup(contents, convertEntities='html')
    classes = ['borda borda-forte', 'borda semborda', ' borda-forte',
                ' semborda', 'time borda', 'borda', 'time ', '']

    stdout.write('% 18s' % 'TIME')
    for label in ['P', 'J', 'V', 'E', 'D', 'GP', 'GC', 'SG', '(%)']:
        stdout.write('% 5s' % label)
    stdout.write('\n')

    for item in soup.findAll('td', {'class':classes}):
        if item.find('span'):
            temp = '%02d. % 15s' % ((current / columns)+1,
                item.span.find('a') and f(item.span.a) or f(item.span))
            stdout.write(temp.encode('utf-8'))
        else:
            stdout.write('% 5s' % f(item))
            if not current % columns: stdout.write('\n')
        current += 1
    reactor.stop()

if __name__ == '__main__':
    deferred = getPage(URL)
    deferred.addCallback(parser)
    deferred.addErrback(failure)
    reactor.run()

E o resultado, no terminal:

$ python paulista.py
              TIME    P    J    V    E    D   GP   GC   SG  (%)
01.      Palmeiras    9    3    3    0    0    7    0    7  100
02.         Santos    6    2    2    0    0    4    1    3  100
03.    São Caetano    6    2    2    0    0    3    0    3  100
04.        Guarani    6    2    2    0    0    2    0    2  100
05.       Mirassol    4    2    1    1    0    5    3    2   66
06.      São Paulo    4    2    1    1    0    3    1    2   66
07.    Corinthians    4    2    1    1    0    3    2    1   66
08.    Ponte Preta    4    3    1    1    1    2    1    1   44
09.     Bragantino    3    2    1    0    1    4    3    1   50
10.       Paulista    3    2    1    0    1    2    2    0   50
11.    Santo André    3    3    1    0    2    1    2   -1   33
12.        Barueri    2    2    0    2    0    4    4    0   33
13.          Oeste    2    2    0    2    0    2    2    0   33
14.         Ituano    1    2    0    1    1    1    2   -1   16
15.    Botafogo-SP    1    2    0    1    1    4    6   -2   16
16.  Guaratinguetá    1    2    0    1    1    2    4   -2   16
17.        Marília    1    3    0    1    2    3    8   -5   11
18.       Noroeste    0    2    0    0    2    1    4   -3    0
19.     Portuguesa    0    2    0    0    2    0    3   -3    0
20.     Mogi Mirim    0    2    0    0    2    0    5   -5    0

Algum tempo atrás, o Urban Dictionary teve uma API para que outros sistemas e sites pudessem fazer busca nos termos de lá, que são muito legais.

Hoje, o link para a API não funciona mais, e virou um termo do dicionário. Procurando no google, encontrei um link que menciona uma API de acesso via SOAP com WSDL, mas também não é muito útil porque segundo o administrador de lá (15 de setembro) o formulário que emite as chaves de acesso está quebrado.

Em outras palavras: não existe mais API pro Urban Dictionary. Apesar a API de SOAP ainda funcionar, não é possível emitir novas chaves e por isso não é possível ter novos usuários com acesso a essa API. Não apenas eu, mas um outro cara gostaria de ter acesso ao Urban Dictionary através de uma API mais simples.

Enquanto eles não se manifestam, fiz um crawler usando BeautifulSoup, que resolve esse problema de um jeito meio tosco, mas pelo menos é simples e funciona.

#!/usr/bin/env python
# coding: utf-8
# 20080108 AF

import urllib
from BeautifulSoup import BeautifulSoup

def urbandict(search, limit=5, cutafter=256):
    query = urllib.urlencode(dict(term=search))
    url = 'http://www.urbandictionary.com/define.php?' + query
    response = urllib.urlopen(url)
    soup = BeautifulSoup(response.read(), convertEntities='html')

    fix = lambda item: item.replace('\r', '').replace('\n', '')
    cut = lambda item: len(item) > cutafter and item[:cutafter] + '(...)' or item
    extract = lambda item: isinstance(item, unicode) and item or \
        (item.name == 'a' and item.contents[0] or '')

    for item, count in zip(soup.findAll('div', attrs={'class':'definition'}), range(limit)):
        yield cut(' '.join([fix(extract(k)) for k in item.contents])).encode('utf-8')

for term in urbandict('stupid'):
    print term + '\n'

O limite padrão para a quantidade de respostas retornadas é 5, embora todas elas sejam capturadas. Graças à organização do site, isso é possível de forma simples pois os resultados são agrupados em DIVs cuja classe é “definition”.

Também, as seguintes funções modificam a exposição dos resultados:

• fix: remove CRLF, e faz com que cada resultado seja uma única linha
• cut: corta cada resultado após cutafter bytes, e adiciona (…)

Dessa maneira, fica fácil embutir os resultados do Urban Dictionary em qualquer lugar: um site, um programa no desktop, etc.

$ python ud.py
Someone who has to look up “stupid” in the dictionary because they don’t know what it means.

1) George W. Bush.  2) Karl Rove  3) Dick Cheney  4) You get the idea!

If you are reading this, you already know what BeautifulSoup is and how useful it is while working with XML/HTML in Python (in case you are not familiar with it, I’d encourage you to read its documentation). So I’ll just skip to the main reason of this post: a bug in parsing the <script> tags in HTML documents.

According to the documentation, BeautifulSoup knows how to handle the body of a <script> tag, meaning that it knows to treat its content as a pure string and not perform any additional parsing on it. Unfortunately, I’ve discovered a corner case where it behaves incorrectly.

Here is the sample HTML that will reveal the bug:

<html>
<head></head>
<body>
  <script type='text/javascript'>
    document.write('</script>');
    document.write('<div></div>');
  </script>
</body>
</html>

The problem is that the string ‘</script>’ tricks the parser to believe that the end of the <script> tag is reached and so instead of getting a single Tag from the <script> HTML tag it basically results in 2 elements: a Tag and a NavigableString that contains the rest of the <script> tag (i.e. what comes after the ‘</script>’ string: '); document.write('<div></div>');).

This basically means that for any HTML that contains a similar fragment rewriting it will lead to broken <script>s. Unfortunately, I haven’t been able to figure out a solution. My impression is that this parsing happens at a very low level and this makes me think that the bug might not be one of BeatifulSoup but rather a bug in SGMLParser.

The affected version is 3.0.7a. Meanwhile it looks like a new release has seen the light, but I haven’t tested it yet. The new BeautifulSoup 3.1.0 has replaced the SGMLParser with HTMLParser (in the attempt to make BeautifulSoup compatible with Python 3.0) so this bug might be already fixed.

If we are at bugs, I’d also like to mention one in Python 2.5.2 MacOS:

MemoryError
Python(72261) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Exception exceptions.MemoryError: MemoryError() in  ignored

Things are much simpler with this one, even if the displayed information doesn’t offer enough details. The above bug is basically the result of adding strings to a list in an infinite loop (so a programming problem, but with no indication of the error).

Python kennt verschiedene Module zum verarbeiten von XML Dokumenten:

-sax

-xml.dom.minidom

-Beautifulsoup

Sax ist sehr komplex und ebenso kompliziert, Beautifulsoup ist vor allem um normale HTML Dateien zu verarbeiten, besonders für APIs dagegen ist xml.dom.minidom geeignet, da es sich durch seine Einfachheit auszeichnet.

import urllib2
import xml.dom.minidom

#  Lade eine Seite als XML Objekt

def getXml(url):
url = url.replace(‘ ‘,’%20′)
return xml.dom.minidom.parseString(urllib2.urlopen(url).read())

# Lade Seite

doc = getXml(‘http://example.com/test.xml’)

titles = doc.getElementsByTagName(‘title’)

# Auf Inhalt eines bestimmten Elements zugreifen

myContent = doc.getElementById(‘content’).childNodes[0].data

#Auf alle Inhalte einer Liste von Tags zugreifen

divs = doc.getElementsByTagName(‘div’)

divContents = [div.childNode[0].data for div in divs]

Try it out for yourself at utilitymill.

import BeautifulSoup
import urllib

def html2text(node):
    if not hasattr(node, 'contents'):
        return node.replace('n', ' ')
    if node.isSelfClosing:
        return ' '
    return ''.join([html2text(x) for x in node.contents])

def lookup(username):
    html = urllib.urlopen("http://umn.edu/lookup?UID=" + username).read()
    soup = BeautifulSoup.BeautifulSoup(html)
    data = {}
    for heading in soup('th'):
        key = heading.contents[0][:-1]
        val = html2text(heading.findNext())
        data[key] = val
    return data