Прошлый пост говорил о решение проблемы с скачиванием картинок “в лоб”. Вдумчивое курение Danbooru API на пару с DartWantuz позволило найти более элегантное решение.
Мой вариант на коленке со старыми куками:

#! /bin/bash

uag="Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 (.NET CLR 3.5.30729)"postcount=`curl -b $2 "http://danbooru.donmai.us/post/index.xml?tags=$1&limit=1"|pcregrep -o 'posts\ count=\"[^"]+'|sed -e 's/posts\ count=//' -e 's/\"//'`

rm -f get2.danbooru.txt

let "pcount=postcount/1000+1"

for ((i=1; i<=$pcount; i++))do  wget "http://danbooru.donmai.us/post/index.xml?tags=$1&limit=1000&page=$i" --load-cookies="$2" -U "$uag" -O - |pcregrep -o -e 'file_url=[^ ]+'|sed -e 's/file_url=//g' -e 's/\"//g'  >>get2.danbooru.txtdone;

wget -nc -i get2.danbooru.txt

Запуск: %scriptname% тег файл_с_куками

Его вариант с логином:

#!/bin/bashTAGS=$(echo $* | sed -e 's/ /%20/g')# опять эти пробелы...LOGIN='login'PASSWORD='password'# сцука, настройкиAUTH=`curl -s -c danbooru.txt -F"commit=login" -F"url=" -F"user[name]=${LOGIN}" -F"user[password]=${PASSWORD}" \         http://danbooru.donmai.us/user/authenticate`# получаем кукисы с доставкой до danbooru.txtXML=`curl -b danbooru.txt -s "http://danbooru.donmai.us/post/index.xml?limit=1000&tags=${TAGS}&page=1"`TOTAL=$(echo $XML | egrep -o "[0-9]+" | head -n 4 | tail -n 1)# получаем кол-во страниц[ $TOTAL -le 1000 ] && TOTALPAGES=1 || TOTALPAGES=$(($TOTAL / 1000))# считаем уже не так хитропопо, в отличии от гельбуры

echo "TOTAL ${TOTAL} images"while [ $TOTALPAGES != 0 ]; docurl -b danbooru.txt -s "http://danbooru.donmai.us/post/index.xml?limit=1000&tags=${TAGS}&page=${TOTALPAGES}" \        | egrep -io "http:\/\/danbooru\.donmai\.us\/data\/[a-z0-9]+\.(jpg|png|gif|jpeg)" | uniq \        | xargs -d '\n' wget -P files# берём за рога, и качаем, и качаемlet 'TOTALPAGES--'donerm danbooru.txt# удаляем файл с куками, дабы не оставить следов

Запуск: %scriptname% список тегов

Если “интузязизьм” не угаснет, то будет универсальный скрипт для нескольких подобных галерей.

Having a bit of inspiration after a productive day at the office, I decided to implement this small feature that’s been on my mind for weeks: restriction maximum picture dimensions and/or size during the preload process.  There are two big reasons: 1.) FTP push and 2.) because I have a lot of super-high-quality image sources.  The Grace Park site is the first to take advantage of this, getting a maximum width of 400 (about 25% the width of some of the sources!).

PicMax is a variable specified in the site text at the site level, e.g. at the very top.  I wanted it to follow this syntax:

picmax=WIDTHxHEIGHT,SIZE

I also wanted each one of those components to be optional.  You can specify all, one, some, or none and achieve the effect you’re looking for.  The aforementioned fansite simply specifies “400″ which means “400 maximum width”.  However, what if you just want height?  Easy: “x400″ (400 maximum height).  You see the ‘x’ and ‘,’ are delimiters that need to be present in order to specify what’s after them.  So to cap all pictures at ten thousand bytes, you’d say “,10000″.  Maximum width and size but no height: “400,10000″.  How do I achieve such magic?

preg_match(‘/^([0-9]+)?x?([0-9]+)?,?([0-9]+)?$/i’, $picmax, $m))

Damn I love regular expressions!  Now, to retain your sanity you can specify a zero in the whatever spot you don’t want filled in and the code will operate the same.  When I first approached the parsing I wondered how I’d structure the pattern, but it turns out just making everything optional solved it.  Crazy!

When pictures are resized for scale, their maximum file size is capped at their previous file size.  This prevents the unintentional effect of shrinking one and having it grow on disk (the opposite desired effect).  Usually resizing dimensions is enough to squish the byte size, but it’s difficult to know what to set the JPG quality value to and re-saving any lossy format is an easy way towards bloat.

Всегда для того чтобы пропарсить большие строки использовал StringTokenizer, а сегодня первый раз столкнулся с классом Scanner (каюсь это моя отсталось ибо класс появился ещё в J2SE 5.0), но все же лучше поздно чем никогда. Отличие от StringTokenizer`а в том что тут разделители (delimeters)  задаются регулярными выражениями. Это очень удобно к тому же парситься могут не только строки но и потоки, файлы и много чего ещё. Примеры достаточно красочно описывающие возможности есть в javadoc`е указанном выше приведу кратенький пример:

String input = "<html><table border='1' cellspasing='0' cellpadding='0' width='300'>";
Scanner s = new Scanner(input);
s.findInLine("width='(\\d+)'>");
MatchResult result = s.match();
for (int i=1; i<=result.groupCount(); i++)
System.out.println(result.group(i));
s.close();


Вывод будет следующим:

300


http://www.rubular.com

This is my first post about my code so I thought I start with a fairly simple topic, pagination.  Pagination with Codeigniter’s pagination library is fairly straight forward but I have picked up a few tricks that I’d like to share.  This is however a bit more complicated than regular pagination because we will be paginating based on the first letter of a artist name in a database.  This example is based on the artists/catalog function and the source can be found here.  See example here

$this->load->library('pagination');

$letter = $this->uri->segment(3);
$offset = $this->uri->segment(4);

$data['letter'] = substr($letter, 0, 1);

if (!ereg('^[A-Za-z0-9]$', $data['letter'])) {
  redirect('artists/');
}

$this->load->model('ArtistModel');
$config['per_page'] = '100';
$data['results'] = $this->ArtistModel->loadByLetter($data['letter'], $config['per_page'], $offset);

$config['base_url'] = base_url() . '/artists/catalog/' . $data['letter'] . '/';
$config['uri_segment'] = 4;
$config['first_link'] = 'First';
$config['last_link'] = 'Last';
$config['num_links'] = 6;
$config['total_rows'] = $this->ArtistModel->getTotalForLetter($data['letter']);

$this->pagination->initialize($config);
$data['links'] = $this->pagination->create_links();

Pretty standard stuff if you follow the manual page except for a few things. First off, I take the ‘uri segment 3′ and I substr it to just one character. Therefore, if someone types in ‘abcderf’ I just get ‘a’.

On line 8-10 You see an ereg function. This function tests to make sure that $data['letter'] is a character or a number, and if not redirects to the basic artists page.  This could be done by passing the variable through the form validation class like I have done with searches but for 1 character it seems pretty silly to fire up a whole library to do what a simple ereg could accomplish.

After this point the $config variable is all pretty standard but the important part is the model functions because this is based on finding the first character of a name.

Here is the first model functions required

function loadByLetter($letter, $num, $offset)
 {
 if($letter == '0')
 {
 $this->db->where("artist REGEXP '^[0-9]'");
 $this->db->where('verified', 1);
 $this->db->order_by('artist', 'ASC');
 return $this->db->get('artists', $num, $offset);

 } else {
 $this->db->like('artist', $letter, 'after');
 $this->db->where('verified', 1);
 $this->db->order_by('artist', 'ASC');
 return $this->db->get('artists', $num, $offset);
 }
 }

As you can see we have the variables letter, num and offset which are all important in selecting data from the database. If the letter is the number zero (0), which is the default click options on the right side menu for artists beginning with numbers, it will look for all artists starting with a number. This is the case because we use $this->db->where(“artist REGEXP ‘^[0-9]‘”); to select the data from the database.  This regexp function will select all artists that begin with a number 0-9. We then only select those that are verified and arrange them by name.

If the $letter variable is not zero then we use a like function $this->db->like(‘artist’, $letter, ‘after’);. This will select all things that start with $letter, are verified and order them by artist name.  The ‘after’ portion of the like statement means that the wildcard (%) will only be placed after the $letter variable.  The after is crucial in making sure we only get artists that start with that letter.

The offset variable is used to tell the select statement how many to skip.

The other important model function we need is one that will tell us how many artists we have for that particular letter. The functions before are always using a limit of $num which for us was set to 100. We then need to use the following function to find the total.

function getTotalForLetter($letter)
 {
 $this->db->like('artist', $letter, 'after');
 $this->db->where('verified', 1);
 return $this->db->count_all_results('artists');
 }

This function will use the same like function as before but without the limit.

Then that’s it. Pass the $data['links'] variable to your view file and the pagination class takes care of the rest.

–Drew Town

Below is a really simple example on how to search for e-mail addresses within records of your database.

select * from table
where field_name  regexp ‘\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b’;

The regexp function similar to the ‘LIKE’ keyword.  The regular expression that follows regexp is used to search for any e-mail address variation.

CodeCopy:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;

namespace CodeCopy
{
    public class EmailValidator
    {
        public static bool IsValidEmail(string email)
        {
            if (string.IsNullOrEmpty(email))
                return false;

            string strRegex = @"^([a-zA-Z0-9_\-\.]+)@((\[[0-9]{1,3}" +
                  @"\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([a-zA-Z0-9\-]+\" +
                  @".)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$";
            Regex re = new Regex(strRegex);
            if (re.IsMatch(email.Trim()))
                return (true);
            else
                return (false);
        }
    }
}

Flex has formatter classes that can be used to format raw data into customized strings. You can combine these with data binding to be able to format multiple fields simultaneously. All formatters are subclasses of the Formatter class and built in versions include:

CurrencyFormatter, DateFormatter, NumberFormatter, PhoneFormatter, ZipCodeFormatter.

Each formatter has properties related to it’s type so a CurrencyFormatter has currencySymbol and precision properties and a PhoneFormatter has an areaCode property (area code to add to a 7 digit number, -1 means do not add and is the default). They all have pretty sensible defaults and more data can be found in the Adobe reference docs. To use a defined formatter, just call it’s format() method providing the value to be formatted.

All validators are a subclass of the Validator class. The following are provided validators but you may define your own:

CreditCardValidator, DateValidator, EmailValidator, NumberValidator, PhoneNumberValidator, SocialSecurityValidator, StringValidator, ZipCodeValidator.

All validators have a source attribute which is the id of the control that is being validated and defines where any error message will appear on failure. The property attribute defines where the actual information being validated is stored (ie if we’re validating a TextBox called myText, the source would be myText and the property would be text). If the validation fails, the control is highlighted in red and hovering over it produces an error message.

Like formatters, validators can have properties related to it’s type so a ZipCodeValidator has a domain property (which can be “US Only“, “US or Canada” or “Canada Only“) and a CreditCardValidator requires source and property attributes for cardType and cardNumber.

By default, the validator listens for the ValueCommit event on the source component which corresponds to when the user leaves that component. You can change this behavior or call the validate() method to force validation.

Often, just detecting the validation error is not enough, you want to prevent the user from proceeding until they fix the error. In this case, you need to listen for the ValidationResultEvent (calling validate() directly will return an object of this class). The event type property will either be VALID or INVALID.

Regular Expressions

Regular expressions can be used to help design custom validators. These are special strings that define a pattern a client string should match. You can find out more about defining regular expressions here.

The RegExp class allows you to define a regular expression string using it’s constructor via new which has a second optional parameter for flags (i=case insensitive, x=white space ignored etc). Alternatively you can define a literal expression directly for a RegExp variable. In this case you don’t need to escape the ‘/’ character (because you’re not providing it as a string) but you do need to put one at the start and end of the pattern – traditional regex style (any flags would go after the second /).

To execute the regex, use the search() method of the String object you are checking, passing the RegExp object you’ve created. This will return -1 if there is no match for the pattern (or the position in the string where the pattern begins).

You can build a custom validator by extending mx.Validators.Validator. Call the super() method from the constructor and override the protected method doValidation(value:Object):Array. The doValidation() method returns an Array of ValidationResult objects. Start by clearing your results Array then call the super class version of the method (super.doValidation()). If the input value object is not null, call it’s search() method providing your RegExp. If it’s not found, call the push() method on the results Array object to add a new ValidationResult object. This should specify the properties isError:Boolean – true if there’s an error, subField:String – the name of the subfield (if any) the error is associated with, errorCode:String – the error code for the problem (eg “notNumber”) and the errorMessage:String – a longer, more descriptive error message (eg “account ids cannot contain letters, they are 7 digits long”).

You can set the required property on a validator to true to ensure that if the field is left blank, it fails validation. You’ll need to call the validate() method directly for this at a point you know the user “should” have entered the data (eg on a submit button click).

For getting information about the caller issuing the logging request uses converters which based on usefull regular expression:

^\tat (?:(.+)::)*(\w+)\/*(.*)\(\)\[(?:(.+)\:(\d+))?\]$

This pattern based on a Jonathan Branams’s regular expression, but it’s modified for correct parsing calls from constructor and calls from classes in default package.

You can try it pattern and find many other usefull patterns on RegExr.

Regular expressions (”regex’s” for short) are sets of symbols and syntactic elements used to match patterns of text and they are pretty powerful. Regular expressions have been around for a very long time (in computer industry scale) and was first introduced as part of the powerful UNIX search tool grep.

The regex syntax used commonly today is compliant with Extended Regular Expressions (EREs) defined in IEEE POSIX 1003.2 (Section 2.8). EREs are supported by Apache, PHP4+, Javascript 1.3+, MS Visual Studio, MS Frontpage, most visual editors, vi, emac, the GNU family of tools (including grep, awk and sed) as well as many others. Extended Regular Expressions (EREs) will support Basic Regular Expressions (BREs are essentially a subset of EREs). The BRE syntax is considered obsolete and is only still around to preserve backward compatibility.

I believe mastering at least the most basic elements of regex is essential for any programmer. Further I know that having direct access to references, examples, ready to use patterns etc. is essential to speed up your work.

This is a toolbox for getting started and/or becoming more serious about regex. It provides details on commonly needed regexs that you can just pick up and use right away. Lets get started!

Read the rest of this entry »

http://www.tripwiremagazine.com/tutorials/tutorials/massive-regular-expressions-toolbox.html

]]> http://daills.wordpress.com/2009/07/30/regex-to-filter-isbn-10-and-isbn-13/ Thu, 30 Jul 2009 20:51:26 +0000 daill http://daills.wordpress.com/2009/07/30/regex-to-filter-isbn-10-and-isbn-13/

In case of a project of two friends of mine, i have been asked to write a regular expression pattern that matches ISBN 10 and ISBN 13 in different notation ways. So the result looks crappy but it works nearly perfect … here it is:

((?=(?:978|979)[0-9 ]{14})(?:\d{3}[ ]{1}\d{1,5}[ ]{1}\d{2,7}[ ]{1}\d{2,7}[ ]{1}\d{1})|(?=(?:978|979)[0-9-]{14})(?:\d{3}[-]{1}\d{1,5}[-]{1}\d{2,7}[-]{1}\d{2,7}[-]{1}\d{1})|(?=(?:978|979)\d{10})(?:\d{13})|(?=[0-9xX]{10})(?:(?:\d{9}[xX]{1}|\d{10}))|(?=[0-9-xX]{13})(?:\d{1}[-]{1}\d{3,5}[-]{1}\d{3,5}[-]{1}[0-9xX]{1})|(?=[0-9xX ]{13})(?:\d{1}[ ]{1}\d{3,5}[ ]{1}\d{3,5}[ ]{1}[0-9xX]{1}))

It matches:

• 1. 1234567891
• 2. 1234567891011
• 3. 3-446-19313-1
• 4. 3 446 19313 1
• 5. 978-3-86680-192-9
• 6. 978 3 86680 192 9

Notice: I’m far-off from being a regex guru so please don’t argue about the syntax

Vamos agora com um exemplo prático do uso de expressões regulares, uma pagina que vai aplicando uma máscara enquanto o usuário vai digitando:

Para o exemplo usaremos a formatação de telefone com a mascara (99) 9999-9999, e as expressões regulares:

(/\D/g) //Permite apenas numeros.

(/^(\d\d)(\d)/g,”($1) $2″) //Coloca “(   )”

(/(\d{4})(\d)/,”$1-$2″) //Coloca “-” após a seqüência de 4 caracteres.
O código HTML que chama a função da mascará para cada tecla pressionada ficará assim:

<html>
<head>
<title>MÁSCARA</title>
<script type="text/javascript">
/*Função Pai de Mascaras*/
function Mascara(o,f){
      v_obj=o
      v_fun=f
      setTimeout("execmascara()",1)
    }

    /*Função que Executa os objetos*/
    function execmascara(){
        v_obj.value=v_fun(v_obj.value)
    }

    /*Função que padroniza telefone (11) 4184-1241*/
    function Telefone(v){
        v=v.replace(/\D/g,"") //Retira todo valor que não for número.
        v=v.replace(/^(\d\d)(\d)/g,"($1) $2") //Coloca "(   )"
        v=v.replace(/(\d{4})(\d)/,"$1-$2") //Coloca "-"
        return v
    }
	</script>
	</head>
	<body>
	Telefone:
		<input type="text" maxlength="14" size="12" onkeypress="Mascara(this,Telefone)" />
	</body>
</html>

Понадобился мне тут список прокси в формате ip:port. Вбил в гугле запрос и пошел на первый попавшийся сайт. Вобщем он мне понравился, но вручную доставать оттуда всю нужную инфу не очень хотелось… Чтож.. Автоматизация рулит. Выбор пал на C#. Просто и удобно. Создаем проект оконного приложения, добавляем кнопочку и два textBox’а c MultiLine = true. Жмем на кнопку чтоб сгенерился код (при желании пишем вручную=)).
Ах, да! Чуть не забыл. В самом начале нужно дописать

using System.Text.RegularExpressions; //это для использовния регулярных выражений


дальше идет обработчик события кнопки


private void button1_Click(object sender, EventArgs e)
{
Regex rx = new Regex("[0-9]{1,3}[.][0-9]{1,3}[.][0-9]{1,3}[.][0-9]{1,3}"); //вот так извлекаем ип
Regex rPort = new Regex("[>][0-9]{1,4}[<]"); //а вот так порт
MatchCollection me; //коллекция для найденного
MatchCollection mPort; //коллекция для найденного
me = rx.Matches(textBox1.Text); //извлекаем
mPort = rPort.Matches(textBox1.Text); //извлекаем
int ind = 0;
string zz = null;
foreach (Match m in me) //перечисляем все ип-адреса
{
if (m.Value != zz) //это сравнение с предыдущим, чтоб повторений не было
{
textBox2.Text += (m.Value + ":"+

+ mPort[ind].Value.Substring(1, mPort[ind].Value.Length-2) +

+ Environment.NewLine); //и заносим в список все это дело
ind++;
}
zz = m.Value; //сохраняем значение ип, оно у нас будет предыдущим))
}
}


Вот что получилось:

Сразу говорю, что программа только для этого сайта!
И блин без отступов!! Как с отступами делать в этом вордпрессе???

CPF: Permite uma seqüência de três digitos, um ponto novamente três digitos um ponto seguido de três digitos, hífen e mais dois digitos
padrão 999.999.999-99

“d{3}.?d{3}.?d{3}-?d{2}”

Outros exemplos:

Data:
“^(([0-2]d|[3][0-1])/([0]d|[1][0-2])/[1-2][0-9]d{2})$”

IP:
“^(([1]?[0-9]{1,2}|2([0-4][0-9]|5[0-5]))\.){3}([1]?[0-9]{1,2}|2([0-4][0-9]|5[0-5]))$”

HORA:
“/^([0-1][0-9]|[2][0-3]):[0-5][0-9]$/”

SQL:
SELECT: “SELECT\s[\w\*\)\(\,\s]+\sFROM\s[\w]+”

INSERT: “INSERT\sINTO\s[\d\w]+[\s\w\d\)\(\,]*\sVALUES\s\([\d\w\'\,\)]+”

UPDATE: “UPDATE\s[\w]+\sSET\s[\w\,\'\=]+”

DELETE: “DELETE\sFROM\s[\d\w\'\=]+”

A consistência dos dados é muito importante para qualquer sistema, um dos meios mais utilizados para padronizar os dados aceitos é a expressão regular.

Mas o que são expressões regulares?  Para resumir, podemos dizer que:

“expressões  regulares são caracteres que juntos formam uma regra de entrada de dados. Os dados que obedecerem a todos os padrões estabelecidos serão validados.”

Podemos usar expressões regulares para tudo o que segue um padrão como: rg, cpf, cnpj, telefone, e-mail, data, cep,senhas e etc.

Varias linguagens já aceitam expressões regulares como: C, VB, JavaScript, Genexus, perl, Phyton e etc.

Abaixo temos uma pequena lista de padrões de expressão regular:

You can find basic information about How Regular expressions can be used in Siebel in below links,

http://siebeldev.blogspot.com/2009/04/regular-expression-validation.html

http://download.oracle.com/docs/cd/B40099_02/books/eScript/eScript_JSReference252.html

For the Non English character set support, RexExp pattern should be writed with Unicode characters.

You can get information about Unicode below link,

http://www.unicode.org/

You can find a simple pattern about Turkish First Name  validation below,

“^[\u0041-\u005A]{1}\u002E?[\u0041-\u005A|\u0061-\u007A|\u0130|\u0131|\u00D6|\u00F6|\u00DC|\u00FC|\u00C7|\u00E7|\u011E|\u011F|\u015E|\u015F]*$”

Pattern explanation:

First character should be Upper Case(\u0041-\u005A]{1})

Second character could be “.” (\u002E?)

Other characters can be anything in Turkish Alphabet([\u0041-\u005A|\u0061-\u007A|\u0130|\u0131|\u00D6|\u00F6|\u00DC|\u00FC|\u00C7|\u00E7|\u011E|\u011F|\u015E|\u015F])

Nell’articolo “Analisi quantitativa del testo – un progetto” illustravo l’idea di un software con cui categorizzare in automatico un documento rispetto ad una serie di documenti di riferimento. Il risultato sarebbe una serie di parole e frasi che riassumano gli argomenti trattati nel documento analizzato.
Qui comincio a descrivere un prototipo in PHP di questo progetto.

1. Il client e il server.

Innanzitutto l’analisi del testo si basa sul confronto di un documento con un gigantesco corpus di documenti più o meno correlati ad esso. L’utente del sistema esegue (fa eseguire) alcuni calcoli sul testo del proprio documento e ne confronta i risultati con quanto emerso dagli stessi calcoli eseguiti sul corpus. A questo punto le differenze assumono un significato (dalla quantità si passa alla qualità) e permettono l’identificazione di attributi con cui caratterizzare il documento analizzato. Fine.

Non si tratterebbe quindi di un motore di ricerca, dove lo spider va a cercare informazioni sulla risorsa, ma al contrario di un sistema in cui è la risorsa a descriversi ad un ente che restituisce alcune deduzioni.
“Come sono fatta, io, in confronto all’italiano scritto?“; “come sono fatta, io, in confronto alle altre novelle del mio stesso autore?“; “come sono fatta, io, in confronto a tutti i romanzi pubblicati in Sardegna in questi ultimi dieci anni?“.

Per prima cosa i documenti in analisi devono andare ad alimentare il corpus dei documenti di riferimento. Questo significa che se sto facendo analizzare un nuovo libro, i risultati dei calcoli vengono aggiunti a quelli precedentemente già memorizzati. In questo modo il corpus crescerebbe al diffondersi del servizio di categorizzazione: più testi vengono categorizzati, più testi finiscono nel corpus.
Sapendo in cosa consiste l’analisi statistica, poi, i client si possono permettere di inviare al servizio di categorizzazione solamente dati statistici semi-lavorati, ovvero non hanno bisogno di inviare tutto il testo né tutto il risultato dei conteggi. Il server conserva solamente dati statistici semi-lavorati.

Il processo si sta delineando quindi in questo modo:

• il server contiene conteggi relativi ad una serie di documenti già analizzati (e un’aggregazione di tutti questi conteggi: il famoso corpus)
• il client chiede al server se un certo documento è stato conteggiato di recente; in caso di conteggio aggiornato, il server lo confronta con i dati aggregati e restituisce al client le etichette
• in caso di conteggi non aggiornati (o di documento non ancora analizzato), il client (ri)esegue i suoi conteggi e li manda al server; questo fa il suo confronto e restituisce le etichette
• in entrambi i casi i conteggi del client vengono memorizzati anche sul server, così da alimentare il totalone.

La cosa difficile da ottenere è l’identificazione univoca di un documento: lo si può fare con una pagina online (un URL è sufficientemente univoco) o con un libro (sto pensando al codice IBAN), ma già diventa un problema con documenti prodotti dagli utenti (ad esempio).

2. La struttura dei dati.

Sia il client che il server devono contenere, per ogni documento analizzato sino a quel momento, i dati statistici calcolati e una serie di informazioni che ne permettano l’identificazione univoca. Il client si limiterà ai documenti che ha analizzato lui stesso, il server raccoglierà le informazioni di tutti i documenti.
Il server, in più, aggregherà questi dati riassumendo i conteggi in un’unica tabella (tutto l’italiano scritto) o in più tabelle temporanee (tutto l’italiano scritto di Ungaretti, per fare un esempio…).

Per conservare, calcolare e analizzare le parole ci verrà comodo utilizzare un DataBase, strumento perfetto quando si tratta di contare, gestire e manipolare grandi quantità di dati. Il DB che utilizzerò è MySQL, che tra le altre cose mi permette di eseguire query in sintassi regexp (cosa che potrebbe sempre tornare utile, trattandosi di parole) e di affidargli la responsabilità di stabilire se un dato debba essere aggiunto oppure aggiornato (mediante la sintassi ON DUPLICATE KEY UPDATE, di cui qui di seguito).

CREATE DATABASE `adt` /*!40100 DEFAULT CHARACTER SET utf8 */;

DROP TABLE IF EXISTS `adt`.`content`;
CREATE TABLE  `adt`.`content` (
  `id` varchar(50) NOT NULL,
  `author_id` int(10) unsigned NOT NULL,
  `mod_date` datetime NOT NULL,
  `check_date` datetime NOT NULL,
  PRIMARY KEY  (`id`),
  KEY `Dates` (`mod_date`,`check_date`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS `adt`.`x1`;
CREATE TABLE  `adt`.`x1` (
`content_id` varchar(50) NOT NULL,
`word1` varchar(50) NOT NULL,
`count` int(12) unsigned NOT NULL,
PRIMARY KEY USING BTREE (`content_id`,`word1`),
KEY `Sum` (`count`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS `adt`.`x2`;
CREATE TABLE  `adt`.`x2` (
`content_id` varchar(50) NOT NULL,
`word1` varchar(50) NOT NULL,
`word2` varchar(50) NOT NULL,
`count` int(12) unsigned NOT NULL,
PRIMARY KEY USING BTREE (`content_id`,`word1`,`word2`),
KEY `Sum` (`count`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS `adt`.`o3`;
CREATE TABLE  `adt`.`o3` (
`content_id` varchar(50) NOT NULL,
`word1` varchar(50) NOT NULL,
`word2` varchar(50) NOT NULL,
`word3` varchar(50) NOT NULL,
`count` int(12) unsigned NOT NULL
PRIMARY KEY USING BTREE (`content_id`,`word1`,`word2`,`word3`),
KEY `Sum` (`count`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

Le tabelle con il prefisso “x” contengono i risultati dei primi conteggi: ogni lemma (o forma), bigramma o trigramma appare una volta sola, già accompagnato dal numero di occorrenze; il numero successivo al prefisso “x” indica quante parole sono state considerate (n-gram).
Per far sì che un’istruzione INSERT non dia errore in occasione di un lemma già presente è opportuno utilizzare la sintassi ON DUPLICATE KEY UPDATE, come in questo esempio:

INSERT INTO x1 (content_id,word1,count) VALUES ($id,$word,1) ON DUPLICATE KEY UPDATE count=count+1;

In questo modo con MySQL è possibile evitare di controllare mediante una SELECT se un lemma è già stato inserito e poi conseguentemente lanciare una INSERT oppure una UPDATE.

La tabella “content” contiene tutti i dati relativi al testo originale; per i raggruppamenti di testi mi devo ancora attrezzare (potrei ipotizzare ad esempio una riga relativa al contenuto “ALL”, raggruppamento obbligatorio e presente di default: ma ci devo ancora pensare).

3. Leggiamo un testo.

Innanzitutto è necessario caricare in memoria il testo da analizzare. Non si tratta, per ora, di realizzare un vero e proprio parser, bensì di preparare la struttura che ce lo renderà possibile:

<script language="php">

$dblink = mysql_connect( 'localhost', 'USERNAME', 'PASSWORD' );
mysql_select_db( 'DBNAME', $dblink );
if( ini_get( 'max_execution_time' ) )
{
	$time_out = ini_get( 'max_execution_time' );
} else {
	$time_out = 30;
}
ini_set( 'max_execution_time', 300 );

$fp = fopen( "TEXT_TO_PARSE.txt", "r" );
$content_id = 1;

$lines = array();
$current = 0;
while( $line = fgets( $fp ) )
{
	if( !ctype_space( $line[0] ) )
	{
		$current++;
		$lines[$current] = utf8_decode( trim( $line ) );
	}
}
fclose( $fp );

ini_set('max_execution_time', $time_out);
</script>

Apriamo in lettura un file (che deve essere UTF-8), creaiamo l’array $lines, e per ogni linea non vuota del file riempiamo una cella dell’array. In questo modo l’array $lines conterrà tutte le righe che compongono il testo da analizzare. Niente di difficile, sino a qui.

4. Salviamo le parole.

Per ora non stiamo ancora analizzando nulla: prima dobbiamo salvare le parole sulle apposite tabelle (la tabella “x” per le parole singole, la tabella “x” per le coppie di parole, la tabella “x” per le sequenze di tre parole, e così di seguito: quindi “x1″, “x2″, “x3″ eccetera).
Dopo, quando cominceremo a contare le occorrenze, avremo bisogno di sapere, per ogni parola, a quali comunità questa appartiene:

• la lingua del testo in cui è apparsa
• il genere letterario in cui è scritto il testo da cui proviene
• gli argomenti che il testo da cui proviene affronta
• la rete di citazioni in cui il testo da cui proviene si immerge
• il testo da cui proviene

In questo modo ci sarà possibile fare analisi che scendano al livello di profondità che più ci interessa: sarà ad esempio possibile estrarre il lessico di frequenza di tutti i testi di argomento biblico oppure tutte le locuzioni più frequenti del testo X, eccetera.
Per creare un tabella che contenga tutte quelle informazioni di dettaglio (“content”) sarà sufficiente fare riferimento al testo dal quale le parole provvengono: è il testo, difatti, che può essere caratterizzato da un genere, da alcuni argomenti, da uno o più autori, da una data di edizione… Sarà il testo a fare da crocevia per tutti i tipi di relazioni che potremmo voler stabilire tra le parole.

E quindi ora, nel momento di salvare le parole su di un DB, sarà sufficiente associarle al testo dal quale provvengono. Per il momento stabilisco che l’identificativo del testo sia arbitrariamente il numero 1.

5. Prima le parole singole…

Il codice PHP che scrive su “x1″ tutte le parole di un testo è il seguente:

<script language="php">
// @TODO: L'attuale check di eventuale presenza di questo content_id sul DB
// si basa su di un id che deve essere univoco!!
if( !mysql_fetch_array( mysql_query( "SELECT id FROM x1 WHERE id = '$content_id' limit 1" ) ) )
// @TODO: identificare i fine frase per permettere le analisi a livello
// più dettagliato che il testo. L'idea è di estrarre un array direttamente
// da tutto il testo originale scindendolo ogni volta che appare un punto:
// $fulltext = preg_replace( '/[\.\n]+/', '.', $fulltext );
// $lines = explode( ".", $fulltext );
{
	foreach( $lines as $key => $value)
	{
		$sql = "INSERT INTO x1 VALUES ";
		$value = preg_replace( '/[^\w]+/', ' ', $value );
		$value = preg_replace( '/[\s]+/', ' ', $value );
		$value = preg_replace( '/^[ ]+$/', '', $value );
		if ($value)
		{
			$line_arr = explode( " ", $value );
			foreach( $line_arr as $key2 => $item )
			{
				$sql .= "($content_id,'$item',1),";
			}
		}
		$sql = substr( $sql, 0, -1 );
		$sql .= "  ON DUPLICATE KEY UPDATE count=count+1;";
		mysql_query( $sql );
	}
	echo "EOF<br>";
} else {
	echo "Content already present.<br>";
}
</script>

Mi connetto al DB, prendo ogni cella dell’array $lines (per ora ipotizzo semplicemente di averlo: poi vedrò come passarmelo per davvero), accorpo tutte le diverse spaziature in uno spazio solo, creo un array che contenga tutte le parole e aggiungo le informazioni delle singole parole (ovvero id del testo e parola) alla query SQL che infine eseguo (per velocizzare l’esecuzione della query ne compongo una sola per tutta una riga).
Stampo a monitor un “End of File” che rassicuri l’utente.

6. …poi tutte le altre.

Il codice PHP che scrive su “x2″, “x3″, “x4″, “x5″ e “x6″ tutte le parole di un testo è il seguente:

if ($key-5 >= 0) {
 mysql_query("INSERT INTO adt.x6 VALUES ('0', '".$line_arr[$key-5]."',
    '".$line_arr[$key-4]."', '".$line_arr[$key-3]."', '".$line_arr[$key-2]."',
    '".$line_arr[$key-1]."', '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x5 VALUES ('0', '".$line_arr[$key-4]."',
    '".$line_arr[$key-3]."', '".$line_arr[$key-2]."', '".$line_arr[$key-1]."',
    '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x4 VALUES ('0', '".$line_arr[$key-3]."',
    '".$line_arr[$key-2]."', '".$line_arr[$key-1]."', '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x3 VALUES ('0', '".$line_arr[$key-2]."',
    '".$line_arr[$key-1]."', '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x2 VALUES ('0', '".$line_arr[$key-1]."',
    '".$line_arr[$key]."')");
 } 

 elseif ($key-4 == 0) {
 mysql_query("INSERT INTO adt.x5 VALUES ('0', '".$line_arr[$key-4]."',
    '".$line_arr[$key-3]."', '".$line_arr[$key-2]."', '".$line_arr[$key-1]."',
    '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x4 VALUES ('0', '".$line_arr[$key-3]."',
    '".$line_arr[$key-2]."', '".$line_arr[$key-1]."', '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x3 VALUES ('0', '".$line_arr[$key-2]."',
    '".$line_arr[$key-1]."', '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x2 VALUES ('0', '".$line_arr[$key-1]."',
    '".$line_arr[$key]."')");
 }

 elseif ($key-3 == 0) {
 mysql_query("INSERT INTO adt.x4 VALUES ('0', '".$line_arr[$key-3]."',
    '".$line_arr[$key-2]."', '".$line_arr[$key-1]."', '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x3 VALUES ('0', '".$line_arr[$key-2]."',
    '".$line_arr[$key-1]."', '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x2 VALUES ('0', '".$line_arr[$key-1]."',
    '".$line_arr[$key]."')");
 }

 elseif ($key-2 == 0) {
 mysql_query("INSERT INTO adt.x3 VALUES ('0', '".$line_arr[$key-2]."',
    '".$line_arr[$key-1]."', '".$line_arr[$key]."')");
 mysql_query("INSERT INTO adt.x2 VALUES ('0', '".$line_arr[$key-1]."',
    '".$line_arr[$key]."')");
 }

 elseif ($key-1 == 0) {
 mysql_query("INSERT INTO adt.x2 VALUES ('0', '".$line_arr[$key-1]."',
    '".$line_arr[$key]."')");
 }

Ok, mi rendo conto che questa parte del giocattolo è perfettibile (tanto per usare un eufemismo); ad ogni modo il risultato è tanti begli n-gram.

Al prossimo appuntamento con questo progetto comincerò a verificare se davvero il confronto tra i dati statistici di un testo e del suo corpus di riferimento fa emergere informazioni qualitative interessanti.

Grazie a chi è riuscito a seguirmi fino a qui…

  `mod_date` datetime NOT NULL,

I have found this regular expression to be usefull

// Regex for parsing URIs.
var HTTP_URI_RE = /^(https?:)\/\/([\w-_.]+(:[0-9]+)?)(.*)$/;
var FILE_URI_RE = /^file:\/\/(.*)$/;
var PATH_RE     = /^(\/?[\w\-_\.\/\%\*]*)(\#([\w\-_\.\/\%\*]*))?$/;
var FILE_RE     = /^(.*\/)*([^.]+\.[\w]+)$/;
var SEARCH_RE   = /^([^&=]+)(=([^&]*))?(&(.*))?/;


preg_match_all("#(\w+)=\[(.*?)\]#s", $param_line, $matches);

sample string = {gallery width=[129] url=[http://google.com] height=[122]}

Oh – the possibilities!
sed, grep and a lot of editors support the use of regexp. It’s about time I learned it!

Function rgxEmail(email)
Dim regEx, Match, Matches
If email <> “” Then
Set regEx = New RegExp
regEx.Pattern = “^([a-zA-Z0-9_\-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([a-zA-Z0-9\-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$”
regEx.IgnoreCase = True

If regEx.Test(email) Then cckEmail = True Else cckEmail = False
End If
End Function

Java introduced the java.util.regex package in version 1.4. It is a powerful addition, and yet, one should really be an artist in order to use it right. Assuming that a regular expression is proved for correctness, it may still run extremely slow (even take hours) if it is not wisely written.

Continue reading for understanding the origin of the problem, or jump directly to the last section, containing 10 useful performance tips for regular expression writers in Java.

Can it really be so slow?

 Suppose that we want to match only strings composed of sequences of ‘a’s or ‘b’s. A legal regex would be:

(a*b*)*

However, if you run this regex against the string “aaaaaaaaaaaaaaaaaaaaaaaaaaaaax” for example, it takes several minutes to terminate and report that there is no match!
Of course, a better regex in this case would be:

(a|b)*

This one takes less than a millisecond on my machine, on the same input. Obviously, there is a performance issue here.

Why does this happen?

 Like most regex engines, Java uses NFA (Non deterministic Finite Automaton) approach. The engine scans the regex components one by one, and progresses on the input string accordingly. However, it will go back in order to explore matching alternatives when a “dead end” is found. Alternatives result from regex structures such as quantifiers (*, +, ?) and alternation (e.g. a|b|c|d). This exploration technique is called backtracking.
In the catastrophic example above, the engine will actually explore ALL the decompositions of the series of ‘a’s into smaller series, before it realizes that there is no match. This example shows how the backtracking algorithm can result in exponential time evaluation (in the length of the input sting). It also reveals an important property of NFA engines: the worst cases will always be inputs that almost match the pattern. If a match is found, the exploration stops.

The other main approach for regex engines is DFA (Deterministic Final Automata). In this approach, the regex compilation actually builds an automaton, to be used when traversing input strings. Inputs are traversed character by character, with no going back. This guarantees linear timein the length of the input string, regardless of the regex complexity. Instead of trying different match possibilities serially (as in NFA), DFAsimulates a parallel scanning on all possibilities.

So why does Java (and .NET, Perl, Python, Ruby, PHP etc) use NFA and not DFA, which has much better asymptotic behavior? The reason is that NFA has some significant benefits:

• Its compilation is faster, and requires much less memory
• It allows some powerful features (See Sun’s tutorial for details on them):
  • Capturing groups and back references
  • Lookarounds
  • Advanced quantifiers (Possessive and Reluctant)

It is important to note that the popular terms NFA and DFA for regex engines are inaccurate. Theoretically speaking, these 2 models have the same computation power, meaning that there can’t be a matching rule that can be expressed in one of them but not in the other. In practice, there was a need for more features, so the two implementation types diverged in their semantics. NFA engines were given more power, making them superior to DFA engines in computation power.

Due to the speed of DFA and the unique features of NFA, there are 2 more “integrative” approaches for regex implementations. Some implementations use both engines (e.g. GNU egrep, which chooses the specific engine at runtime), and some of them really implement a hybrid version (e.g. Tcl regex), enjoying all the benefits.

Tips

 Following are some tips on how to avoid regex efficiency issues in Java. Many of them are aimed at reducing backtracking.

 1)  Pre-compile
Trivial, but worth mentioning. If you use a regular expression more than once, remember to compile it first as a pre-process step:
Pattern p = Pattern.compile(regex,flags);
…

//Usage stage
Matcher a = p.matcher(input);

 2)  Reluctant quantifiers vs greedy quantifiers
The default quantifiers (* + and ?) are greedy. That means that they start by matching the longest possible sequence, and then they step back gradually if backtracking is needed. In case that you know that the match is usually short, you should use reluctant quantifiers instead. They start by the smallest match, and progress if needed.
For example, suppose that you want to match only strings containing the sequence “hello”. The regex .*hello.* will do the job, but if you know that ‘hello’ usually appears near the beginning of the text, then .*?hello.* will run faster on average.

 3)  Use possessive quantifiers where possible
Unlike reluctant quantifiers which affect performance but do not affect regex behavior, possessive quantifiers may really change the meaning of the regex.
When using *+ instead of *, the first attempted match will be greedy (i.e. longest match, just as with *), but there will be no backtracking in case of failure, even if this causes the complete match to fail. When does this become useful?
Suppose that you need to match text in quotes. The regex \”[^\”]*\”will work fine. However, it will do unnecessary backtracking on negative cases (e.g. “bla bla bla). Using the regex \”[^\”]*+\”instead will eliminate the backtracking, without changing the regex meaning.

Independent grouping can have the same effect, and it allows even more control (See Sun’s tutorial ).

4)  Avoid capturing groups
Any expression in parentheses is considered a capturing group by default. This has a small performance impact. Make your groups “non-capturing groups” when possible, by beginning them with (?: instead of with (.

5)  Use alternation wisely
When using alternation (e.g. Paul|Jane|Chris), the order in which the engine tries to match the options is the same order in which they appear. Therefore, you can take advantage of this property, and order your options from the most common one to the less common one. This will improve the average time of positive matches.

 6)  Avoid multiple interpretations
Write your regex in a way that minimizes the number of different ways to match a particular input string. For example, the regex (a*b*)*given in the beginning of this article, allows interpreting the string “aabb” in too many ways:
(a2b2)
(a1)(a1)(b1)(b1)
(a2)(b2)
(a1)(a1b2)
etc…

The regex (a|b)* on the other hand, forces a unique interpretation on a positive input.
This is very important in order to reduce backtracking, in cases of almost a match.

 7)  Lookarounds
Lookarounds  allow adding restrictions on sequences to the left/right of current position. Specifically, with negative lookahead you can search strings that do not contain some sequence (cumbersome thing to do without this feature!). How can this help improve performance?

Suppose you want to capture the URL from a link tag. Consider the following regex:
<a .* href=(\S*).*/>
For legal tags it will find a match and capture only the href attribute as required (\S stands for a non-whitespace character) .  On some illegal tags, however, exessive backtracking will occur. For example: “<a href= href=href=…. href=something”.
The following regex will prevent this, by replacing the “.*” expression with something more specific, that does not match “href’”:
<a ((?!href).)* href=(\S*)((?!href).)*/>

 8)  Specify lengths
Java has a regex optimizer that checks the input string’s length against the min/max length derived from the regex contents. This allows failing the match in some cases immediately. In order to help this mechanism, specify repetition number on groups whenever possible (e.g. [01]{6}, which matches all binary strings of length 6).

 9)  Extract mandatory strings
Sometimes, strings which are mandatory are hidden inside groups or alternations:
(hello|hell|heel)
This expression could be simplified to:
he(llo|ll|el)
By doing so, the regex optimizer has more information to work with.

 10)  Benchmark your regex
When the regex is used in a performance critical area of your application, it would be wise to test it first. Write a micro-benchmark application that tests it against different inputs. Remember to test different lengths of inputs, and also inputs that almost match your pattern.

Links: