When I asked Gino (alias Roberto Baldi mostmobbed) a software solution for the “halting problem”, he told me “should not be so difficult”!

“In computability theory, the halting problem is a decision problem which can be stated as follows: given a description of a program and a finite input, decide whether the program finishes running or will run forever, given that input.” (from Wikipedia)

Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist; but as Gino says, Turing was a pessimist…

Why I’m scratching about the halting problem? Because I need power at BugsVoice user’s fingertips. In particular, I need to allow the execution of user’s JavaScript code on my server, and knowing if these scripts end may be comforting.

If you already heard about XSS  you probably know that “third party” code execution, authorized or not, could be a nightmare even in the client… can you imagine how can be ugly on your server a “pirate” code?

Going more in detail, BugsVoice is a service that receives a “bug” from a customer’s server, process the request locally, serves a friendly feedback to the user and stores the bug in its database.

JavaScript (JS from here on) server-side execution is involved in the “processing” phase.

We supply a pre-filled and certified set of “rules” for processing bugs, but we even allow customers to create their own rules.

JS gives you the power of inspecting the error to understand what happened and gives your customer an error better than a “500 server error” in order to comfort it and recover a situation where your application credibility is going down. An interesting reading about error recovering and error feedback is  the book “Defensive design for the web” from 37 Signal.

The complete BugsVoice process includes mainly three parts:

1) on the customer’s server side,  an error page that catches the exception, collects as much information as possible (logged user, time, server status, database status, memory etc.) and redirects the user to our BugsVoice server (see how to configure an error trapping page on BugsVoice blog for more details).

2) our server, reading user preferences recovers the error template. Each template is fully dynamical and customizable; it introduces some “variables” that can be filled from the error happened. Then our server creates two JS objects: the “bug” object filled with the error collected and the “template” object filled from layout skeleton.

3) the JS rules are executed to fill “template” from “bug” or for rejecting the request.

4) the layout is rendered to the user by  using “template” and “bug” objects. The bug is stored on our server.

5) the user feedback is collected and stored.

6) a “thank you” page is displayed to the user.

Then there is the error management but this is interesting “only” for BugsVoice’ users, not for this post. Here some error pages from BugsVoice:

So every user can create its own rules in order to inspect, for instance, the received bug’ stacktrace trying to discover if a database problem happens, or if there is a problem with the latest version of  some browser.

Coming back to rules execution:
during step 3) we get rules from the user configuration and we execute them on our server. We use the Java SE 6 scripting features supplying an ECMAScript engine to run rules.  A scripting engine instance is isolated from the JVM environment and you must declare the resource (libraries) you want to made available in the execution context.

Before executing them, the context is fed by “bug” and “template” objects. Then we run the rules…(drum roll!).

A basic (and friendly) rule example :

if (bug.code==404)  errorPage.errorMessage="Page missing: you get this error because of...";

Of course this code is safe, but what happens if an evil user composes a pleasant rule like

while(true);

or

function snake(s){
  return "s"+snake(s);
}
snake(":-<");

… or even worst?

Sadly Turing beats Gino 1-0, and there is no general solution to the question “does this rule ends?”.

The only possible solution is to narrow the scope of the problem by introducing some fences.

A solution is to set up an external observer using  multi-threading and watch dogs in order to kill processes after a while, but the best solution is to avoid infinite loop situations.

Rules in our context are used mainly for discovering string patterns in the error stacktrace and for building better feedback; we do not need to iterate or create complex functions, so reducing the set of possible JS statement is possible without loosing “power”.

Luckily in JS there is a limited set of statements for iteration and recursion; so if we are able to “kill” bad intentions by forbidding dangerous statement like “while”, “for” or function definition we can run rules with confidence.

This way  we reduce the complex halting problem to the (quite) easy problem of  HTML sanitization  (where you must  remove some unaccepted tags. See XSS war: a Java HTML sanitizer ).

Actually identifying “while” or “for” statements in a complex code is not as easy as finding the “while” string. The find/replace approach  it’s too rough, and here we need a more accurate solution in order to understand the difference between

while (true);

and

var dummy= “while(true)”;

that is obvious for us but not for a string searcher…

You must use something to analyze the code token by token.

ANTLR 3 supply all you need for tokenizing, parsing and walking your code. You need a JS grammar and then ANTLR will build all the stuff. We used the ES3 grammar from Xebic Reasearch  (BSD license) based on the original work of Patrick Hulsmeyer, that fits perfectly our needs.

With the AS3 grammar we built  parser, lexer and walker to analyze rule’s code to intercept every forbidden statement and avoid accepting dangerous scripts (at least I hope this). Only the rules that pass the test will be saved on the system and will be available to the script engine.

Ok, I can confess you, the post’ title is a little misleading, there is no way to solve the halting problem at least without cheating!

Preface

November, 4th. Release Candidate 1 of Database Designer for PostgreSQL 1.2.9 become available. Among three changes comparing to the last beta there is the one which attracts attention — “Execute Script In Single Transaction (Alt + F9)” functionality added. World community shocked.

“What means added? We thought it always was executed in single transaction…” – resounded from all sides.

November, 14th. MicroOLAP Headquarters. Explanatory mission entrusted to the best agent… Me.

pgAdmin shows the last result set

Right now there is no any opportunity to remember who got the idea about script slicing in SQL Executor. The gist was — each returned result set must be displayed.

Have a look how pgAdmin handles multiple result sets. As you can see only the last is available while others are discarded (we may read about this on the Messages tab).

One more notice. Multiple statements in pgAdmin always executed in the single transaction context. This is not a miracle since PQsendQuery function from client library used.

By the way, the fact that PQsendQuery used give us a hope that someday pgAdmin will handle all result sets.

PgMDD shows all result sets in separate tabs

Just for note, I’m not saying pgAdmin is a dinosaur or something. I like it a lot. This is “must have” tool for sure. I’m using it because of other GUI administration utility absence.

As you probably guessed PgMDD creates separate tab for each result set from the very first release.

There is one more important issue why script slicing was implemented. Database Designer is some kind of ideal world. You may use any names, any functions, any data types for model creating.

But real life is cruel. Generated script must work in any conditions even if some statements may fail, e.g. old server version, non-existent role, lack of privileges for some operations, object with the same name already exists etc. That’s why PgMDD’s SQL Executor should give the developer right of choice — abort execution or proceed anyway.

How it’s made

Let me one phrase before I begin: there is no any SQL parser library (or suite) on the market which suits even the basic needs (I mean PostgreSQL dialect of course). I guarantee this!

God is my witness, our team tried every 3rd party library we meet. Without success.

In Russian speaking IT folklore there is an adage “Переписать всё нахрен!”. Loose translation is “Rewrite all from scratch!”

OK, the moment of glory. We made it ourselves using the native PostgreSQL grammar. Yeah, bite me unbelievers!

Let’s omit technical details. I know nobody cares anyway. Our parser is absolutely… no, I mean absolutely compatible with PostgreSQL. But with 8.3.x version. It’s just a matter of time to update it, but we missed the moment.

So we have two reasons to add “Execute in Single Transaction” functionality:

• Ability to ROLLBACK all changes made by script in case of need
• PgMDD parser cannot proceed with some PostgreSQL 8.4.x features

That’s all folks!

So I have been playing with VPRML (Vorper Markup Language) and it’s parser, and I think I have a good start on how its going to be structured.

When the user wants to write a post/page he would like to write without any code as if it was a word processor,but all the webpages need to be stored into html format.To convert normal text and images into HTML code directly a javascript program called WYSIWYG editor is used which parses the text and adds HTML tags accordingly. A good WYSIWYG editor also displays the HTML code symultaniously.
A lot of editors are available freely for download and we choose “WIDGEDITOR” by Cameron Adam

because its easy to use and customize.

After tweaking and editing it according to the requirements and   looks this is what we got:

The converted html code is put into the database and displayed on the frontend.All images and media are stored in the html text with their locations referenced.

Do you know that Checkpoint stores it logs in binary format. This will be only decoded to to readable format when its being called by the SmartTracker program which comes with Checkpoint R70 software.

Alternatively you can use other commercial product such as “ManageEngine – Firewall Analyzer” & “Sawmill” to read the log file.

What happen if you dont have money to buy those software and would like to conduct investigation on a security event which occurs recently in your organization.

You can still use the checkpoint “fwm logexport” command to convert the binary format log file to readable format, such as “ascii”.

Follow the steps below to convert your file.

Steps

1. Login to your Chekpoint R70 as expert mode via SSH.

2. Go to the directory where your logs file are kept.

3. Issue the command below:

“fwm logexport -i 2009-11-03_235900.log -o 2009-11-03_235900_read.log -p”

4. There should be new file “2009-11-03_235900_read.log” created in your current directory. Use the cat, vi command to read the file.

Where:

“-i ” is your input log file which is in binary format.

“-o” is you output log file which will be in readable format.

“-p” is to exclude the port number from being resolve

I started playing with Scalas parser combinators recently.  I had no purpose, I was just trying to figure them out.  They promise to provide a parsing engine built with BNF (Backus-Naur Form) that performs lexical anlysis and evaluation.  I find that they deliver on this promise, but the implicit promise of simplicity was not delivered.

It is probably my fault for assuming things would be simple.  The keyword to my mistake is ‘combinators’.  The concept is to use simple parsers and combine them into parsers of ever increasing complexity.  So my assumption of simplicity is falsified when a parsing engine is built up into something useful.

Here are two things I learned:

1. When you construct a parser, you end up with a function that creates a function.  This created function then needs an input to process.
2. There does not appear to be a simple test for EOI (end of input).  The function ‘phrase()’ will do this test, but cannot be used at the top level.

Lesson One: Using a Parser Combinator

I went nuts looking for this information.  Everyone talks about creating PCs, noone gives an example of using one.  Here is my parser:

class Sentencer extends StandardTokenParsers {
    def sentence: Parser[Any] = {ident | ident ~ sentence}
}

Very simple, it tries to match a list of words. A sentence is defined as either a single word (ident), or a word followed by a sentence. Since the ’sentence’ part is to the right of the word part, this is a ‘right recursive’ expression. The ‘ident’ is a StandardTokenParsers function that produces a Parser which matches individual words (ie. tokens) from the input stream.

Here is the same parser with a run() method to process a string input. The method returns a ParseResult object. Note that ‘lexical’ is a local variable that is an instantiation of the ‘Scanners’ trait. ‘Scanner’ is hidden within ‘Scanners’. Using Scanner as the input is only valid for TokenParsers. If you are using a byte level parser, there are other scanners available.

class Sentencer extends StandardTokenParsers {
    def run(text:String) = {
        var fn = sentence()
        fn(new lexical.Scanner(text));
    }
    def sentence: Parser[Any] = ident | ident ~ sentence
}

Now, here is how you invoke the parser:

new Sentencer.run("one two three")

The result is a ParseResult object which displays as the following string:

[1.5] parsed: one

As you can see, my parse only matched the first element of the input. Now on to -

Lesson Two: Matching to the End of Input

There is a function in Parsers named phrase(). As with most of the Parsers functions, it takes one or more parsers and returns a new combined parser. In this case, it takes a single parser, then returns a Success object if the underlying parser reached the End of Input; and a Failure object otherwise. Success and Failure are both extensions of the ParseResult abstract class.

My first instinct was to wrap it around my entire parser object like this:

    def run() = {
        var fn = sentence
        var fnp = phrase(fn)
        fnp(new lexical.Scanner(text))
    }

Now when I run, it still only matches the first token (‘one’), then returns Failure. It took me a day or two to wrap my head around exactly what I had done. I finally came up with a better place to use the phrase() function:

def sentence = phrase(ident) | phrase(ident~sentence)

When I run this version, it matches the entire input. The key is that it first matches the ‘one’, then fails. On failure, the pipe parser (‘|’) will try the alternative phrase(ident~sentence). ‘ident’ is matched, then the remainder of the input is matched against ’sentence’. Rinse and repeat to get the following ParseResult:

[1.14] parsed: (one~(two~three))

Given the recursive nature of the definition, it turned out I could get the same thing with the slightly simpler:

def sentence = phrase(ident) | ident~sentence

Eventually I decided to ditch ‘phrase()’ for my own EOI parser (end of input):

class Sentencer extends StandardTokenParsers {
    def run(text:String) = {
        var fn = sentence
        fn(new lexical.Scanner(text));
    }

    def sentence: Parser[Any] = { ident ~ EOI | ident ~ sentence }

    def EOI: Parser[Any] = new Parser[Any] {
        def apply(in: Input) = {
            if (in.atEnd) new Success( "EOI", in )
            else Failure("End of Input expected", in)
        }
    }
}

Bueno para los que se preguntan como hacer un parser a partir de una gramática que hemos diseñado, aquí le enseñare brevemente como hacer un parser descendente recursivo

Tenemos que tomar en cuenta lo siguiente:

1. Nuestra gramática no debe ser recursiva a la izquierda.Por ejemplo la producción

E::=Er

es recursiva  la izquierda, para este tipo de gramáticas nuestro parser no sería funcional, asi que hay que quitarle la recursividad a la izquierda

2.Debemos construir los conjuntos: First, Follow, para cada producción de nuestra gramatica.

Estructura del parser:

LookAhead: Esta variable(global), inicialmente es el token de mas a la izquierda de la entrada.

Por cada NO TERMINAL de la gramática debe existir un procedimiento del parser.Para hacer mas fácil su programación, el nombre del procedimiento tendrá el nombre del no terminal.

Cada una de las opciones del no terminal formaran el cuerpo del procedimiento.

Procedimiento Match: Con este procedimiento sabremos si es el TERMINAL correcto, lo detallare a continuación

void Match(token simbolo){

if(lookahead==simbolo)

lookahead=siguienteSimbolo;/* para obtener el siguiente simbolo se puede hacer un procedimiento para pedir el siguiente toke que recibimos del analizador lexico*/

else

ERROR
}

Bueno, ahora a programar, escribiré un parser para la siguiente gramática:

terminales: x,y,z;

E:=xP | H     /*  E produce x segido de la producción P ó la producción H       */

P:=Tz

H:=y

T:=yx;

****************************************************************

var token lookahead;//variable del tipo del token que ustedes utilicen

void main {

//Iniciar Scanner si s necesario

lookahead=nexToken() // funcion que devuelve el token siguiente

E();

}

void E(){

if(lookahead==x)

Match(x);

P();

else

H();

}

void P(){

T();

Match(z);

}

void T(){

Match(y);

Match(x);

}

void H(){

Match(z);

}

Y así de sencillo es un parser descendente Recursivo

Si se dan cuenta el parser recorre el árbol de arriba hacia abajo de allí viene descendente,

Y Recursivo se debe a que el parser puede llamarse a si mismo directa o indirectamente por medio de sus funciones

Espero que les sirva de algo,

Dudas, comentarios o sugerencias, no duden en escribir

saludos

Parser ErrorDescription: An error occurred during the parsing of a resource required to service this request. Please review the following specific parse error details and modify your source file appropriately.
Parser Error Message: Ambiguous match found.

=_=

In today’s Programming Praxis problem our task is to write a parser for simple regular expressions. Since Haskell has a very good parser library called Parsec, we’re going to be using that. Let’s get started.

First, some imports:

import Control.Applicative ((<$>), (*>), (<*), (<*>))
import Data.Char
import Text.Parsec
import Text.Parsec.String

Next we define our data structure. There are seven constructs we have to implement, split into two groups based on whether or not they can be followed by a star or not.

data Elem = Lit Char | Esc Char | Any | Set Bool [Elem] deriving Show
data Chunk = Elem Elem | BoL | EoL | Star Elem deriving Show

The parser itself is not too difficult if you know how the operators from Control.Applicative work. <$> means apply the function on the left to the result of the parser on the right. <*, *> and <*> take only the result on the left, right and both sides respectively.

regex :: Parser [Chunk]
regex = (++) <$> bol <*> many chunk where
    bol = option [] (const [BoL] <$> char '^')
    chunk = choice [Star <$> try (element <* char '*'),
                    const EoL <$> try (char '$' <* eof),
                    Elem <$> element]
    element = choice [esc <$> try (char '\\' *> anyChar),
                      const Any <$> char '.',
                      Set False . expandSet <$> set "[^",
                      Set True . expandSet <$> set "[",
                      Lit <$> noneOf "]"]
    esc c = if elem c "nt" then Esc c else Lit c
    set s = try (string s *> many1 element <* char ']')
    expandSet (Lit a:Lit '-':Lit b:xs)
        | validRange a b = map Lit [a..b] ++ expandSet xs
    expandSet (x:xs) = x : expandSet xs
    expandSet _ = []
    validRange a b = b > a && ((isLower a && isLower b) ||
                               (isUpper a && isUpper b) ||
                               (isDigit a && isDigit b))

With the parser written, the function to parse a string is trivial:

parseRegex :: String -> Either ParseError [Chunk]
parseRegex = parse regex ""

Some tests to see if everything is working properly:

main :: IO ()
main = mapM_ print [parseRegex "[0-9][0-9]*",
                    parseRegex "^..*$",
                    parseRegex "hello",
                    parseRegex "^ *hello *$",
                    parseRegex "^[^x].*[0-9] *x$"]

Piece of cake. Next time we do the implementation.

I love Python!

Last night I created a small Python script for a friend, in a couple hours, to get the definitions of a list of Spanish words using SpanishDict. It works by sending a request to SpanishDict and parsing the HTML it gets back, only keeping the definition of the word. If you run it from the command line you have the choice of passing files or individual words. Here’s the help message:

Usage: spanishdict.py [options] file [file2, file3, ...]
Options:
  -h, --help            show this help message and exit
  -o OUTPUT             where to send the output, file name or '-'
to indicate stdout [default: {first file}.defs]
  --words               interpret the input as words, not file names [default: False]
  --delimiter=DELIMITER
                        delimiter between words in the files [default: \n]

The script is available in my code repository in the ‘python’ directory or you can view it online.

I’ve recently resumed work on the question-answering chatbot, and am trying to get it to comprehend a broader range of questions and statements.   The “big idea” is to create a number of “sentence patterns” that the pattern matcher can recognize and respond to.  The reason this is a “big” idea is because I am trying to avoid anything algorothmic or procedural — everything is to be done by specifying OpenCog hypergraphs, and NOT by writing C++ code, or scheme code (or python code…etc). The reason for working entirely with patterns and hypergraphs, rather than with C++ or scheme, is because this puts the “knowledge” of the system into a form that AI routines can manipulate it: learning algos can learn new hypergraphs; statistical algos can gather usage information on which hypergraphs get triggered, and so on.  This is all easer said than done: although I’ve eliminated a fair amount of question-answering code previously written in C++, I’ve also had to write some new scheme code. Bummer.

Patten matching is now used through-out all of the OpenCog NLP pipeline, although not in a unified manner. The Link Grammar parser uses patterns (called “disjuncts”) to determine how the words in a sentence can link to one-another, thus “parsing”, or pulling the grammatical structure out of a sentence (this paper provides an excellent overview). The RelEx dependency relation extractor applies patterns on the link-grammar output to extract syntactic relations. For example, the sentence “John threw a rock” becomes

_obj(throw, ball)
_subj(throw, John)

after RelEx gets done with it. And now, there are a dozen patterns inside of OpenCog that can pick out certain kinds of questions and statements from RelEx output, and pattern-match questions to find answers to them.

For example, the new OpenCog patterns convert “The capital of France is Paris” into

capital_of(France, Paris)

and similarly, “What is the capital of France?” into

capital_of(France,what)

Treating “what” as a variable, there is yet another pattern that matches up the form of the question to the form of the answer, thus deducing that “what” must be “Paris”.

Somewhat harder is using patterns to distinguish similar from dis-similar concepts, so that sentences like “John threw a green ball” aren’t used as answers to questions such as “Did John throw a red ball?”: the word “ball” with modifier “green” has to be detected as a different entity than the word “ball” with modifier “red”: these are two different entities (called “semes” in the code). In fact, out of laziness, I’ve punted on this one: the promotion of word-instances to “semes” is done by code, rather than by pattern matching. But soon, I hope, this will change. In the meanwhile, the README file provides a more detailed discussion.

Here are some patterns that work these days:

<me>         John threw a green ball.
<me>         Fred threw a red ball
<me>         Mary threw a blue rock
<me>         who threw a ball?
<cogita-bot> Syntax pattern match found: Fred John
<me>         who threw a red ball?
<cogita-bot> Syntax pattern match found: Fred

<me>         Did Fred throw a ball?
<cogita-bot> Truth query determined “yes”: throw

<me>         Did Fred throw a red ball?
<cogita-bot> Truth query determined “yes”: throw

<me>         The color of the book is red.
<me>         What is the color of the book?
<cogita-bot> Triples abstraction found: red

<me>         the cat sat on the mat
<me>         what did the cat sit on?
<cogita-bot> Triples abstraction found: mat

And here are some that don’t yet work: “Did Fred throw a green ball?” — gets no reply, because the system can’t find an answer, and doesn’t make the common-sense leap of “can’t find answer-> answer must be no”. Another common-sense problem is illustrated by: “Did Fred throw a round ball?” — the system doesn’t know that balls are round, and simply assumes that a “round ball” is some special kind of “ball”. Oh well. There’s work to be done.

You can try out the chatbot yourself (when its up, and not broken!) on the IRC chat channel #opencog on the freenode.net chat servers.

– Linas Vepstas

I made a python option parser, I wonder what python-ideas will think of it…

http://code.google.com/p/pyopt/

It was amazing cognac time and talking about parser with a good old college friend. E.g. to demonstrate how PostgreSQL analyses some queries.

Statement

inside server’s brain will become:

stmtmulti
    stmt
        SelectStmt
            select_no_parens
                simple_select
                    SELECT
                    target_list
                        target_el
                            a_expr
                                c_expr
                                    columnref
                                        relation_name
                                            ColId
                                                field1
                            AS
                            ColLabel
                                customer
                    from_clause
                        FROM
                        from_list
                            table_ref
                                relation_expr
                                    qualified_name
                                        relation_name
                                            ColId
                                                "FooSchema"
                                        indirection
                                            indirection_el
                                                .
                                                attr_name
                                                    ColLabel
                                                        "Bar"

Will add some more in case of interest

Boa tarde ae a todos

Hoje vo passar aqui um parser de xml com ajax e jquery bem simples e fácil de fazer.

Amanhã vo passar um para criar e montar o xml com mysql e php

bom é simples o código

$.ajax({ //Inicio do evento
type: “GET”, //Metodo que o ajax vai utilizar para pegar a informção
url: “noticia.xml”, //Caminho do arquivo
dataType: “xml”, //Tipo dos dados
success: function(xml) { //Se for bem sucedido
$(xml).find(“noticia”).each(function(){ //Agora o jquery vai procurar cada sessão do xml no caso aqui é a tag noticia que tem como filhas as tags titulo e texto
var titulo = $(this).find(“titulo”).text(); //A variavel que recebe o conteudo da tag titulo
var texto = $(this).find(“texto”).text(); //A variavel que recebe o conteudo da tag texto
noticia += “<strong>”+titulo+”</strong><br><br>”+texto;//Aqui ele monta o texto da noticia pode ser do jeito que você precisar colocar dentro de div lista tanto faz rs
}); //Fecha o laço
} //Aqui eu coloquei só no caso de sucesso aqui você pode continuar o ajax no caso de erro
}); //Fecha o ajax

Bom o código é simples pode ser personalizado sem problemas

Se tiver dúvida pode deixar um comentário

Até!

Webtutoriais:2ED55B30

When a new user registers on a Job-Board, he/she has to fill all the required information manually. Only a handful job-boards provide an option to the candidate to browse his/her resume from PC and all the required fields by job-boards gets filled automatically.

Should you think, it could be another crucial success factor for your job-board; you might want to look at our offering at http://onlineresumeparser.com.

The candidate resume can be in any format like .doc, .docx, .pdf, .rtf, .html or .txt, whether in file folder or email or already available on any other job-portal or career networking site, the resume parser will parse the canidate details with an accuracy level of 80% to 95%. Also the resume gets saved to the database in .doc format irrespective if the browsed resume had some other format. Thus candidate spends time only to correct the information which is not accurately parsed and total time to register on Job-board gets reduced.

Do a signup and sign in; parse a few resumes and check out the accuracy level yourself. You need to have .net framework 2.0 installed on your pc to use this application.

We provide open API integration of our resume parser for all job boards/ applicant tracking systems/HR applications across globe.

En raid, nous sommes 10 ou 25 joueurs, chacun avec ses forces et ses faiblesses, ses envies, ses habiletés et ses “handicaps”… Lorsqu’on est seul, jouer est facile : on fait ce qu’on veut, comme on veut, quand on veut. Mais lorsqu’on se retrouve en groupe, il n’est plus question de ne penser qu’à soit et de n’en faire qu’à sa tête. Il devient IN-DIS-PEN-SA-BLE de penser d’abord aux autres, et ceci est valable aussi bien pour le joueur lambda que je suis, que pour le ou les “leader(s)”.

Je ne suis pas un donneur la leçon, je suis comme tout le monde et je ne cherche qu’à progresser dans ce jeu que j’aime beaucoup. Pour cela, j’utilise depuis longtemps un outil vraiment formidable qui me sert à prendre conscience de mes forces, de mes faiblesses, à mieux connaitre ma classe de personnage et aussi globalement à mieux me connaitre en tant que joueur. Autant d’éléments qui m’aident à progresser. Cet outil s’appelle le “Journal de combat“. Je l’enregistre dès que j’y pense quand le raid ou la rencontre prévue va demander des efforts et un apprentissage. Une fois le journal de combat enregistré, il est alors possible de l’analyser grâce à des programmes en ligne comme wow web stats, wow meter online ou encore World of logs qui a l’avantage de présenter l’information de façon grahique, très parlante, et dont les chiffres importants peuvent rapidement être trouvés.

Il ne s’agit pas de savoir qui fait le plus de DPS. Il ne s’agit pas de savoir qui fait le plus de HPS. Il ne s’agit pas de savoir si untel est “meilleur” qu’untel ou d’une façon générale de s’espionner les uns les autres dans le raid. Il s’agit pour chacun de savoir ce qu’il vaut par lui-même, de prendre ça pour base, et ensuite de s’auto-évaluer dans ses prochaines performances : est-ce que mon sort “X” est rentable ? Est-ce que j’aurais pu faire quelque-chose à cet instant alors que je vois que j’ai été immobile ? Il y a plein de questions qu’on peut avantageusement se poser en analysant le journal de combat à travers un site comme Worldoflogs.com.

Enfin, si on veut progresser.

Si vous faites partie des gens qui veulent progresser dans World of Warcraft, en tant que joueur et non en tant que personnage, alors je vous invite à regarder comme moi attentivement les statistiques de vos raids et à y chercher ce qui peut vous aider à mieux remplir votre rôle tout en gardant à l’esprit que…

…le plaisir de jouer d’abord.

Parce que World of Warcraft n’est qu’un jeu, un loisir, un plaisir, et que personne ne devrait y cliquer le bouton “Connexion” comme il glisse sa carte de pointage au boulot : le raid est une partie très exigente du jeu qui n’est pas pour tous les caractères, et il n’y a aucun jugement de valeur à avoir à ce propos car chacun est libre de ses goûts, de ses envies, de ses besoins et de ses plaisirs.

Bonne journée.

-Cc

Hi everyone! It was a long time since I posted something new here, because my network keeps going down… But this time, I’m bringing you news about the Conspiracy engine. Missed him? Me too… Well, it has a brand new face since many new features have been added that make him stand much higher then before.

To start, a small problem with the vertices was corrected. Somehow the vertex data was getting scrambled all the way through the CMOL (Conspiracy Mesh Optimization Pipeline), and I ended up removing some of the optimization routines that could be handled in another way by DirectX himself. And by doing so, the GenerateAdjacency problem was also solved! This way, the meshes are more perfect and faster than ever!

Also, I’ve implemented a XML parser system from scratch. With that, I created implicit support for something else: Scene graphs! And this is a very important part of the engine, as some objects might now be loaded directly through a simple configuration file. This is great for scenarios, lights, etc., but also for the future creation of a visual editor for the engine!

With so many new features, something logical occurred! A game is being developed on top of it and I can guarantee that it is already in a very advanced state! Stand by for more news!

С некоторого времени я стал делать на Python часть моих повседневных задач по анализу языков, трансляторам и пр. Вначале для вспомогательных целей, а потом и для парсинга небольших языков, прототипирования грамматик, деревьев AST, трансформаций кода. Многие при этом подумают про OCaml, но в Unix-среде (привет spb-archlinux!) от Python с его библиотеками пользы больше.

Для задач парсинга я написал библиотеку funcparserlib. Эта библиотека предназначена для создания парсеров по методу рекурсивного спуска на основе функциональных комбинаторов. Также я написал вводное руководство по funcparserlib (на английском), которое будет интересно всем, увлекающимся функциональным программированием (FP) и/или языком Python. Рекомендую его почитать!

Вот, например, такие картинки деревьев можно легко получать с помощью funcparserlib:

>>> print dotparser.pretty_parse_tree(tree)
Graph [id=g1, strict=False, type=digraph]
`-- stmts
    |-- Edge
    |   |-- nodes
    |   |   |-- n1
    |   |   |-- n2
    |   |   `-- SubGraph [id=n3]
    |   |       `-- stmts
    |   |           |-- Edge
    |   |           |   |-- nodes
    |   |           |   |   |-- nn1
    |   |           |   |   |-- nn2
    |   |           |   |   `-- nn3
    |   |           |   `-- attrs
    |   |           `-- Edge
    |   |               |-- nodes
    |   |               |   |-- nn3
    |   |               |   `-- nn1
    |   |               `-- attrs
    |   `-- attrs
    `-- Edge
        |-- nodes
        |   |-- SubGraph [id=n3]
        |   |   `-- stmts
        |   `-- n1
        `-- attrs

Итак, предлагаю взглянуть на руководство, а питонистам — попробовать funcparserlib, посмотреть другие доки и примеры на сайте библиотеки.

Отличительные особенности библиотеки funcparserlib:

• Несколько необходимых удобных комбинаторов парсеров (API всего 14 вызовов). Код получается компактным, очень похожим по языку на xBNF-грамматики
• Маленький размер самой библиотеки: всего лишь 0.5 KLOC с комментариями
• Обнаружение ошибок по методу длиннейшего разобранного префикса даёт разумные сообщения об ошибках разбора
• Маленький токенизатор на основе регулярных выражений позволяет следить за позицией лексем в тексте, выдавать её в сообщениях

При своём небольшом размере, библиотека является достаточной для написания парсеров весьма больших грамматик. Но главное предназначение — разбор небольших языков и языков DSL (предметно-ориентированных).

Для Python существуют несколько библиотек синтаксического анализа. Сравним некоторые из них с funcparserlib:

• pyparsing. Самая популярная библиотека. Имеет не очень большой размер кода (3.7 KLOC), очень избыточный разношёрстный API (около сотни вызовов), довольно медленная (по простым тестам в 3 раза медленнее, чем funcparserlib)
• LEPL. Библиотека с большой функциональностью, опциями и пр. (API содержит около сотни вызовов) Имеет очень большие для данной задачи исходные коды (около 15 KLOC). Быстрая, по утверждению авторов

Библиотека funcparserlib возникла поначалу из игрушечного примера парсера JSON, который я написал в 2008 году. Пример был создан, чтобы показать, что можно писать парсеры, в точности соответствующие формальной грамматике языка. Летом 2009 года я вернулся к парсерам на Python и решил дописать библиотеку, добавить токенизатор на regexps, выполнить оптимизации и т. д. На данный момент доступна версия 0.3.2, по которой я написал довольно много документации (на английском).

Теперь funcparserlib включает вполне приличный парсер JSON как один из примеров. Этот парсер поддерживает JSON со всеми нюансами и по скорости всего в 3 раза медленнее, чем специализированная библиотека simplejson. А исходного кода — в 8 раз меньше, намного более читаемого

A couple of weeks ago I started work to build a universal parser for SQL.  The work started with a basic need I had for minimal parsing of SQL statements in the GarinDriver JDBC project.  I was unable to find any open source parser code that would fully support the basic structure used by MySQL, PostgreSQL, Oracle, and other similar systems.   In particular, I wanted support for q’<>’ used by Oracle in addition to the SQL standard string encoding, MySQL’s use of C style string escaping, and $tag$ style strings for PostgreSQL and H2.  Conceptually I wanted to build ONE parser that could be configured to properly parse ANY dialect and that would support a framework for adding additional exceptions in the future.

The basics of my parser framework is now complete. It is smart enough to parse MySQL, H2, HSQLDB, Oracle, PostgreSQL, and ISO SQL2003 variants. The shallow parsing framework uses a base Token class and a range of subclasses to describe the most basic elements of the SQL grammar. The parser identifies the core keywords, statement boundaries, string, comment, and identifier limits but does not look into deeper language issues like statement structure.   One benifit of the shallow parsing is that it allows for flexibiltiy and does not require a complete BNF style grammar.  Even with only this basic level, it is possible to leverage this code for useful stuff beyond the GarinDriver, LiquiBase data migration solution.

The following bare bones test page demonstrates how the parser can be used to format and add color to SQL batches.  The code would also be useful for someone building a universal SQL editor.
http://markfarnsworth-dev.appspot.com/RenderSql

Overall, I am happy with the approach and while I plan to use a more established framework like ANTLR or XTEXT for future deep parsing efforts I feel my homegrown framework provides a better approach for shallow parsing vs. the larger and more complex language tools that I have reviewed.

I recently completed some work on the GarinDriver to research a more flexible model for database schema change management. This work gave me a chance to dig a little deeper into the nuts and bolts of Java database interactions. Working to track schema changes and support a wide range of databases increased my appreciating for powerful open source database platforms available today. In particular, working with H2 Database gave me a chance to explore what I now think is one of the hidden treasures of the open source DB marketplace. Overall it was a fun bit of hobby coding and I believe it will be useful in future real world projects.

Working on the driver, one challenge that went beyond my expectation was parsing the various dialects of SQL. In a review of current open source projects, I did not find any public license parsers that can parse statements from ALL database systems. In a sense, we are supposed to have a standard but it is really well followed by the vendors and as such writing a universal parse is a bit difficult. I had some free time this weekend so I took on the challenge since it is an area where the open source community really does not currently have a good solution. For example the tools that ship with Eclipse can not parse the $tag$ style strings from PostgreSQL or the q’[O'Brian]‘ style supported by Oracle. In any case,  I had some free time and interest so I decided to build my own parser.  Without such a parser it would be impossible to provide full support for the statement execution model needed to support the GarinDriver desgin.  Workarounds were possible but a proper parser felt like the best approach.

Building a parser is complex and often involves specialized tools (i.e. ANTLR, BISON, JAVACC, FLEX, LEX, YACC, etc). There are benefits and drawbacks to the traditional grammar definition and parser generator approach.   My initial review it seemed that supporting the often contradictory approaches used by different systems would be quite difficult with these tools.

Since all that I needed for the GarinDriver/LiquiBase project was “shallow parsing” my approach was to use a small framework of Java classes. The use of my own framework allows me to share logic across the SQL dialects and provided a chance to explore idea for a more dynamic approach to “shallow parsing”. So far the approach seems to be working out nicely although I am considering building a deep parser at some point later on with the Eclipse XTEXT project but for now the flexiiblity of a home grown hand coded parser has proved to be a viable approach for “shallow parsing”.

Parser Fun:

• Standard SQL comments have EOL style (–) and the block style (/* */).
• Block style comments nest within each other so parser must count the nesting levels.
• MySQL supports pound sign comments in addition to the standard forms.
• HSQLDB supports // style comments in addition to the standard forms.
• Standard SQL uses double quotes for identifiers and single quotes for text strings.
• The parser must support both including quote symbol doubling for escapes (i.e. O”Brian or My “”big”" table.
• Oracle supports q’[O'Brian]‘ in addition to the standard style.
• MySQL uses C style string encoding (i.e. ‘O\’Brian’ ).
• PostgreSQL supports $tag$O’Brian$tag$ style in addition to the standard forms AND the mysql format.

My parser uses object oriented techniques to define a base token concept, extend the base concept for SqlStatements, and then extend the SqlStatements to define the dialect variants in what I hope will be an extendable framework. This approach provides more flexibility for future growth vs. more static models and parser generator tools. In any case the approach seems to work and the ability to use object orientation to extend the parser in new directions seems like a good thing.

As with other hobby projects, my work in this area is EPL and hosted on Google code. The documentation is sparse but if you are looking for SQL parser code you may find that this code can help you develop new and interesting tools for working with SQL systems.

http://code.google.com/p/garinparser/

If you decide to use the parser please let me know. Also, if you can define a legal SQL statement that does not parse correctly with my parser let me know since I like to eat tasty bugs.