Joel Spolsky nel 2003 scrive  – The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) -

Gioia e dolore di ogni informatico.

Perchè gioia? Prima dei sistemi di encoding, fra i quali Unicode è un po’ il principe, regnava l’ASCII “chaos”, nell’articolo è spiegato molto bene, e come sappiamo, la standardizzazione è il punto di partenza per creare insieme qualcosa di grande.

Perchè dolore? Bisogna provare per credere! Non avete idea di quanti siti web scritti ante-2000 sono scritti in un encoding, hanno un meta tag che indica quello sbagliato e magari l’http server ne usa un altro. C’è sempre da impazzire un po’ con queste cose: un po’ è perchè sono implicitamente complesse, un po’ perchè molta più gente di quanto pensiate, definendosi sviluppatore o peggio ‘informatico’ (?), pavoneggia la sua ignoranza parlando di charsets e affini. “Ah ma tanto io faccio applicazioni plain text: non mi interesano queste cose!”. Ecco, se un americano può sembrare solo ignorante a dire questo, un italiano è anche peggio… perchè?

Gli accenti maledizione! Quante volte avete visto strani punti interrogativi o quadratini su siti internet, e-mail o documenti vari? Ebbene, quello è un problema di encoding. E se vi dicessi che i cinesi hanno svariate centinaia di lettere nel loro linguaggio? e badate bene, qui non si parla ancora di internazionalizzazione del software, quello è uno step molto successivo.

Io consiglio sempre questo articolo di Joel on software per iniziare a farsi un’idea di questo mondo incredibile.

Most developers already struggled with textual data from some third party system and getting garbage special characters and the like because of wrong character encodings.  Some days ago we encountered an obscure problem when it was possible to login into one of our apps from the computer with the password database running but not from other machines using the same db.  After diving into the problem we found out that they SHA-1 hashes generated from our app were slightly different. Looking at the code revealed that platform encoding was used and that lead to different results:

The apps were running on Windows XP and Windows 2k3 Server respectively and you would expect that it would not make much of a difference but in fact it did!

Lesson:

Always specify the encoding explicitly, when exchanging character data with any other system. Here are some examples:

• String.getBytes(“utf-8″), new Printwriter(file, “ascii”) in Java
• HTML-Forms with attribute accept-charset="ISO-8859-1"
• In XML headers <?xml version="1.0" encoding="ISO-8859-15"?>
• In your Database and/or JDBC driver
• In your file format documentation
• In LaTeX documents
• everywhere where you can provide that info easily (e.g. as a comment in a config file)

Problems with character encodings seem to appear every once in a while either as end user, when your umlauts get garbled or as a programmer that has to deal with third party input like web forms or text files.

The text file rant

After stumbling over an encoding problem *again* I thought a bit about the whole issue and some of my thought manifested in this rant about text files. I do not want to blame our computer science predecessors for inventing and using restricted charsets like ASCII or iso8859. Nobody has forseen the rapid development of computers and their worldwide adoption and use in everyday life and thus need for an extensible charset (think of the addition of new symbols like the €), let aside performance and memory considerations. The problem I see with text files is that there is no standard way to describe the used encoding. Most text files just leave it to the user to guess what the encoding might be whereas almost all binary file formats feature some kind of defined header with metadata about the content, e.g. bit depth and compression method in image files. For text files you usually have to use heuristical tools which work  more or less depending on the input.

A standardized header for text files right from the start would have helped to indicate the encoding and possibly language or encoding version information of the text and many problems we have today would not exist. The encoding attribute in the XML header or the byte order mark in UTF-8 are workarounds for the fundamental problem of a missing text file header.

User CSSCAN & CSALTER.

See here

Questa è curiosità, ma che è importante sapere qualora si dovesse lavorare direttamente con i codici ASCII: non è affatto vero che un carattere è univocamente determinato da un codice numerico, è invece vero che un carattere ha un codice numerico univoco all’interno di un un character set specificato!

Consideriamo il carattere £ (pound): il suo codice ASCII nelle comuni tabelle di riferimento è 163.
Tuttavia CHR(163) = £ solo se il charset del database è WE8ISO8859P1 o WE8MSWIN1252, ma in un database con charset AL32UTF8 la funzione CHR(163) ritorna un carattere diverso, poichè CHR(49827) = £.

Potete controllare le vostre impostazioni:

SELECT *
FROM V$NLS_PARAMETERS
WHERE PARAMETER = 'NLS_CHARACTERSET';

SELECT CHR(163) FROM DUAL;


Al posto di CHR(), è consigliabile usare la funzione Oracle UNISTR() che vuole come argomento il codice esadecimale ASCII del carattere da rappresentare e, convertendolo nel codice relativo al charset del database, lo visualizza correttamente. Il codice deve essere passato come stringa di testo in formato ‘XXX’ (leading zeros su 4 cifre, precedute da backslash).

• Conversione di un carattere (se il charset è AL32UTF8 non ritorna gli stessi valori):
  
SELECT UNISTR('0A3'), CHR(163) FROM DUAL;

  Al posto di UNISTR('0A3') si può utilizzare un blocco TO_CHAR(163, 'fm0XXX') che effettua la conversione dell’ASCII 163 nel corrispondente UNICODE esadecimale 00A3.
  

• Conversione di una parola con un carattere speciale:
  
SELECT UNISTR('Espa\' || TO_CHAR(241, 'fm0XXX') || 'a') FROM DUAL;
SELECT UNISTR('M\' || TO_CHAR(246, 'fm0XXX') || 'ller') FROM DUAL;

  Le parole España e Möller verranno visualizzate correttamente, indipendentemente dalla configurazione dell’ambiente.
  

• A questo punto una funzione che normalizzi le lettere accentate italiane, indipendente dal charset, potrebbe essere:
  

  CREATE OR REPLACE FUNCTION Normalizza(i_string IN VARCHAR2)
  RETURN VARCHAR2
  IS
  	o_string VARCHAR2(4000) := i_string;
  BEGIN
  	o_string := REPLACE(o_string, UNISTR('\' || TO_CHAR(224, 'fm0XXX')), 'a');
  	o_string := REPLACE(o_string, UNISTR('\' || TO_CHAR(232, 'fm0XXX')), 'e');
  	o_string := REPLACE(o_string, UNISTR('\' || TO_CHAR(233, 'fm0XXX')), 'e');
  	o_string := REPLACE(o_string, UNISTR('\' || TO_CHAR(236, 'fm0XXX')), 'i');
  	o_string := REPLACE(o_string, UNISTR('\' || TO_CHAR(242, 'fm0XXX')), 'o');
  	o_string := REPLACE(o_string, UNISTR('\' || TO_CHAR(249, 'fm0XXX')), 'u');
  	RETURN o_string;
  END;

Vi que tem muita gente ( como eu tive ) com problemas de codificação utf-8 nas páginas .jsp geradas a partir do Netbeans IDE.

Chega de stress após criar todo o layout usando caracteres especiais e depois, quando renderizado no Browser, a grande decepção: mesmo sabendo que o seu projeto está configurado, por padrão, a usar a codificação de caracteres UTF-8 ele teima em mostrar erros.

A solução é bastante simples, basta insrir o seguinte código XML na sua página:

<?xml version="1.0" encoding="UTF-8"?>

ATENÇÂO: a codificação deve, obrigatoriamente, ser “UTF-8” e não “utf-8“.

Found this post useful? Want to find more resources? Discuss RPG gaming with others who share the same interest? Post your currently game and see what others think? Get help on starting up a game? Click Here to go to the new forum!

This blog is moving! Click Here to go to the new blog!

This Tutorial was request by RPG_Addict in the “Request A Tutorial!” section of the blog.

The Requirements

Alright, so many of you may be wondering exactly how can you create a custom window skin? Well, it gets really easy once you know what you are doing so here’s a quick guide to get you understanding!

First, your windowskin would need to be sized to 128 x 128 pixels. To get the best quality from them you’ll want a 32-bit PNG format.

You’ll need to use a graphic making program to do this. You could use MS Paint or paint.NET. Both are free program. (I’ll make a new post soon as to where to get paint.NET and a few other resources from.)

Or if you have it, you could use Adobe PhotoShop C2/C3/C4!

After doing this you are ready for the next step.

Of course, this is for RPG Maker VX and RPG Maker VX Game Projects only.

The Steps

To be able to create a successful Custom Window Skin using correct colours, you’ll need to learn the 6 letters. Once you learn these, you’ll be able to create your own Custom Window Skin for RPG Maker VX Games in no time!

RPG Maker VX Game Project Custom Window Skin Large

Letter A. Window Background

This part would be 64 x 64 pixiels. You’ll be putting this in the top left section of your window skin. You’ll want to use solid colours for this. To get the best effects out of it, try experimenting with gradients and other patterns. E.g. You could try putting star shades onto it and see how that turns out. Be creative and have fun!

Letter B. Window Frame – Effects on the Background

I love this part! You would want this part to be 64 x 64 pixiels and put it at the bottom left corner. It will put effects over your background. You could put any pattern you wish on this and see how it looks. E.g. this default windowksin will put lines over the background giving the message a better overall effect. So it works like another image onto the background.

Letter C. Window Frame & Scrolling Arrows

Alright. The tricky part. You’ll need the corners to be 16 x 16 pixiels yet keeping the frame size at a width of 16 to give it that curvy effect. This would cover the border of the windowskin. The inside would cover the arrows shown when the window is scrolling up, down, left or right in-game. I usually leave this alone and just change the colours, but you can try to customise it and change the arrow shapes. You never know you might make something really special!

Letter D.  The Flashing Border

This would be the thing that shows a flashing border on your window skin in-game. It flashes whatever you are selecting. If you have not noticed this before, open up a VX game and look at your menu. You’ll notice whatever you’ve selected is flashing. Again I usually leave this, but if you want you can customize to fit you.

Letter E. Arrow Boxes

These arrows are the arrows that animate moving upwards and downwards right in the centre bottom of the windowksin each time you are reading a piece of text. My recommendation would be to make these arrows the same at the ones in Letter C. But again it’s all up to you.

Letter F.

Ah….finally! The final letter. These will show you the 36 colours used in your RPG Maker VX Game Project. The first colour would be the default colour. To change the text colour in game you’d use the “n[colournumberhere]” command. More detail of this will appear on the blog in the future. You can change these colours to what you want, but I highly recommend leaving them as they are.

Final Notes

Well, hopefully that can get you started. If you have anymore questions just post a comment below and you’ll receive some answers…..eventually >.>

Una de mis herramientas favoritas, e indispensable, cuando estoy trabajando es el Web Developer ToolBar para Firefox, todavía no conozco a alguien que diseñe Sitios Web y no lo utilice.

Una de las opciones que uso muy seguido es el Validador, valido el  HTML y CSS localmente. Hoy, validando y limpiando un archivo CSS, al final me quedaba este error:

The @charset rule may only occur at the start of the style sheet. Please check that there are no spaces before it. @charset “utf-8″;”

Pase un buen rato dándole vueltas y no encontraba la solución. Casi me vuelvo loco porque después el mensaje de error decía que no estaba incluyendo el @charset.

A pesar de que googlie para ver si encontraba la solución, y la fuente del problema, me di por vencido y deje de validar localmente, y para tener una “segunda opinión” utilice el Markup Validation Service de la W3C, y adivinen que, cero errores, ni uno, el código era 100% valido!!

Después de muchas vueltas no encontre el porque del error al validar localmente usando el Web Developer, parece que es alguna pulga en el addon, pero al menos voy a poder dormir en paz sabiendo que el CSS era valido .

Found this post useful? Want to find more resources? Discuss RPG gaming with others who share the same interest? Post your currently game and see what others think? Get help on starting up a game? Click Here to go to the new forum!

This blog is moving! Click Here to go to the new blog!

This Tutorial was request by Jackrmesser in the “Request A Tutorial!” section of the blog.

Hope you find this useful. People are constantly saying “Help! The side view battle system won’t work for me!”. If it works for others the problem is that you have not set the system up correctly.

The answers to the most frequently asked questions can be found in this same link I will be posting below.

STEPS TO GETTING THE SIDE VIEW SYSTEM WORKING

• RPG Tankentai Sideview Battle System” is by far the best side view battle system for VX currently. Therefore we should use the English translated version of it.

• Go to a community or forum where this script is posted. E.g. I would go [Link moved to new blog]

Found this post useful? Want to find more resources? Discuss RPG gaming with others who share the same interest? Post your currently game and see what others think? Get help on starting up a game? Click Here to go to the new forum!

This blog is moving! Click Here to go to the new blog!

Want some more battle sprites for the font view system? Or the side view system? Here are two links I recently came across which contain awesome battle sprites. I recommend you use these in your projects. I’ll post some quick pics next to the links.

First, for the font view battle system

It's a monsterrrr

^ Wow. It’s a scary mask….thing.

[LINK REMOVED. AT NEW BLOG.]

Next, for the side view battle system. And these contain ulimited battlers. You’ll never need another sprite if you browse through this site.

^ There a load more where these ceom from.

[LINK REMOVED. AT NEW BLOG.]

Hope these help you as much as they helped me. =)

If you can’t find names to give credit, give credit to the URL’s.

I used the following methods to convert my server to UTF-8, using standard Ubuntu packages, eg mysql-server from synaptix, nothing had to be recompiled.

They may not work in your environment. Use them at your own risk, and BACKUP everything first. (Preferably test before using it on any production server.) I post them only as a personal memory aid. If you use the information here and your server bursts into flames, you accept responsibility for that.

PuTTY /SSH:
In PuTTY 0.60 you can change the option “Window > Translation > Received data assumed to be in which character set:” to ‘UTF-8′. After this all received data will be interpreted as UTF-8 and displayed correctly. (For anyone ssh’ing to the mysql server and using the mysql> prompt from there!)

Bash:
try adding something like this to your ~/.bash_profile
echo -ne ‘\e%G\e[?47h\e%G\e[?47l'
There's no point trying to work in UTF-8 if your computer thinks you are speaking ISO 8859-1!! This should even fix vi, so AltGr+4 correctly gives the Euro '€' and AltGr+` [backtick] gives the broken pipe ‘¦’, instead of displaying nothing, or something odd like an accented A¦ pair. (or a dot in bash). Of course, Windows notepad will choke on the extended character, but something like notepad++ will work to display it.

MySQL:
A) Edit /etc/mysql/my.cnf, locate the section that starts [mysqld], add the lines below to configure your server, and then restart mysql. This should switch the server to UTF-8 ‘mode’.

[mysqld]
# utf8
init-connect=’SET NAMES utf8′
character-set-server=utf8
collation-server=utf8_general_ci

B) # /etc/init.d/mysqld restart

afterwards you should find the following all correctly updated to utf-8
> Variable Session value Global value
> character set client utf8 latin1
> character set connection utf8 latin1
> character set database latin1 latin1
> character set results utf8 latin1
> character set server latin1 latin1
> character set system utf8 utf8
> collation connection utf8_general_ci latin1_swedish_ci
> collation database latin1_swedish_ci latin1_swedish_ci
> collation server latin1_swedish_ci latin1_swedish_ci

This pinched from https://lists.ubuntu.com/archives/ubuntu-users/2006-February/067367.html because it was so very useful =)

C) Now follow this guide: http://www.nicknettleton.com/zine/php/php-utf-8-cheatsheet which you can also find copied below. Note that I was able to convert my latin1 encoded database (containing a wordpress blog) and all its tables (one-by-one) into utf8 using the below commands whilst the server was ‘hot’ – but take care! (and backups!!). Put db/table names with hyphens into backticks, e.g. `wp-blog`.

——–
To support worldwide languages, you need to use UTF-8 encoding for your web pages, emails and application, rather than ISO 8859-1 or another common western encoding, since these don’t support characters used in languages such as Japanese and Chinese.

Happily, UTF-8 is transparent to the core Latin characterset, so you won’t need to convert all your data to start using UTF-8. But there are a number of other issues to deal with. In particular, because UTF-8 is a multibyte encoding, meaning one character can be represented by more one or more bytes. This causes trouble for PHP, because the language parses and processes strings based on bytes, not characters, and makes mincemeat multibyte strings – for example, by splitting characters ‘in half’, bodging up regular expressions, and rendering email unreadable.

There are a number of great articles online about UTF-8 and how it works – Joel Spolski’s comes to mind – but very few about how to actually get it working with PHP and iron out all the bugs. So, here to save you the time we put in, is a quick cheatsheet and info about a few common issues.

1. Update your database tables to use UTF-8

CREATE DATABASE db_name
CHARACTER SET utf8
DEFAULT CHARACTER SET utf8
COLLATE utf8_general_ci
DEFAULT COLLATE utf8_general_ci
;

ALTER DATABASE db_name
CHARACTER SET utf8
DEFAULT CHARACTER SET utf8
COLLATE utf8_general_ci
DEFAULT COLLATE utf8_general_ci
;

ALTER TABLE tbl_name
DEFAULT CHARACTER SET utf8
COLLATE utf8_general_ci
;

2. Install the mbstring extension for PHP

Windows: download the dll if it’s not in your PHP extensions folder, and uncomment the relevant line in your php.ini file: extension=php_mbstring.dll
Linux: yum install php-mbstring

3. Configure mbstring

Do this in php.ini, httpd.conf or .htaccess. (Remember to prepend these with ‘php_value ‘ in httpd.conf or .htaccess.)

mbstring.language = Neutral ; Set default language to Neutral(UTF-8) (default)
mbstring.internal_encoding = UTF-8 ; Set default internal encoding to UTF-8
mbstring.encoding_translation = On ; HTTP input encoding translation is enabled
mbstring.http_input = auto ; Set HTTP input character set dectection to auto
mbstring.http_output = UTF-8 ; Set HTTP output encoding to UTF-8
mbstring.detect_order = auto ; Set default character encoding detection order to auto
mbstring.substitute_character = none ; Do not print invalid characters
default_charset = UTF-8 ; Default character set for auto content type header

4. Deal with non-multibyte-safe functions in PHP

The fast-and-loose way to do this is with the following php configuration:

mbstring.func_overload = 7 ; All non-multibyte-safe functions are overloaded with the mbstring alternatives

But there are problems with this. php.net has a warning about this potentially affecting the whole server. And even if this isn’t an issue for you, mbstring can make a mess of binary strings.

So, a better route is to search your application code for the following functions, and replace them with mbstring’s ’slot-in’ alternatives:

mail() -> mb_send_mail()
strlen() -> mb_strlen()
strpos() -> mb_strpos()
strrpos() -> mb_strrpos()
substr() -> mb_substr()
strtolower() -> mb_strtolower()
strtoupper() -> mb_strtoupper()
substr_count() -> mb_substr_count()
ereg() -> mb_ereg()
eregi() -> mb_eregi()
ereg_replace() -> mb_ereg_replace()
eregi_replace() -> mb_eregi_replace()
split() -> mb_split()

5. Sort out HTML entities

The htmlentities() function doesn’t work automatically with multibyte strings. To save time, you’ll want to create a wrapper function and use this instead:

/**
* Encodes HTML safely for UTF-8. Use instead of htmlentities.
*
* @param string $var
* @return string
*/
function html_encode($var)
{
return htmlentities($var, ENT_QUOTES, ‘UTF-8′) ;
}

6. Check content-type headers

Check through your code for any text-based content-type headers, and append the UTF-8 charset, so the browser knows what it’s working with:

header(‘Content-type: text/html; charset=UTF-8′) ;

You should also repeat this at the top of HTML pages:
(replace square brackets with angle ones!!)

[meta http-equiv="Content-type" value="text/html; charset=UTF-8" /]

7. Update email scripts

Email can be tricky. You’ll need to update the content-type for any emails and text-based mime parts to use UTF-8 encoding. You’ll also need to alter the way in which headers are encoded to use UTF-8. mbstring provides a function mb_encode_mimeheader() to handle this for you, but it does make a mess of address lists: you’ll need to encoding the name and address parts seperately, then compile them into an address list.

Be sure to encode the subject and other headers too – Korean speakers will tend to put Korean text for the subject.

9. Check binary files and strings

Finally, double check any binary files and strings handled by PHP, particularly uploads, downloads and encryption. In some cases it may be necessary to revert to ASCII just before a download or processing a binary string.

——–

MySQL clients:
Check the options in your mysql clients – should be switched to UTF8 mode also, or you might get your results in latin1, or something wierd like that.

Perhaps use tips from http://tlug.dnho.net/?q=node/276 if you are trying to mysqldump & restore from latin1 to utf8.

PHP:
Use the above to set your PHP / front end into UTF8 (see mbstrings section).

Found this post useful? Want to find more resources? Discuss RPG gaming with others who share the same interest? Post your currently game and see what others think? Get help on starting up a game? Click Here to go to the new forum!

This blog is moving! Click Here to go to the new blog!

Behold the results. It took a while to gather 100 responces, but hopefully it’ll help you all with your games. xPP

Fonte: http://www.explorando.viamep.com/2008/07/caracteres-especiais-e-codificacao.html

Found this post useful? Want to find more resources? Discuss RPG gaming with others who share the same interest? Post your currently game and see what others think? Get help on starting up a game? Click Here to go to the new forum!

This blog is moving! Click Here to go to the new blog!

Wow. Here’s a fantastic video to a RPG Maker VX project in works by SkyDreamers! This project is worthwhile playing and keeping a check on. You may find it to help you come up with inspirational and fascinating ideas in your own RPGcreation. After all, minigames are a huge amount of fun to play. I’ll post the video below.

^ For more information on what Skydeamer’s is about, or any information on this minigame project , check out the official website found in the link page and at the end of this post.

http://www.skydreamers.co.nr

Found this post useful? Want to find more resources? Discuss RPG gaming with others who share the same interest? Post your currently game and see what others think? Get help on starting up a game? Click Here to go to the new forum!

This blog is moving! Click Here to go to the new blog!

You can now have your say on what features you would like to see in the project by filling in this quick survey found here!

I will post all results on this webblog once I collected them. I will also create bar charts and the likes using MS Excel spreadsheet if I get enough votes in.

If you are stuck creating your own game this could be a huge help for you!

[Link move to new blog]

=)