Many website designers design really scrappy websites that do not follow standards at all. I myself tend to write all my XHTML to be XHTML1.1 compliant. As a reader of this blog, I will assume you also attempt to follow standards.

Usually I implement everything to pass xhtml transitional validation. One thing I usually ignore however, is the character encoding.

Put simply, character encoding allows a browser to display and render the document as originally intended. For instance, browsing a site developed using a Japanese-based encoding (e.g. JIS X 0208) will not display correctly unless you have the JIS X 0208 character set installed on your computer.

Without specifying a character encoding, a default character encoding is used. So specifying a character encoding when developing sites that use other characters is a must. But a more important reason exists even if you only develop english websites using UTF-8 or ISO 8859-1. It is a potential security vulnerability.

Essentially, when a character encoding is not specified it could allow for a potential XSS-style attack. This can be achieved by encoding the javascript code using UTF-7. When a clients webbrowser attempts to autodetect the type of encoding used, it will detect it as UTF-7, and the javascript code can then be executed.

Yeap! I know it has been a very long time that i have posted something in this corner of web.. But i am extremely sorry, quite a bit busy with my college work. In the near future, its hopeless, that i am going to post something too.  I am extremely sorry for that…

But i assure you that i’ll post at least one post per week…

I am also partially busy because I am currently trying to make my Nokia N72 mobile, which doesn’t have native unicode support, to support unicode (Tamil). This is making me to break my head. If you have any idea or suggestions for that please contact me.

That’s all for now… Have a nice day…

To have a better idea about what is Unicode, let us first look at what is a CodePage.

CodePage

A Code page is another name for character encoding (A character encoding system consists of a code that pairs each character from a given repertoire with something else, such as a sequence of natural numbers, octets or electrical pulses, in order to facilitate the transmission of data (generally numbers and/or text) through telecommunication networks or storage of text in computers).

Code Page

Some relevant Terms:

Character: a, b, c, A, B,…

Coded character: A=65, B=66,…

Character Set: A set of characters, to be used together (e.g. Latin alphabet)

Code page: A set of coded characters (e.g. ISO-8859-1, Shift-JIS)

Locale: Code page + properties and rules (e.g. isdigit, collation, …)

Why the Need for Unicode

Every standard code page supports only a certain group of languages (e.g Western European, Eastern European, Japanese).

Within one computer system only one code page can be supported in a clean way. Therefore a universal code page that supports all letters, punctuation signs, technical symbols etc. of all languages is required.

Unicode Features

Unicode is a superset of all existing character sets. Unicode encodes plain text (no rendering information). It defines characters, not glyphs (semantics, not visual representation). Unicode unifies characters used in different scripts (CJK* Unification; CJK= Chinese, Japanese, Korean).

In Unicode there is a space for 1,000,000 characters. 64,000 characters coded by one 16bit code point. Further characters coded by two 16bit code points (surrogates).

Unicode - Detailed view

Unicode Encoding Forms

UTF-8

byte-based encoding scheme; one character is coded with 1-4 bytes; compatible with 7-bit ASCII.

UTF-16

16bit units; often used characters occupy one 16bit unit; further characters are coded with two 16bit units

UTF-32

32bit units; fixed size for all characters

Note: All encoding forms support the same amount of characters.

Unicode in SAP

UTF-8

for external communication (e.g file, network); no endian problems; minimum average data size; limited backward compatibility to non-Unicode systems.

UTF-16

internal (in memory); best compromise between memory usage and algorithmic complexity; fits to Java and Microsoft environment; best way to migrate existing ABAP and C programs.

Следва поредното извинително съобщение, породено от липса на интернет връзка и на компютър, както и от противоречията в ежедневното съществуване и мислене на автора сред търсенията и поривите на другите хоминиди.
Този четвъртък, семейството ни се сдоби с нова ISP услуга на мястото на старата. Услугата е неограничен по трафик месечен интернет достъп, Евроком НЕТ е името, а тарифата е най-евтината в листата. Факт е, че върши добра работа, а достъпът е чрез кабел, разклонен от този за кабелна телевизия и завършваш с модем, като последният продължава с пореден кабел към компютъра. За щастие, основната ни машина пристигна успешно вкъщи, след еднодневна сервизна поправка от страна на Multirama Пловдив, от които закупихме тази PC конфигурация през юни 2008. Поначало смятахме, че проблемът е от вентилаторите, поради ужасен стържещ шум, но изглежда се оказа, че и видеокартата е била опасно разместена или нещо подобно, звучеше сериозно, но техниците го поправиха за по-малко от ден и сега помощникът ми е отново в действие.
Междувременно забелязах, че някой

Thanks to dino kokalis for Bulgarian translation.

е изпратил своя български преводен файл за PhotoScape и този превод е включен в официалната версия 3.3 на програмата. Явно някой ме е изпреварил и е открил за себе си поредната добра безплатна/с отворен код програма и е побързал да я побългари… При бързия преглед на превода
(може да изисква Character Encoding = Cyrillic( Windows-1251) за правилно рендване на текста) и сравнението му с оригиналните английски низове, ми стана ясно, че авторът на българските низове се е постарал, но изглежда има място и за поправки тук таме – на места има правописни грешки, а на други се сещам за по-подходящи начини за превод на съдържанието на някои от съобщенията, изскачащи при кръжене с курсора(т.нар. ToolTips messages), както и на други по-важни места. Може би ще пооправя тези малки пропуски и/или ще пратя свой вариант за превод(на текстовия интерфейс на менютата на програмата, накратко – локализация/интернационализация) скоро.
Дотогава, дали да не се заема с превод на английските низове(strings) от англ. езиков файл на безплатната Greenfish Icon Editor Pro?   Вероятно пак ще се забавя, но ще го направя. Все пак, няма перфектно локализиран езиков файл, всички се учим и напредваме постепенно. Проектът HUTPIB е в застой, но все ще се намерят родолюбиви люде готови безплатно да локализират безплатна програма, която харесват и използват с удоволствие(или от нужда за изпълнение на специфична функционалност).
EDIT(16 April 2009): Започнах с поправянето на грешките от вече съществуващия преводен файл, ще спомена тук когато съм готов и ще изпратя файла на екипа на PhotoScape, които при добро желание ще го пуснат за изтегляне от секцията им с преводи. Разбира се, веднъж приключил с това, ще се връщам понякога към превода, за да го усъвършенствам и да премахвам грешки.
Прогрес – 133/1046 реда завършени.

♂♥♦→☻¢♠<œ•••☻

I really, really detest the Market Felt typeface used in the Notes application in the iPod touch and iPhone. It is hard to read, hard to edit points into, and looks plain childish.

Fortunately, I stumbled upon a way to easily have your notes displayed with Arial Helvetica. And, no, you do not need FontSwap, jailbreaking or any complicated deep system maneuvering. All you need is right there in the iPhone OS.

For this walkthrough I am using a second-generation iPod touch with firmware version 2.1.1 loaded. This also works with the iPhone.

1. In the Home Screen, tap on Settings.
   
2. Tap on General, followed by Keyboard (it is near the bottom of the page).
    

   

3. Tap on International Keyboards.
   
4. Scroll down to the end of the page and tap on Chinese (Simplified).
   
5. In the page that comes up, you will see two settings, Handwriting and Pinyin. Turn both settings on.
   
6. Go back to the Notes application. Create a new note or tap on an existing note to open it. If you are creating a new note, write a word or two so that you can easily verify that the typeface changes.
7. Tap in the note area to bring up the on-screen keyboard. Tap on the button to the left of the space bar, the one with an icon of a globe. The keyboard layout will change to one meant for Chinese handwriting recognition input. Don’t worry; just pay attention to the next step.
   
8. On the left of the input area, you will see four rows of buttons. The first button at the top is the Backspace. The second button below it is what we are interested in. For those of you who do not understand Chinese, the button says ‘Space’.
   
9. Tap on this button. The typeface of your note will change from Marker Felt to Arial Helvetica! But do read on! The next step is very important.
   
10. Tap the globe button twice, or if you have many input languages set up, tap it till the space bar flashes the words ‘English (US)’ (or the name of your native language). Your input language is now back to default, and you can continue to edit your note. Even if you close your note, the typeface will remain as Arial. That’s it!

Unfortunately, you will have to repeat the last few steps (7-10) for each new note you create, but this method does keep you from having to modify core iPhone files.

Care to know what just happened?

There are two ways to input Chinese characters on a computer, either by handwriting recognition or by a method known as Pinyin, in which a user spells out the phonetic pronunciation of Chinese words with the Latin alphabet on a standard keyboard. Each resulting Chinese character is then encoded—popularly with an encoding method called Unicode—into a document. This goes for the input of any other non-Western language as well.

Encoding non-Latin characters into a document requires a compatible font. Luckily for us, Market Felt is not a Unicode font. So, by picking a non-Latin input method, we are forcing the input engine to switch to a Unicode font such as Arial Helvetica so that it can display both Latin and non-Latin characters correctly. If you would like to know more about the magic that goes on in the background regarding character encoding, Wikipedia has an entry on the topic.

Note: This article has been corrected since publication.

Have you ever had problems finding HTML codes for special characters like these (♠ ♣ ♥ ♦ ) fellas? Or have you searched for simple ASCII codes or unicode encoding? I found this page called lookuptables.com yesterday and it has quite a few different tables you can easily print out or visit when you need to know something. It even has stuff like encoding for braille. It’s quite handy because all those different code tables are on one website, easily accessable. Check it out.

Die Programmiersprache PHP verarbeitet bis Version 5 nur 1-Byte-Zeichen. Strings, die etwa UTF-8 codierte Zeichen enthalten, werden von den Standart-Stringfunktionen oft fehlerhaft verarbeitet.

Die Funktion utf8_decode() bietet eine rudimentäre Möglichkeit, UTF-8 Strings in 1-Byte-Strings zu konvertieren. Dabei kann allerdings Informationsverlust auftreten.
Mit den Multibyte-Stringfunktionen bietet PHP allerdings eine mächtige Methodensammlung, um etwa UTF-8-Strings zu verarbeiten. Mit Hilfe von Regular-Expressions können damit die meisten Stringmanipulationen geleistet werden. PHP muss, um die Multibyte-Stringfunktionen nutzen zu können, mit der mbstring-Erweiterung kompiliert werden.

Eine Alternative bietet die Scriptsammlung der splitbrain-Entwickler, deren Funktionen klassische Stringfunktionen simmulieren und dabei UTF-8-Strings verarbeiten können, ohne auf die mbstring-Erweiterung angewiesen zu sein.

Da die Sammlung keinen Ersatz für str_word_count() enthält, habe ich mal einen geschrieben.
utf8_str_word_count():

/**
* Works like str_word_count() for UTF8-strings
* does not support locale information
* does not exclude "'" or "" on beginning of a word
* may behave strange if $format is not 0, 1 or 2
*/
function utf8_str_word_count($string,$format=0,$charlist='') {
$array = preg_split("/[^'\-A-Za-z".$charlist."]+/u",$string,-1,PREG_SPLIT_NO_EMPTY);
switch ($format) {
case 0:
return(count($array));
case 1:
return($array);
case 2:
$pos = 0;
foreach ($array as $value) {
$pos = utf8_strpos($string,$value,$pos);
$posarray[$pos] = $value;
$pos += utf8_strlen($value);
}
return($posarray);
}
}


Achtung, die Funktion verwendet weitere UTF-8-Funktionen (utf8_strpos und utf8_strlen) aus der Scriptsammlung von splitbrain.

Die Entstehung dieser Funktion ist in diesem Forenbeitrag dokumentiert.

Auf nicknettleton gibts noch wichtige Tipps, um das gesamte System inkl. Datenbank auf UFT-8 umzustellen.

Ein String ist eine Folge von Zeichen. Dieser kann von einer Programmiersprache in Variablen gespeichert und mittels Funktionen weiter verarbeitet werden.
Es gibt allerdings unterschiedliche Arten, wie Zeichen abgespeichert werden, character encodings genannt. Ist ein String etwa als ASCII gespeichert, können darin nur Zeichen aus der ASCII-Zeichentabelle verwendet werden. Jedes Zeichen nimmt dabei ein Byte Speicherplatz ein.

Da in den Sprachen der Welt eine Fülle verschiedenster Zeichen zur Anwendung kommen, wurden zur Standartisierung der Speicherung zunächst der ISO-Standart, später Unicode entwickelt.
UTF-8 ist der aktuelle Standart zur Speicherung von Zeichen. Er enthält eine Vielzahl von Zeichen aus vielen Sprachen. Zur Speicherung werden ein bis acht Byte verwendet.

If you are looking for some quick help in using a large character repertoire in HTML authoring, see the document Using national and special characters in HTML.

Several technical terms related to character sets (e.g. glyph, encoding) can be difficult to understand, due to various confusions and due to having different names in different languages and contexts. The EuroDicAutom online database can be useful: it contains translations and definitions for several technical terms used here.

-> tutorial <-

In the earlier blog, the character not being displayed properly by the browser is the browser issue and the code to deal with supplementary character is correct. The link http://www.unicode.org/cgi-bin/GetUnihanData.pl?codepoint=10177 shows the character that will be displayed in browser(which is the one present in xml file when opened by browser). So, to work with supplementary characters, create new String by passing character[] containing the low and high surrogate pairs or the byte[] specifying the encoding. The byte[] can for any code point can be obtained using the link :
Byte for codepoint

public class UnicodeTest{
public static void main(String… args) throws IOException
{
System.out.println(“is valid codepoint : ” + Character.isValidCodePoint(0×10FFFF));
System.out.println(“is valid codepoint : ” + Character.isValidCodePoint(0×20FFFF));
int cp = 0×10177;
System.out.println(“is valid codepoint : ” + Character.isValidCodePoint(cp));
char[] ch = new char[2];
ch = Character.toChars(cp);
int low = ch[0];
int high = ch[1];
System.out.println(“Low Surrogate Pair : ” + low + ” Hexadecimal : ” + Integer.toHexString(low) + ” Binary String : ” + Integer.toBinaryString(low));
System.out.println(“High Surrogate Pair : ” + high + ” Hexadecimal : ” + Integer.toHexString(high) + ” Binary String : ” + Integer.toBinaryString(high));
String st = new String(ch);
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(“krishna.xml”), “UTF-8″));
out.write(“”);
out.write(“”);
out.write(st);
out.write(“”);
out.close();

}
}

In the above case, the supplementary character written in xml was not the one it is intended to be. Don’t know whether it is the limitation of browser to display the supplementary characters or anything wrong in the way a supplementary character be handled in java code.

From the start, java had used UTF-16 encoding for encoding the characters. Thus, in the earlier stages when the unicode character set was limited to 16 bits and hence was given full support by java character which was using the utf-16 encoding. Once the unicode was extended to support till the range U+10FFFF, the earlier UTF-16 encoded characters cannot represent characters more than U+FFFF. Hence, in J2se5, support was provided through the Character class. So, the primitive char still supports only the characters till code point: UTF+FFFF. The Java 2 platform uses the UTF-16 representation in char arrays and in the String and StringBuffer classes. In this representation, supplementary characters are represented as a pair of char values, the first from the high-surrogates range, (\uD800-\uDBFF), the second from the low-surrogates range (\uDC00-\uDFFF).

A char value, therefore, represents Basic Multilingual Plane (BMP) code points, including the surrogate code points, or code units of the UTF-16 encoding. An int value represents all Unicode code points, including supplementary code points. The lower (least significant) 21 bits of int are used to represent Unicode code points and the upper (most significant) 11 bits must be zero. Unless otherwise specified, the behavior with respect to supplementary characters and surrogate char values is as follows:

* The methods that only accept a char value cannot support supplementary characters. They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter(‘\uD840′) returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.
* The methods that accept an int value support all Unicode characters, including supplementary characters. For example, Character.isLetter(0×2F81A) returns true because the code point value represents a letter (a CJK ideograph).

Again got confused in unicode. Some of the terms used are:
1. Coded Character Set
A character Set(collection of characters) where each character has been assigned a unique number. E.g., Unicode character set, where every character is assigned a hexadecimal number.
2. Code Points
The numbers that can be used in a coded character set. Valid code points for Unicode character set is : U+0000 to U+10FFFF (Unicode :4 standard)
3. Supplementary Characters
Characters that could not be represented in the original 16-bit design of Unicode. U+0000 to U+FFFF are referred to as Base Multilingual Plane(BMP) and the others are supplementary characters.
4. Character Encoding Scheme
Mapping from the numbers of one or more coded character sets to sequences of one or more fixed-width code units. e.g., UTF-32, UTF-16, and UTF-8
4. Character Encoding
Mapping from a set of characters to sequences of code units. e.g., UTF-8, ISO-8859-1, GB18030, Shift_JIS.

UTF-16
UTF-16 uses sequences of one or two unsigned 16-bit code units to encode Unicode code points. Values U+0000 to U+FFFF are encoded in one 16-bit unit with the same value. Supplementary characters are encoded in two code units, the first from the high-surrogates range (U+D800 to U+DBFF), the second from the low-surrogates range (U+DC00 to U+DFFF). This may seem similar in concept to multi-byte encodings, but there is an important difference: The values U+D800 to U+DFFF are reserved for use in UTF-16; no characters are assigned to them as code points. This means, software can tell for each individual code unit in a string whether it represents a one-unit character or whether it is the first or second unit of a two-unit character. This is a significant improvement over some traditional multi-byte character encodings, where the byte value 0×41 could mean the letter “A” or be the second byte of a two-byte character.

Check the JSR-204 for java unicode support : JSR-204

Supplementary Character Support Approach

• Use the primitive type int to represent code points in low-level APIs, such as the static methods of the Character class.
• Interpret char sequences in all forms (char[], implementations of java.lang.CharSequence, implementations of java.text.CharacterIterator) as UTF-16 sequences, and promote their use in higher-level APIs.
• Provide APIs to easily convert between various char and code point based representations.

Good blog on unicode support in j2se5 : John Conner blog
Highlights:
# char is a UTF-16 code unit, not a code point
# new low-level APIs use an int to represent a Unicode code point
# high level APIs have been updated to understand surrogate pairs
# a preference towards char sequence APIs instead of char based methods

The UTF-8 encoding rules

1. Characters U+0000 to U+007F (ASCII) are encoded simply as bytes 0×00 to 0×7F (ASCII compatibility). This means that files and strings which contain only 7-bit ASCII characters have the same encoding under both ASCII and UTF-8. A single byte is needed for any of these characters!
2. All characters >U+007F are encoded as a sequence of several bytes, each of which has the most significant bit set. Therefore, no ASCII byte (0×00-0×7F) can appear as part of any other character.
3. The first byte of a multibyte sequence that represents a non-ASCII character is always in the range 0xC0 to 0xFD and it indicates how many bytes follow for this character. All further bytes in a multibyte sequence are in the range 0×80 to 0xBF. This allows easy resynchronization andmakes the encoding stateless and robust against missing bytes
4. All possible 231 UCS codes can be encoded
5. UTF-8 encoded characters may theoretically be up to six bytes long, however 16-bit characters (the ones implicitly supported in the UCS-2 encoding, and by Str Library) are only up to three bytes long
6. The sorting order of Bigendian UCS-4 byte strings is preserved
7. The bytes 0xFE and 0xFF are never used in this encoding

The following byte sequences are used to represent a character:

U-00000000 – U-0000007F: 0xxxxxxx
U-00000080 – U-000007FF: 110xxxxx 10xxxxxx
U-00000800 – U-0000FFFF: 1110xxxx 10xxxxxx 10xxxxxx
U-00010000 – U-001FFFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
U-00200000 – U-03FFFFFF: 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
U-04000000 – U-7FFFFFFF: 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

The xxx bit positions are filled with the bits of the character code number in binary representation. The rightmost x bit is the least-significant bit. Only the shortest possible multibyte sequence which can represent the code number of the character can be used. Note that in multibyte sequences, the number of leading 1 bits in the first byte is identical to the number of bytes in the entire sequence.