UTF-8, gettext, embed and benchmarks

1. restricting the extensions in use etc. to only thread-safe ones was AFAIK not on the agenda when g2 was initially designed. also, it doesn't make much sense, even now, to get such a burden, there are still too many non thread-safe extensions.

2. yes that's true. that's why we have an entry in our faq which explains this fact. installing further languages on the system isn't a difficult task.
but it's true, gettext as an (external) dependency isn't that great. we looked for php based gettext replacements, and there are at least two of them, but they are very slow.

3. internally, g2 works 100% in utf-8. browser input is utf-8, output is utf-8, interactions with the filesystem are translated from/to utf-8.
you're mentioning issues with usernames in utf-8 and non-mb*() functions.
i'm not sure if we actually have such issues. got an example?

have you checked out how g2 embedding works?
g2 and your CMS don't share the same user tables. they are loosely coupled and kept in sync'.

Tests done:

Local machine
P4 2.54Ghz
Windows OS
PHP 5.0.4 (no accelerators)
Default new G2 install (completely empty)

With gettext page generated in 0.6541 seconds
Without gettext page generated in 0.5053 seconds
[edit]
My CMS on this machine generates pages in 0.0731 seconds
[/edit]

http://urbanduck.net/gallery/main.php
p4 2.8
Linux OS
PHP 4.4.0
with gettext 5.1533 seconds first time and 1.326 seconds on a page refresh

http://historicalengine.org/gallery/main.php
Dual XEON
Linux OS
PHP 4.4.0
with gettext 0.6209 seconds

Modified main.php is attached below

djmaze, you work for coppermine? no, i didn't know that, but i don't really care cool to be involved in multiple projects

integration / embedding:
as yo our way of integration / embedding...
i did a lot of research in this field last fall / this spring and this was the result. there's this thing i call tight integration (sharing db tables) and the other approach, loose coupling / integration. i tried first the former approach, because it's more obvious and sounds good. i ran into too many issues and realized that the second approach is much cleaner / better than it originally looks like.

modern CMS have events / hooks. e.g. xaraya, drupal, mambo, typo3, wordpress, ... all allow to register some function such that it is executed when such an event occurs.
we use this for user create / update / delete (and also group create / update / delete) events.
each time a user is created in the CMS, our function is called and it automatically creates a corresponding user in G2 and maps the CMS with the G2 user in a mapping table. loosely coupled, but always in sync' and there are no major disadvantages.

so you see, it isn't actually synchronization / periodic sync' stuff, it is a real integration and the two systems don't get out of sync'.

i'll have to write down all the pro's and con's and issues i ran into when playing with the two approaches.

gettext
what's your suggested alternative to gettext? we need a very flexible translation, want to support i18n.

I agree that using a pure PHP solution is going to be far faster than using gettext. However, I have yet to see a pure PHP solution that provides all the functionality of gettext and can touch it on speed.

For example, if you want to translate the following message:

"I have %d apple"

you also have to translate the plural form:

"I have %d apples"

In English this is no problem. There are two plural forms, and the rule for them is easy. But how about in Gaelic which has 5 plural forms? Or Polish which has 3 plural forms that are totally different? It is tricky to get this right. Before I started using gettext for G2 I investigated a few other solutions and none of them properly solved the pluralization problem.

If you can derive a localization scheme that properly covers pluralization, we'd certainly consider it.

There's a reimplementation of gettext in pure PHP here:
http://savannah.nongnu.org/projects/php-gettext/

This is more portable, but from my benchmarks it's far slower, which is to be expected because it's competing against a pure C library.

Yup, that's all we need. We specify two plural forms for english (one vs. many) and provide a count. Localizers use a plural forms definition to describe how many plural forms that they have and then create a localization accordingly, and gettext chooses the correct form based on the rule. The rules get quite complicated, eg:
http://www.gnu.org/software/gettext/manual/html_mono/gettext.html#SEC150

This is a little simplistic, since you'd have to apply a rule to get the specific translation here. But yes, something along those lines should do it.

I agree -- this sucks. I haven't found a solution anywhere that handles this properly. We are forced to break it up into two strings with two different counts and then merge the strings together in order to make this work (and that's always error prone).

The other big advantage we get from gettext is that it's a standard format which means that there are lots of tools out there to audit .po files, assist in translation, do reporting on them, etc. We've definitely found that useful in the past.

I'm not married to gettext -- I'll happily use a better solution if one comes along. As you've seen in our code, we have a single place where we make all gettext calls so it would be easy for us to switch to a different localization scheme if there's a better one around. Certainly php-gettext looks like a viable alternative. We'd very much prefer to keep the gettext .po format since we have a considerable amount of localized data in that format, but I imagine we could write a tool to migrate it to a different data structure if a better one came along.

If you can help us dig up a thread safe replacement for gettext that matches it on translation features (a must), has comparable performance, and has some kind of reasonable tools support I'd strongly consider switching. I hate the fact that we're bound to the system locales!

I have a question about the plural form. In Chinese there's not difference between singular and plural. Is it possible to specify it in the PO files? Currently I translate both the msgstr[0] and msgstr[1] to the same string. It seems to be a waste.

Thanks So what's the link to your new CMS under development? I'll keep tabs on the approach you're taking and see if we can benefit from something along the same lines. And come on by #gallery on freenode if you want to talk about this in real time.

I tried it on my own site and it seems to work. So I updated and submitted the change. Hope I didn't just break someone's site.

valiant, will do. Have to get the kids to school right now. First day of school. %&*#%^&...

Finally i've figured it out and the solution is pretty simple.
i use $LNG array with values like:
$LNG['year'] = array('%d year', '%d years');

now here's my ngettext equivelant

function nget($var, $n)
{
	$txt = $LNG[$var];
	$i = ($n == 1 ? 0 : 1);
	if (!empty($txt['plural'])) {
		$total = count($txt)-2;
		$plural = str_replace('n',$n,$txt['plural']);
		eval("$i = ($plural)");
		if ($i > count($txt)-2) { $i = count($txt)-2; }
		elseif ($i < 0) { $i = 0; }
	}
	return sprintf($txt[$i], $n);
}

Now the trick:
when your language isn't of the default 2 forms then you add another entry in the array with the key 'plural'

$LNG['year'] = array(
	'%d year',
	'%d years',
	'%d decades',
	'plural' => 'n==1 ? 0 : (n%10!=0 ? 1 : 2)'
);

This will result in:

nget('year', 1): 1 year
nget('year', 2): 2 years
nget('year', 10): 10 decades

ofcourse it's just an example, but you get it.