[Mod] Internationalization Library [intllib]

[Wuzzy]

Oh crap, oh crap, oh crap, oh crap, oh crap, oh crap, oh crap, oh crap, oh crap, oh crap!

I just made some tests.
First, I tested intllib alone in the Empty subgame (subgame without mods). It works.
Now I added one dependant mod: teletool.

And ta-da! The error appears. This is the error you asked for:

Code: Select all

2016-11-09 00:13:32: ERROR[Main]: ModError: Failed to load and run script from /home/wuzzy/src/minetest/git/minetest/bin/../mods/teletool/init.lua:
2016-11-09 00:13:32: ERROR[Main]: Attempt to access external file /home/wuzzy/src/minetest/git/minetest/bin/../mods/intllib/intllib.lua with mod security on.
2016-11-09 00:13:32: ERROR[Main]: stack traceback:
2016-11-09 00:13:32: ERROR[Main]: 	[C]: in function 'dofile'
2016-11-09 00:13:32: ERROR[Main]: 	.../src/minetest/git/minetest/bin/../mods/teletool/init.lua:3: in main chunk
2016-11-09 00:13:32: ERROR[Main]: Siehe debug.txt für Details.

And what is at line 3 of teletool/init.lua, you ask? Look here:

Code: Select all

local S
if (minetest.get_modpath("intllib")) then
	dofile(minetest.get_modpath("intllib").."/intllib.lua")
	S = intllib.Getter(minetest.get_current_modname())
else
	S = function ( s ) return s end
end

(full source code)

The offending code is quite obviously the “dofile”. This is a blatant violation of mod security because a file from outside the teletool mod is attempted to be loaded.
Oddly, this is not even the boilerplate code from the intllib README. But when I looked on earlier versions of intllib, the dofile was still there. Probably the code here was just copied from an older version of intllib.
When I use the new intllib boilerplate, it works fine. The fix is (luckily) quite simple.

But the “Oh crap!” part of this is that I have found many mods in my mod directory which currently contain the bad “dofile”:

Code: Select all

awards
food_sweet
homedecor
maptools
prehistoric_modpack

bedrock2
dice2
doc_minetest_game
hbarmor
hbhunger
hudbars
inventory_icon
magicbeans_w
mana
mtg_plus
orienteering
origin
pedology
pep
pseudonodes
returnmirror
teletool
tsm_surprise
wateringcan

All these listed mods will break with mod security AND intllib enabled. Quite a sneaky bug. And yes, many of my own mods are on the list! :-(

But I have planned to do a Minetest mod release orgy in this or the next week anyway. I will “spam” the forums with mod updates at this time. So I am quite glad I found the bug BEFORE that. :-)

[kaeza]

Yeah, changes always break something. Only solution is to nudge these modders a bit to update their mods.

[Wuzzy]

Whew. I just updated all my affected mods. They should now all work with intllib AND mod security enabled.
Some of the mods not from me were just outdated on my macihine. After an update, the bug was gone. I informed the authors of the remaining mods.

It looks bad. Very, very bad. A GitHub search for the offending code gives over 500 results:
https://github.com/search?utf8=%E2%9C%9 ... rchresults

There are some false-positives here, but many hits are justified. But on GitHub many projects are also just dead, those can be ignored as well.

Still, this does not look good. :-(
It would take ages to inform all the modders. It seems that basically all mods which supported intllib early on are basically broken.
Just fixing mods is not enough. Modders must be made explicitly aware of the change. I bet many modders just copy-paste the boilerplate code from another mod because they are too lazy to open GitGub. xD And so the broken code spreads. :-(

And yes, you are right. The breakage was required. There was no other way. Sadly. :-( The alternative would be having no mod security. Which is of course crazy. ;-)

I bet the real fun begins when the next Minetest gets released. Then, mod security is enabled by default. xD

Sigh. I fear it will be a lot of work to fix all those mods. Let's hope not too many modders abandon Minetest too early.

[ShadowNinja]

[intllib-using mods fail security tests]
Oddly, this is not even the boilerplate code from the intllib README. But when I looked on earlier versions of intllib, the dofile was still there.

The dofile was basically just a way to make sure intllib was already loaded, but Minetest already has a mechanism for that: depends.txt. So I removed that hack and now you're just supposed to add intllib to depends.txt -- although I left the old intllib.lua file there for compatibility. Just make sure you add that the depends.txt line in your patches if it isn't already there.

[Wuzzy]

Ah, thanks for the background on this. And yeah, all my intllib-dependant mods have “intllib?” in depends.txt already, but thanks for the reminder.

[kaeza]

Sort of important update!

Intllib now supports using gettext message catalogs for translation.

The old conf-like interface is still available. You will need to change some stuff to support the new features. Please check out the docs for details.

[burli]

Is there somewhere a tutorial how to use intllib? I try to add it, but it doesn't work. And if I use Getter I get this message

Code: Select all

WARNING[Main]: intllib.Getter is deprecated.Please use intllib.make_gettext_pair instead.

I want to use and support this

Edit: It's working now. And I just replaced Getter with make_gettext_pair

[kaeza]

Is there somewhere a tutorial how to use intllib? I try to add it, but it doesn't work. And if I use Getter I get this message

Code: Select all

WARNING[Main]: intllib.Getter is deprecated.Please use intllib.make_gettext_pair instead.

I want to use and support this

Edit: It's working now. And I just replaced Getter with make_gettext_pair

There's some docs in doc/developer.md. If you have any doubts, feel free to ask.

EDIT: If this error is not from your own mod, kindly ask the mod developer to update. The warning can be safely ignored.

[Wuzzy]

Wow, holy shit! I just looked into the documentation for the new Gettext support and I am not amused:

1) A completely new boilerplate. Great, just great. Just a few months after I fixed the boilerplate of all my scripts! >_>
2) Forces me to include a entirely boilerplate Lua file in each of my scripts, with no guarantees it won't change in future

Sorry, my motivation to update to Gettext is low, especially since I have no idea if you will change the boilerplate in a breaking manner in the future, forcing me to go through the whole treadmile a 3rd, 4th etc. time.

Its ok that the old syntax is still supported, but I really hate the fact that I must include a complete Lua boilerplate file which you will probably change and break in future as well. Is there any way you can get rid of large boilerplate? If not, can you at least promise you won't touch the boilerplate for a long period of time?

Sigh. Maybe it's better I just wait until Minetest finally gets real mod translation support.

PS: Don't get me wrong: I fully support the move to Gettext. I am just very annoyed because I feel all this “boilerplate updating work” is a huge timesink.

[kaeza]

The boilerplate was split for that very reason: so in case it needs to be updated (it probably won't be anytime soon), all you have to do is download the file from the repo which can be done from a shell script with `wget` or `curl` (the raw URL is at the top of the file).

The fact that `.txt` translations still work even with the new API kind of shows my commitment to maintain backwards compatibility as far as is possible.

That said, you are not required to update. Both formats have advantages and disadvantages. E.g. the gettext catalog (`.po`) format has excellent tools for working with it and supports plural forms properly; the conf-like (`.txt`) format is easier to edit by hand. Just choose whichever you feel is best for your project.

[Wuzzy]

Is there a way to automatically convert existing *.txt translations to *.po?

[bell07]

Is there a way to automatically convert existing *.txt translations to *.po?

You can write a small lua script for qa_block or adjust my script https://github.com/bell07/minetest-qa_b ... verter.lua

[Linuxdirk]

Wait a second ... Does that mean that intllib now supports standards compliant PO files?

[Wuzzy]

Yes. :-)

You can write a small lua script for qa_block or adjust my script https://github.com/bell07/minetest-qa_b ... verter.lua

This script should be included in the intllib repository. Thank you.

[Wuzzy]

Intllib will finally become irrelevant in 0.5.0:
viewtopic.php?f=18&t=18349

[codexp]

just in case someone needs a simple converter of txt2po, here is one:

https://gist.github.com/codexp/93243abe ... b7a870dd06

[snoopy]

Intllib will finally become irrelevant in 0.5.0:
viewtopic.php?f=18&t=18349

This may be somewhat confusing and seems to concentrate on the Minetest core mainly.

Are there any news on translations efforts and current developments?

[snoopy]

The boilerplate was split for that very reason: so in case it needs to be updated (it probably won't be anytime soon), all you have to do is download the file from the repo which can be done from a shell script with `wget` or `curl` (the raw URL is at the top of the file).

The fact that `.txt` translations still work even with the new API kind of shows my commitment to maintain backwards compatibility as far as is possible.

That said, you are not required to update. Both formats have advantages and disadvantages. E.g. the gettext catalog (`.po`) format has excellent tools for working with it and supports plural forms properly; the conf-like (`.txt`) format is easier to edit by hand. Just choose whichever you feel is best for your project.

I highly appreciate the efforts and the good advice from kaeza. I am aware of several mods which make use of the (`.po`) format and this provides nicely translated interface outputs. E.g. the very nice mods of TenPlus1 show a support to intllib in addition to an improved code base.

Furthermore, I am currently hacking a fork of this mod into the current minetest_game-5.0.1 code tree. This fork will be named game_intllib to avoid hassle with the client intllib mod.

My concept is to provide a central locale folder based inside the game_intllib mod. Currently I have introduced this code by an intllib.lua variant into all the applicable 25 game mods and the first results seem to be promising. However, one should be aware of certain demands to further adapt the game mod code to allow a more common translation (e.g. mod bucket and the name of bucket list).

Please note: Independantly from intllib and minetest_game-5.0.1 there is the Minetest core code. I would like to take the opportunity to express my sincere thanks to both the core developers and the translation efforts in the 5.0.1 core code and the excellent results given on the client interface.

Back to minetest_game-5.0.1 and translation:

I am aware the game developers may be reluctant and are quite occupied by the true coding of the game. I would like to express my sincere thanks to the game developers and their coding efforts for the excellent 5.0.1 game code. The stability and flexiblity make a joyful gaming experience.

Unfortunately, my concept has to touch most of the mods inside the game and has to change several lines of codes to allow the translation tagging as described at the Minetest Wiki.

Is there any advice on how to organise such a project?

Currently, I have established a hacked code tree of the game mods successfully running in a dedicated 5.0.1 server on macOS. A first template.pot provides the majority of translation tags already. My first de.po gives the expected results with game_intllib and no hassle to intllib for the other mods outside the game mods folder.

I am quite aware that any external mods have to get their translations too. However, this should be intllib standard procedure if I am correct. Furthermore, the future of the a.m. client-side translations seems to be debatable.

I make this post to get some advice and better insights. Before considering in providing the code and template to e.g. notabug I want to learn of opinions and how to proceede in a beneficial ways to the Minetest project. For this I would welcome any advice.

Last not least my proposal should be understood as a strong support for intllib. Thank you.

[Wuzzy]

just in case someone needs a simple converter of txt2po, here is one:

Oh, thanks! This will certainly come in handy. Now I just need a converter for the other way around (po2txt) and my life is complete. :)

-------------------

@snoopy:
The major disadvantage of intllib is that it's not for servers. The server will serve the same translation to all clients, i.e. everyone will will see Russian on a Russian server.
You can hack around this all you want, you just can't fix it at mod level.

The builtin client-side translations are the way to go, even if the format sucks at the moment, but it can be improved in future. The core devs actually agreed that we should move to PO, they just haven't figured out yet how to do this.

This may be somewhat confusing and seems to concentrate on the Minetest core mainly.

The reason why the client-side translations are not used in Minetest Game atm is because only a minority cares about Minetest Game anymore, not because the system is bad.

Also, to further destroy your FUD:
MineClone 2 has been using the client-side translations and they work just fine. Evidence: German MineClone 2 server. There. I have just proven to you that, yes, the built-in translation system does, in fact, work.

That said, you are not required to update. Both formats have advantages and disadvantages. E.g. the gettext catalog (`.po`) format has excellent tools for working with it and supports plural forms properly; the conf-like (`.txt`) format is easier to edit by hand. Just choose whichever you feel is best for your project.

Frankly, this is terrible advice. While it is true that nobody is forced to do anything, this does not mean that both formats are equally valid. The PO format is definitely better. The fact that you can edit the TR files “easier by hand” is a really minor advantage IMO and not really important and does not outweigh the major disadvantages. Besides, you can edit PO files by hand as well. While a toolchain is much, MUCH more important.
For TR, we still do not have even the most basic tools/scripts to automatically update our TR files. What a bummer.
That does not mean the builtin translation system is wrong, it just means it's missing important features. But at least I've seen an unofficial “string collection” script somewhere, no idea where it is now. I wish it would be finalized and made official one day.

[snoopy]

@snoopy:
The major disadvantage of intllib is that it's not for servers. The server will serve the same translation to all clients, i.e. everyone will will see Russian on a Russian server.
You can hack around this all you want, you just can't fix it at mod level.

The builtin client-side translations are the way to go, even if the format sucks at the moment, but it can be improved in future. The core devs actually agreed that we should move to PO, they just haven't figured out yet how to do this.

@Wuzzy: Thank you. However, there may be some misunderstanding.

Certainly, the PO-file is the preferred way as already lined out in my proposal. Please not that I am quite happy with the solution found for the 5.0.1 core and that I am aiming at the LUA-code of the exisitng game mods only.

I cannot deduce what is going on for client translation and the forum topic seems to be LOCKED. Unfortunately, you mention some 'prove' but besides a server address I cannot find further information shared on this in your comment.

Please don't get me wrong but I may not agree that much on the server disadvantage. Certainly, a local server may share just one preferred language to the clients and extending the minetest_game with extra mods seems to be a valid solution. This is just the use-case I mentioned and I intentionally want to limit the scope. My goal is to have a pragmatic solution available until a promised optimal solution may come around in the future.

From my experience, the default game together with the ethereal mod and other mods provides a very nice and quite sufficient gaming environment. Please note several basic nodes (e.g. 'bucket' / "Empty Bucket") will survive into the interface even when combined with several mods like farming_redo or mobs_redo.

This may vary for public servers and I would like to learn of the concepts there. However, my aim is the a.m. use-case and I would see not much of a problem in a 'one language only' server concept as an option.

The intllib mod is applying GNU gettext already. My game_intllib is just tweaking this code and the main effort is to introduce translation tags into the LUA-code of the game mods. The applicable gettext tool can already produce a template.pot from the relevant 25 game mods and this in a compatible way.

For the time being I take the liberty to presume that a public server concept would also need the appropriate translation tags in the LUA-code of the game mods anyway. Presumably using GNU gettext as a valid solution this may be almost similar to the limited concept in many aspects.

@Wuzzy: Would you be so kind to differentiate so I can deduce more information?

I would welcome shared information and an open discussion. Until further notice I cannot see that much of a conflict between the two concepts and I would continue for the time being.

In general one may consider the GNU gettext project & documentation for further information.

[snoopy]

@Wuzzy: You did provide some information by direct email to my account which I want to share here.

The core content of your email states that the "Minetest's builtin translation system" would need the enclosure of "all user-facing strings in S("string")" in "all "*.lua source files".

This is just the main effort I already mentioned in my proposal and which is already covered for the most relevant user-facing strings by my current intermediate solution and testing results of today. Please note the major part of this work is already done and is reflected by the template.pot together with intllib in a compliant format.

This is also the default procedure to establish translation tags by help of the gettext tools. You may reconsider my a.m. proposal and consult the a.m. gettext documentation.

@Wuzzy: Thank you. However, would you be so kind to provide entry points and/or sharable documentation sources for the apparent "Minetest's builtin translation system" please?

Notwithstanding other concerns of yours, I would deduce that this sufficiently proves my concept to be a valid procedure and there should be almost no conflict between the two concepts. Happy hacking.

[Linuxdirk]

The reason why the client-side translations are not used in Minetest Game atm is because only a minority cares about Minetest Game anymore, not because the system is bad.

The reason actually is that paramat personally is uninterested in working on translations for MTG.

MineClone 2 has been using the client-side translations and they work just fine. Evidence: German MineClone 2 server. There. I have just proven to you that, yes, the built-in translation system does, in fact, work.

Yes. I first was super skeptical about the weird, uncommon, and and non-standard implementation and the still missing proper toolchain, but yes. The translations work as expected (at least if you don’t compile RUN_IN_PLACE and start Minetest from a folder that contains a “locale” folder)

For TR, we still do not have even the most basic tools/scripts to automatically update our TR files. What a bummer.

If you have a converter for PO->TR you can still use the usual default translations toolchain and convert the PO file wafterwards. Back then I wrote a simple core Lua only parser with zero error tolerance and only supporting single-line translations but actually never used it.

[snoopy]

The reason actually is that paramat personally is uninterested in working on translations for MTG.

I am totally OK with this. Time is a limited resource and in a FOSS project like MT you have to set your priorites like in most if not all of your activities.

I am aware the game developers may be reluctant and are quite occupied by the true coding of the game. I would like to express my sincere thanks to the game developers and their coding efforts for the excellent 5.0.1 game code. The stability and flexiblity make a joyful gaming experience.

Big thanks to @paramat and other developers again.

[Linuxdirk]

The reason actually is that paramat personally is uninterested in working on translations for MTG.

I am totally OK with this. Time is a limited resource and in a FOSS project like MT you have to set your priorites like in most if not all of your activities.

Absolutely. That was no criticism, it was only information because I know some most people do not follow discussions on GitHub.

[snoopy]

Some results of my WIP in the meantime.

Thanks to @kaeza for intllib and the MT game developers for the minetest_game-5.0.1 again.

After tweaking with the code of intllib and establishing a new game_intllib in the minetest_game-5.0.1 mods tree the results are quite promising. The process to find and tag the appropriate user-facing strings is tiresome but straightforward.

In the short time working with the LUA-code of most if not all of the minetest_game-5.0.1 mods I have to admire the efforts and the excellent work of the developers.

Futhermore, the intllib by @kaeza is a reliable and helpful tool and seems to be both robust and compliant to the GNU gettext tool chain.

I was able to first produce the template.pot file and made a first de.po file from this. After finding more and more appropriate user-facing strings in the LUA-code of the other mods I was able to update both the template.pot file and the draft de.po file several times without hassle.

Code: Select all

$ cd ~/Minetest5-dev/minetest_game-5.0.1_dev00/mods

$ ./game_intllib/tools/xgettext.sh ./beds/beds.lua ./binoculars/init.lua ./boats/init.lua ./bones/init.lua ./bucket/init.lua ./butterflies/init.lua ./carts/cart_entity.lua ./carts/rails.lua ./creative/init.lua ./creative/inventory.lua   ./default/chests.lua ./default/craftitems.lua ./default/furnace.lua ./default/nodes.lua ./default/tools.lua ./default/torch.lua ./default/trees.lua ./doors/init.lua ./dye/init.lua ./farming/api.lua ./farming/hoes.lua ./farming/init.lua ./farming/nodes.lua ./fire/init.lua ./fireflies/init.lua ./flowers/init.lua ./game_commands/init.lua ./map/init.lua ./screwdriver/init.lua ./sethome/init.lua ./stairs/init.lua ./tnt/init.lua ./vessels/init.lua ./walls/init.lua ./wool/init.lua ./xpanes/init.lua
Generating template in ./game_intllib/locale/ folder...
Updating ./game_intllib/locale//de.po...
........................................ fertig.
DONE!

The template.pot file and the PO-files are located at the mods/game_intllib/locale/ folder in the minetest_game-5.0.1 mods tree. All applicable mods have a copy of the intllib.lua file and an adjusted depends.txt respectively.

Please find attached as ZIP the template.pot file as an intermediate WIP result.

template.pot.zip

The draft template.pot file as an example of the results.

(3.66 KiB) Downloaded 175 times

The the minetest_game-5.0.1 and the minetest 5.0.1 client are running the tweaked LUA-code and the MT client GUI shows a good mix of translated and yet to be translated strings. With currently 1620 lines of text in the POT-file, a 20 lines POT-header and around 4 lines per translation entry the estimate of currently 400 user-facing strings in the LUA-code already covered is promising.

This means I currently introduced the appropiate translation tags by means of the S("string")" enclosure to 400 user-facing strings in the LUA-code of the applicable mods in the following files:

1. ./beds/beds.lua
2. ./binoculars/init.lua
3. ./boats/init.lua
4. ./bones/init.lua
5. ./bucket/init.lua
6. ./butterflies/init.lua
7. ./carts/cart_entity.lua
8. ./carts/rails.lua
9. ./creative/init.lua
10. ./creative/inventory.lua
11. ./default/chests.lua
12. ./default/craftitems.lua
13. ./default/furnace.lua
14. ./default/nodes.lua
15. ./default/tools.lua
16. ./default/torch.lua
17. ./default/trees.lua
18. ./doors/init.lua
19. ./dye/init.lua
20. ./farming/api.lua
21. ./farming/hoes.lua
22. ./farming/init.lua
23. ./farming/nodes.lua
24. ./fire/init.lua
25. ./fireflies/init.lua
26. ./flowers/init.lua
27. ./game_commands/init.lua
28. ./map/init.lua
29. ./screwdriver/init.lua
30. ./sethome/init.lua
31. ./stairs/init.lua
32. ./tnt/init.lua
33. ./vessels/init.lua
34. ./walls/init.lua
35. ./wool/init.lua
36. ./xpanes/init.lua

This is much fun.

I would welcome any advice on how to improve the MT translation efforts. Until further notice I cannot see that much of a conflict between the two a.m. concepts and I would continue my little project for the time being.