My first experience with the threaded mapgen environment (MT 5.9.0)

[Wuzzy]

Turns out the new feature in Minetest 5.9.0, called "threaded mapgen environment", is easier to use than it sounds.

So a few weeks ago I successfully integrated the [islands] mapgen by TheTermos into Lazarr!
But lately, I noticed its loading times can screw up the spawning. The spawning area shows up too late, which was annoying.

Then I remembered the new Minetest feature about threaded mapgen and finally decided to give it a try. The rationale for this was that normal Lua mapgen slowed down the server because it was not threaded. So the threading feature was exposed to Lua. This makes complete sense. Great.

So I thought it will be difficult but it was not. The mapgen of the game is thankfully simple, so I just had to change few lines of code.

(By the way, here's the finished file: mapgen.lua)

Basically, these steps were needed:

1) Clean the mapgen code of any complex stuff that would
2) Instead of dofile-ing it, call it with mapgen.register_mapgen_script
3) Change minetest.register_on_generated because in the mapgen environment, this function takes a vmanip object as first parameter
4) Remove voxelmanip:write_to_map because this is not permitted nor needed in the mapgen environment

And that was basically it to transform my game, the mapgen already worked!


Well … Except there was a big thing missing. In my game, I use a function called generate_piece to put all the mapgen logic in. It's a global function because I also want to be able to re-generate a piece of the map when I reset a level, and this of course happens after mapgen time. My game is built in a way that levels can be put in (almost) any part in the world and be removed again.

So this created a small problem: Now my mapgen code is stuck in the mapgen environment, so how do I allow levels to call generate_piece again?

I briefly considered doing something extreme: Using a macro system like m4 for the first time in my life.

But I found a better solution:

I simply run the same script twice! Once with dofile in the global environment, once with minetest.register_mapgen_script. And in the script, I modified the code by checking if the code runs either in the global, or the mapgen environment. How do I check it? Like this:

Code: Select all

local IS_IN_MAPGEN_ENVIRONMENT = not minetest.global_exists("lzr_mapgen")

If lzr_mapgen exists (which is created by the mod), I can deduce the script must run in the global environment, because I can access my own globals in that environment, including lzr_mapgen. But if it does not exist, the only reason for that is that I'm in the mapgen environment, since it can't see globals.

The script now does two things:
1) In the mapgen environment, calls minetest.register_on_generated (but the "special" one)
2) In the global environemnt, expose generate_piece into the global namespace so it can be used by the levels mod to regenerate a piece.

Now the only thing left to do is to make the small code adjustments with a few if statements to change the behavior of generate_piece.

Whew!

This setup now works, and I'm happy. There is only one bad thing left: I was forced to copy a global variable lzr_globals.DEEP_OCEAN_Z into the script, which means the game now has two copies of the value, which is redundant and ugly. Since it's not possible to give any data to the mapgen environment (apparently), I'm afraid there's no workaround.

Thankfully, it's only one variable, and I added BIG warnings to both copies to prevent me to inconsistently change the value when future me decides to have a brainfart. ;-)

After I did all of this, the game indeed seems to startup much faster, the critical delay at the start is basically gone. Nice!

------------------

One final comment: It's annoying I cannot pass any data to the mapgen environment. Not even a single number. It seems I can do that for async environment, but I haven't found a parameter or function for the mapgen environment. :-(
So any global variables that change mapgen behavior but also communicate it to other mods are not possible unless you copy them by hand (UGLY).

So for the next version I wish that you could pass data to mapgen environments, similar to how you can do that already with async environments.

[Desour]

I've also used the async mapgen API some time ago, for the hybriddog nether mod.
I can confirm it's not hard to use.
And by using dofile as fallback, you can also easily support older minetest versions.

>If lzr_mapgen exists (which is created by the mod), I can deduce the script must run in the global environment

I found it cleaner to register a separate init file as mapgen script, so I can set globals there, including settings from a common file:
https://codeberg.org/HybridDog/nether-p ... emerge.lua

[wsor4035]

wuzzy: based on the end of your post, you want https://github.com/minetest/minetest/pull/14659 (its also in the 5.10 milestone)

[Wuzzy]

Nice to hear there is already work on this.

[TenPlus1]

@Wuzzy - It's already being used in Farming Redo, and my forks of Caverealms_list and Other_worlds.

[Skamiz Kazzarch]

Dumb idea:
From poking through the API it seems the mapgen env has access to the settings object.
And settings can be set at runtime from code.
so ....
Could that be a way to smuggle in at least a little info into the mapgen env?

Depends of course on how exactly the Settings object works engine side, of which I have absolutely no clue.
Might look into that.

Edit: It works!

This is the file I register with 'register_mapgen_script'

Code: Select all

local c_stone = minetest.get_content_id("c_stone:granite")

-- automatically detect necessary chunk_size
-- though in some circumstances you will want to increase it afterwards
local blocks_per_chunk = tonumber(minetest.settings:get("chunksize")) or 5
local side_lenght = blocks_per_chunk * 16
local chunk_size = {x = side_lenght, y = side_lenght, z = side_lenght}


-- persistent data table for node data
local data = {}

minetest.register_on_generated(function(vm, minp, maxp, chunkseed)
	math.randomseed(chunkseed)

	local emin, emax = vm:get_emerged_area()
	local area = VoxelArea(emin, emax)
	vm:get_data(data)

	-- this is where the magic happens
	local height = tonumber(minetest.settings:get("mapgen_height")) or 0
	print("height: " .. height)

	-- noise index, same as i3d
	local ni = 0
	for z = minp.z, maxp.z do
		for y = minp.y, maxp.y do
			for x = minp.x, maxp.x do
				-- voxel area index, takes into acount overgenerated mapblocks
				local vi = area:index(x, y, z)

				if y <= height then
					data[vi] = c_stone
				end

			end
		end
	end
	-- finishing up
	vm:set_data(data)
end)

And then, in-game I can use:/set mapgen_height 10 and it changes the (newly generated) terrain height with no restart necessary. Thus showing that it's possible to pass info from the main thread to the mapgen thread.

[Wuzzy]

That sounds like a bug or oversight.

[LMD]

14659 has been merged: There will be an API to allow inter-"process" communication (IPC) in 5.10.0.

[Wuzzy]

Cool! But the name is highly misleading. It should be changed. https://github.com/minetest/minetest/issues/15391

[chameli]

I've been working with the threaded mapgen too, and I think it's easy to use once you get the hang of it. Using minetest.register_mapgen_script really makes things smoother. I ran into a similar problem with global variables, but I fixed it by copying them into the mapgen script.

[Wuzzy]

I believe variables can now (5.10.0) be passed between threads/environments with the new "IPC" feature. But I haven't tested it so far.

But if it works, this is a nice update.

[LMD]

They can and it works (at least it worked in my usage and testing so far). I used this to pass generated initialization data when I made the Spiraling Down mapgen use the threaded env because I'm lazy: https://github.com/spiraling-down/game/ ... d2583e17R5.