Server crashed for problem about DB

Post Reply
User avatar
MatyasP
Member
Posts: 140
Joined: Wed Jan 19, 2022 22:12
GitHub: Matyas-Pilz
In-game: MatyasP
Location: Prague, Czech republic
Contact:

Server crashed for problem about DB

by MatyasP » Post

Hello, my server crashed with this in log and can't be started. Where should be the problem? How can I repair it?

2026-03-17 17:18:17: ERROR[Main]: ServerError: Failed to initialize the map database. The world may be corrupted or in an unsupported format.
2026-03-17 17:18:17: ERROR[Main]: Failed to set SQLite3 synchronous mode: database is locked

User avatar
Blockhead
Moderator
Posts: 2984
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Server crashed for problem about DB

by Blockhead » Post

MatyasP wrote:
Tue Mar 17, 2026 17:04
2026-03-17 17:18:17: ERROR[Main]: Failed to set SQLite3 synchronous mode: database is locked
It looks like you have the world open already in singleplayer or another server instance. I get a similar one when using world downloading and running a server at the same time on the same computer. If there's nothing obviously running, try searching for crashed/zombie processes that have it open.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
MatyasP
Member
Posts: 140
Joined: Wed Jan 19, 2022 22:12
GitHub: Matyas-Pilz
In-game: MatyasP
Location: Prague, Czech republic
Contact:

Re: Server crashed for problem about DB

by MatyasP » Post

Blockhead wrote:
Wed Mar 18, 2026 01:51
MatyasP wrote:
Tue Mar 17, 2026 17:04
2026-03-17 17:18:17: ERROR[Main]: Failed to set SQLite3 synchronous mode: database is locked
It looks like you have the world open already in singleplayer or another server instance. I get a similar one when using world downloading and running a server at the same time on the same computer. If there's nothing obviously running, try searching for crashed/zombie processes that have it open.
Thank you! It has more causes (synchronous SQLite, large -wal etc.)

Problem solved.
Part of this message (bellow this) created by AI (Google Gemini that helps me with it step by step) and checked and repaired by human.

Post-Mortem: Luanti Server Crash & SQLite Database Lock

Root Cause
The server experienced a fatal crash triggered by a Lua runtime error in a mod (railbuilder), which attempted to index a nil value during a global step. This occurred during high I/O activity (multiple users building tracks).

The situation was exacerbated by a misconfiguration in minetest.conf: sqlite_synchronous = 1 (NORMAL). On a server with a large 10GB database and heavy mods like AdvTrains, this caused an I/O bottleneck. When the crash happened, the SQLite checkpoint failed, leaving a massive 8.5GB -wal file and locking the database.

Symptoms
  1. Database is locked: The server failed to restart because the previous process didn't release the file lock, and the massive WAL file prevented a quick recovery.
  2. Map Corruption/Ghost Blocks: Attempting to force-start by deleting journal files led to temporary synchronization issues between the old map.sqlite and the uncommitted data.
  3. Extended "Starting" state: Pterodactyl panel stayed yellow for minutes as SQLite struggled to replay the oversized WAL log.
Resolution
  1. Manual Checkpoint: Downloaded the 10GB map.sqlite and 8.5GB map.sqlite-wal to a local high-performance machine (i7-13700).
  2. Database Integration: Executed PRAGMA wal_checkpoint(TRUNCATE); via SQLite3 CLI to force-merge the logs into the main DB.
  3. Optimization: Performed VACUUM; to defragment the 10M+ rows and fix indexing issues.
  4. Code Fix: Patched railbuilder_ui.lua with a nil-check for player positions to prevent the specific crash from recurring.
  5. Config Tuning: Set sqlite_synchronous = 0` and adjusted map_save_interval to 20s to prevent future I/O deadlocks on SATA SSD storage.
Status: Server fully recovered with zero data loss.
Attachments
Luanti20260317problemCPU.png
Luanti20260317problemCPU.png (18.77 KiB) Viewed 1320 times
Luanti20260317problemRAM.png
Luanti20260317problemRAM.png (32.68 KiB) Viewed 1320 times
Luanti20260317problemVykon.png
Luanti20260317problemVykon.png (67.12 KiB) Viewed 1320 times

User avatar
Blockhead
Moderator
Posts: 2984
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Server crashed for problem about DB

by Blockhead » Post

MatyasP wrote:
Tue Mar 17, 2026 17:04
The server failed to restart because the previous process didn't release the file lock,
Did you ever figure out what the previous process was? A Lua mod crash is not usually also a crash of the whole server process, but in dedicated server mode it will usually end the process safely. Was the server crashing with the railbuilder-related error before you started this thread?
MatyasP wrote:
Tue Mar 17, 2026 17:04
[*] Code Fix: Patched railbuilder_ui.lua with a nil-check for player positions to prevent the specific crash from recurring.
Please post the patch so it can be reviewed and, if useful, shared.
MatyasP wrote:
Tue Mar 17, 2026 17:04

The situation was exacerbated by a misconfiguration in minetest.conf: sqlite_synchronous = 1 (NORMAL). On a server with a large 10GB database and heavy mods like AdvTrains, this caused an I/O bottleneck. When the crash happened, the SQLite checkpoint failed, leaving a massive 8.5GB -wal file and locking the database.

[*] Config Tuning: Set sqlite_synchronous = 0` and adjusted map_save_interval to 20s to prevent future I/O deadlocks on SATA SSD storage.[/list]

Status: Server fully recovered with zero data loss.
ref. So now the database is more likely to corrupt in future. It's not a misconfiguration, it's actually a safer configuration. The default is FULL(2), a higher level.

On a multiplayer server with that size, it's time to think about moving to PostgreSQL since it will perform better.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
MatyasP
Member
Posts: 140
Joined: Wed Jan 19, 2022 22:12
GitHub: Matyas-Pilz
In-game: MatyasP
Location: Prague, Czech republic
Contact:

Re: Server crashed for problem about DB

by MatyasP » Post

Blockhead wrote:
Wed Mar 18, 2026 17:24
Thank you for help!
Blockhead wrote:
Wed Mar 18, 2026 17:24
MatyasP wrote:
Tue Mar 17, 2026 17:04
The server failed to restart because the previous process didn't release the file lock,
Did you ever figure out what the previous process was? A Lua mod crash is not usually also a crash of the whole server process, but in dedicated server mode it will usually end the process safely. Was the server crashing with the railbuilder-related error before you started this thread?
I have seen it. End of server log before crash is:

Code: Select all

2026-03-17 15:10:35: VERBOSE[Server]: Server: MapEditEvents modified total 7 blocks:
2026-03-17 15:10:35: VERBOSE[Server]:   MEET_OTHER ______________________________________    1x        7
2026-03-17 15:10:35: INFO[Server]: Server: Player Player2, peer_id=33012: full map send (d=13) completed after 0.724571s, restarting
2026-03-17 15:10:35: INFO[Server]: Server: Player Player3, peer_id=13790: full map send (d=13) completed after 0.724571s, restarting
2026-03-17 15:10:35: INFO[Server]: Server: Player Player4, peer_id=17634: full map send (d=13) completed after 0.724571s, restarting
2026-03-17 15:10:35: VERBOSE[Server]: Server: MapEditEvents modified total 2 blocks:
2026-03-17 15:10:35: VERBOSE[Server]:   MEET_OTHER ______________________________________    1x        2
2026-03-17 15:10:35: VERBOSE[Server]: TOSERVER_INTERACT: action=1, item=1, pointed=[node under=5368,57,1645 above=5368,57,1646]
2026-03-17 15:10:35: INFO[Server]: Server: Player Player1, peer_id=6520: full map send (d=13) completed after 0.452695s, restarting
2026-03-17 15:10:35: VERBOSE[Server]: Server: MapEditEvents modified total 1 blocks:
2026-03-17 15:10:35: VERBOSE[Server]:   MEET_OTHER ______________________________________    1x        1
2026-03-17 15:10:35: INFO[Server]: ServerMap: Written: 1 blocks, 8704 blocks in memory.
2026-03-17 15:10:35: INFO[Server]: ServerMap: Blocks modified by: 
2026-03-17 15:10:35: INFO[Server]:   setNode _________________________________________    1x        1
2026-03-17 15:10:35: VERBOSE[Server]: Server::deletingPeer(): id=17634, timeout=0
2026-03-17 15:10:35: INFO[Server]: Players:
2026-03-17 15:10:35: INFO[Server]: * Player2	RemoteClient 33012: blocks_sent=1552, blocks_sending=11, nearest_unsent_d=4, map_send_completion_timer=0, excess_gotblocks=0
2026-03-17 15:10:35: INFO[Server]: * Player3	RemoteClient 13790: blocks_sent=2054, blocks_sending=0, nearest_unsent_d=12, map_send_completion_timer=0, excess_gotblocks=0
2026-03-17 15:10:35: INFO[Server]: * Player1	RemoteClient 6520: blocks_sent=4367, blocks_sending=0, nearest_unsent_d=3, map_send_completion_timer=0, excess_gotblocks=16
2026-03-17 15:10:35: INFO[Server]: * MatyasP	RemoteClient 6225: blocks_sent=1674, blocks_sending=0, nearest_unsent_d=3, map_send_completion_timer=0, excess_gotblocks=147
2026-03-17 15:10:35: ACTION[Server]: [doc] Wrote player data into /home/container/.minetest/worlds/MtV6_01/doc.mt.
2026-03-17 15:10:35: VERBOSE[Server]: ObjectRef::l_remove(): id=12629
2026-03-17 15:10:35: VERBOSE[Server]: ObjectRef::l_remove(): id=12728
2026-03-17 15:10:35: VERBOSE[Server]: ObjectRef::l_remove(): id=12854
2026-03-17 15:10:35: ACTION[Server]: Player4 leaves game. List of players: Player2 Player3 Player1 MatyasP 
2026-03-17 15:10:35: INFO[Server]: Server: Player Player3, peer_id=13790: full map send (d=13) completed after 0.453152s, restarting
2026-03-17 15:10:35: ACTION[Server]: Error traceback saved to /home/container/.minetest/worlds/MtV6_01/error_mail_contents.txt
2026-03-17 15:10:35: INFO[Server]: setAsyncFatalError: Lua: Runtime error from mod 'railbuilder' in callback environment_Step(): /usr/share/luanti/builtin/common/vector.lua:133: attempt to index local 'b' (a nil value)
2026-03-17 15:10:35: INFO[Server]: stack traceback:
2026-03-17 15:10:35: INFO[Server]: 	[C]: in function '__index'
2026-03-17 15:10:35: INFO[Server]: 	/usr/share/luanti/builtin/common/vector.lua:133: in function 'distance'
2026-03-17 15:10:35: INFO[Server]: 	...t/games/MtV7/mods/1_tools/railbuilder/railbuilder_ui.lua:164: in function 'update_hud'
2026-03-17 15:10:35: INFO[Server]: 	...r/.minetest/games/MtV7/mods/1_tools/railbuilder/init.lua:147: in function <...r/.minetest/games/MtV7/mods/1_tools/railbuilder/init.lua:138>
2026-03-17 15:10:35: INFO[Server]: 	/usr/share/luanti/builtin/common/register.lua:27: in function </usr/share/luanti/builtin/common/register.lua:13>
2026-03-17 15:10:35: INFO[Main]: Players:
2026-03-17 15:10:35: INFO[Main]: * Player3	RemoteClient 13790: blocks_sent=2054, blocks_sending=0, nearest_unsent_d=0, map_send_completion_timer=0, excess_gotblocks=0
2026-03-17 15:10:35: INFO[Main]: * Player1	RemoteClient 6520: blocks_sent=4367, blocks_sending=0, nearest_unsent_d=6, map_send_completion_timer=0, excess_gotblocks=0
2026-03-17 15:10:35: INFO[Main]: * MatyasP	RemoteClient 6225: blocks_sent=1674, blocks_sending=4, nearest_unsent_d=5, map_send_completion_timer=0, excess_gotblocks=0
2026-03-17 15:10:35: INFO[Main]: Players:
2026-03-17 15:10:35: INFO[Main]: * Player1	RemoteClient 6520: blocks_sent=4367, blocks_sending=0, nearest_unsent_d=6, map_send_completion_timer=0, excess_gotblocks=0
2026-03-17 15:10:35: INFO[Main]: * MatyasP	RemoteClient 6225: blocks_sent=1674, blocks_sending=4, nearest_unsent_d=5, map_send_completion_timer=0, excess_gotblocks=0
2026-03-17 15:10:35: INFO[Main]: Players:
2026-03-17 15:10:35: INFO[Main]: * MatyasP	RemoteClient 6225: blocks_sent=1674, blocks_sending=4, nearest_unsent_d=5, map_send_completion_timer=0, excess_gotblocks=0
2026-03-17 15:10:35: ACTION[Main]: Server: Shutting down
2026-03-17 15:10:35: INFO[Main]: Server: Stopping and waiting for threads
2026-03-17 15:10:35: INFO[Main]: Server: Threads stopped
2026-03-17 15:10:35: INFO[Main]: Server: Executing shutdown hooks
2026-03-17 15:10:35: ACTION[Main]: [doc] Server shuts down. Player data is about to be saved.
2026-03-17 15:10:35: ACTION[Main]: [doc] Wrote player data into /home/container/.minetest/worlds/MtV6_01/doc.mt.
2026-03-17 15:10:35: ACTION[Main]: [advtrains]Saved advtrains save files, took 175 ms 
2026-03-17 15:10:35: INFO[Main]: Server: Saving players
2026-03-17 15:10:35: INFO[Main]: Server: Kicking players
2026-03-17 15:10:36: VERBOSE[Main]: ServerEnvironment::deactivateFarObjects(): deactivating object id=1206 on inactive block (-44,1,-92)
2026-03-17 15:10:36: VERBOSE[Main]: Server::ActiveObjectMgr::removeObject(): id=1206
2026-03-17 15:10:36: VERBOSE[Main]: ServerEnvironment::deactivateFarObjects(): deactivating object id=1214 on inactive block (-1,0,-34)
2026-03-17 15:10:36: VERBOSE[Main]: Server::ActiveObjectMgr::removeObject(): id=1214
... (many rows about the same) ...
2026-03-17 15:10:36: VERBOSE[Main]: ServerEnvironment::deactivateFarObjects(): deactivating object id=13083 on inactive block (58,1,-57)
2026-03-17 15:10:36: VERBOSE[Main]: Server::ActiveObjectMgr::removeObject(): id=13083
2026-03-17 15:10:36: VERBOSE[Main]: ServerEnvironment::deactivateFarObjects(): deactivating object id=13087 on inactive block (55,0,-58)
2026-03-17 15:10:36: VERBOSE[Main]: Server::ActiveObjectMgr::removeObject(): id=13087
2026-03-17 15:10:36: INFO[Main]: Server: Saving environment metadata
2026-03-17 15:10:36: INFO[Main]: AsyncEngine: Waiting for 1 threads
2026-03-17 15:10:36: VERBOSE[Main]: virtual ServerMap::~ServerMap()
2026-03-17 15:10:36: INFO[Main]: ServerMap: Written: 4969 blocks, 8705 blocks in memory.
2026-03-17 15:10:36: INFO[Main]: ServerMap: Blocks modified by: 
2026-03-17 15:10:36: INFO[Main]:   Timestamp expired (step) ________________________    1x     1022
2026-03-17 15:10:36: INFO[Main]:   deactivateFarObjects: static data changed _______    1x       11
2026-03-17 15:10:36: INFO[Main]:   setTimestamp ____________________________________    1x      353
2026-03-17 15:10:36: INFO[Main]:   setTimestamp, Timestamp expired (step) __________    1x     3583
2026-03-17 15:10:36: INFO[Main]: ServerMap: Saved map to /home/container/.minetest/worlds/MtV6_01
2026-03-17 15:10:36: INFO[Main]: BanManager: saving to /home/container/.minetest/worlds/MtV6_01/ipban.txt
2026-03-17 15:10:37: ERROR[Main]: ServerError: AsyncErr: Lua: Runtime error from mod 'railbuilder' in callback environment_Step(): /usr/share/luanti/builtin/common/vector.lua:133: attempt to index local 'b' (a nil value)
2026-03-17 15:10:37: ERROR[Main]: stack traceback:
2026-03-17 15:10:37: ERROR[Main]: 	[C]: in function '__index'
2026-03-17 15:10:37: ERROR[Main]: 	/usr/share/luanti/builtin/common/vector.lua:133: in function 'distance'
2026-03-17 15:10:37: ERROR[Main]: 	...t/games/MtV7/mods/1_tools/railbuilder/railbuilder_ui.lua:164: in function 'update_hud'
2026-03-17 15:10:37: ERROR[Main]: 	...r/.minetest/games/MtV7/mods/1_tools/railbuilder/init.lua:147: in function <...r/.minetest/games/MtV7/mods/1_tools/railbuilder/init.lua:138>
2026-03-17 15:10:37: ERROR[Main]: 	/usr/share/luanti/builtin/common/register.lua:27: in function </usr/share/luanti/builtin/common/register.lua:13>
Crashes with using Railbuilder are for me "normal usual crashes, knowed problem". Similar as crashes with some logic in AdvTrains, similar as crashes in stairsplus... I accept that we have some crashes. I know about only one crash (not on start of server) I have repaired (crash after teleport to long distance when is activated terraform light) - and for it, I was needed help by ChatGPT.
https://github.com/Matyas-Pilz/terrafor ... r/init.lua

I know that every crash can make corruption or other problem of world. However, I don't have time and energy to repair or report everytime when is some crash. And my level of programming is too low. Thanks the this good community, thanks the simple code writing and thanks AI, I can create some mods or edit mods by other authors.
Blockhead wrote:
Wed Mar 18, 2026 17:24
MatyasP wrote:
Tue Mar 17, 2026 17:04
[*] Code Fix: Patched railbuilder_ui.lua with a nil-check for player positions to prevent the specific crash from recurring.
Please post the patch so it can be reviewed and, if useful, shared.
Yes, there is:
file: railbuilder_ui.lua
function: update_hud (from row 153)
is now:

Code: Select all

function update_hud(player, force)
    local player_data = railbuilder.datastore.get_data(player)
    local start_pos = player_data.railbuilder_start_pos

    if player:get_player_control().dig then
        force = update_slope_selection_ui(player) or force
    else
        hide_slope_selection_ui(player)
    end

    -- REPAIR: Check if exist last saved position for don't fail the vector.distance
    local last_pos = player_data.ui.hud_update_last_player_pos
    local current_pos = player:get_pos()

    -- If last_pos don't exist (first start), we enforce the update and set the position
    if not last_pos then
        force = true
        player_data.ui.hud_update_last_player_pos = current_pos
        last_pos = current_pos
    end

    -- nothing to update if player did not move
    if force or vector.distance(last_pos, current_pos) > 1 then
        player_data.ui.hud_update_last_player_pos = current_pos -- Aktualizace pozice pro příště
        remove_hud_track_preview_points(player)
        
        if is_start_marker_valid(player) then
            local directions = advtrain_helpers.get_advtrains_dirs(start_pos, player_data.railbuilder_last_direction, player_data.railbuilder_last_vertical_direction) or
                    advtrain_helpers.get_closest_directions(player, start_pos)
            update_hud_track_preview_points(player, start_pos, directions)
            player_data.ui.hud_track_preview_directions = directions
        end
    end
    if is_start_marker_valid(player) then
        trace_player_view(player, player_data.ui.hud_track_preview_directions)
    end
end
Blockhead wrote:
Wed Mar 18, 2026 17:24
MatyasP wrote:
Tue Mar 17, 2026 17:04

The situation was exacerbated by a misconfiguration in minetest.conf: sqlite_synchronous = 1 (NORMAL). On a server with a large 10GB database and heavy mods like AdvTrains, this caused an I/O bottleneck. When the crash happened, the SQLite checkpoint failed, leaving a massive 8.5GB -wal file and locking the database.

[*] Config Tuning: Set sqlite_synchronous = 0` and adjusted map_save_interval to 20s to prevent future I/O deadlocks on SATA SSD storage.[/list]

Status: Server fully recovered with zero data loss.
ref. So now the database is more likely to corrupt in future. It's not a misconfiguration, it's actually a safer configuration. The default is FULL(2), a higher level.

On a multiplayer server with that size, it's time to think about moving to PostgreSQL since it will perform better.
I think that this is the main. You are 4th person that says me that it can solve changing the backend.
You and Warr said that there is better using PostgreSQL (Google Gemini said the same), proller preffers the kvrocks and support from provider said (for my ask about PostgreSQL) that he already wanted to offer me different backend for problems after high load the SQLite.
(my server is hosted on small Czech game hosting provider)

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests