Server Perfomance
-
- Member
- Posts: 11
- Joined: Thu Jul 12, 2018 00:16
- GitHub: frbitten
- IRC: frbitten
- In-game: frbitten
Server Perfomance
Hi
Is there any test or performance study of the minetest server?
I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.
Anyone know any posts on this? I did research and I did not find it.
Thanks
Is there any test or performance study of the minetest server?
I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.
Anyone know any posts on this? I did research and I did not find it.
Thanks
Re: Server Perfomance
it varies wildly on several factors, and therefore it's really difficult to give specific answers. For instance, all of the following are huge factors in how well a minetest server performs:frbitten wrote:Hi
Is there any test or performance study of the minetest server?
I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.
Anyone know any posts on this? I did research and I did not find it.
Thanks
- used storage backend for the map
- number of players
- lag that players may have
- CPU model (core speed, essentially, but cache size matters)
- RAM size, but also speed
- Storage speed
- network congestion factors
- mods used on the server
etc..
You can have a great server on 30$ of hardware, or you could have a terrible server on a beefcake machine, it all depends.
-
- Member
- Posts: 11
- Joined: Thu Jul 12, 2018 00:16
- GitHub: frbitten
- IRC: frbitten
- In-game: frbitten
Re: Server Perfomance
sofar wrote:it varies wildly on several factors, and therefore it's really difficult to give specific answers. For instance, all of the following are huge factors in how well a minetest server performs:frbitten wrote:Hi
Is there any test or performance study of the minetest server?
I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.
Anyone know any posts on this? I did research and I did not find it.
Thanks
- used storage backend for the map
- number of players
- lag that players may have
- CPU model (core speed, essentially, but cache size matters)
- RAM size, but also speed
- Storage speed
- network congestion factors
- mods used on the server
etc..
You can have a great server on 30$ of hardware, or you could have a terrible server on a beefcake machine, it all depends.
I understand these differences and factors that affect the player's experience.
I figured the developers would have done some perfomance testing in a controllable environment. I know that even though it can not be applied exactly in the end use it is a good analytical metric.
But giving a more specific example then. On a server with only "minetest_game" would it work with 1000 simultaneous online players? What would be the hardare needed for this task?
Thanks
Re: Server Perfomance
I don't think anyone attempted more than ~ 50 players at this point. Unless you can find 1000 actual human beings, this would be impossible to simulate.frbitten wrote:But giving a more specific example then. On a server with only "minetest_game" would it work with 1000 simultaneous online players? What would be the hardare needed for this task?
Thanks
- rubenwardy
- Moderator
- Posts: 7026
- Joined: Tue Jun 12, 2012 18:11
- GitHub: rubenwardy
- IRC: rubenwardy
- In-game: rubenwardy
- Location: London, United Kingdom
- Contact:
Re: Server Perfomance
I've seen servers with up to 80 player on them, but they weren't snappy
Minetest isn't intended or designed for thousands of users, that's the scale when you need to start doing clever multithreading
Minetest isn't intended or designed for thousands of users, that's the scale when you need to start doing clever multithreading
You can use bots to stress test the connection, freeminer did thisso far wrote:Unless you can find 1000 actual human beings, this would be impossible to simulate
Re: Server Perfomance
yes, and it's certainly a valid test case (and great data), but it won't actually say much about real world usage, like, every player is building in their own little plot, or, 250 players are attempting to fight one large monster. Both these scenarios are likely for massive servers, but you'd have to make some really smart AI to simulate it. (by all means, I'd love to see an approximation though).rubenwardy wrote:You can use bots to stress test the connection, freeminer did this
-
- Member
- Posts: 11
- Joined: Thu Jul 12, 2018 00:16
- GitHub: frbitten
- IRC: frbitten
- In-game: frbitten
Re: Server Perfomance
Thank you all.
What he wanted to know was the scale that the development of the game was thought of. If we are talking about the house of tens. Surely it would not reach hundreds or thousands.
If someone decides to do the automated test post the results here. Even not being precise for a real environment. It is a valid test for a first moment. Somanete 500 customers walked in a random direction would already be enough to see the level of refactoring that would be required to support real players on that scale.
Thank you
What he wanted to know was the scale that the development of the game was thought of. If we are talking about the house of tens. Surely it would not reach hundreds or thousands.
If someone decides to do the automated test post the results here. Even not being precise for a real environment. It is a valid test for a first moment. Somanete 500 customers walked in a random direction would already be enough to see the level of refactoring that would be required to support real players on that scale.
Thank you
Re: Server Perfomance
not many people run on high end Xeon CPU's with high core frequencies and 10gig ethernet backbone connections. It's entirely possible that you can have 500+ people logged on to a server, and moving about and even doing things. Most people just shell out the money for low end hosting.frbitten wrote: If someone decides to do the automated test post the results here. Even not being precise for a real environment. It is a valid test for a first moment. Somanete 500 customers walked in a random direction would already be enough to see the level of refactoring that would be required to support real players on that scale.
-
- Member
- Posts: 11
- Joined: Thu Jul 12, 2018 00:16
- GitHub: frbitten
- IRC: frbitten
- In-game: frbitten
Re: Server Perfomance
For what you wrote wrote that the software supports this number of players, it would only be a matter of hardware needed to accomplish the task. How does developer know how to give hardare estimates for the server to support more than 500 players simultaneously interacting simultaneously in the same world?sofar wrote: not many people run on high end Xeon CPU's with high core frequencies and 10gig ethernet backbone connections. It's entirely possible that you can have 500+ people logged on to a server, and moving about and even doing things. Most people just shell out the money for low end hosting.
I wanted to set up a game server but without an even shallow estimate I can not analyze the costs.
Thanks
Re: Server Perfomance
If I give you an estimate, and you go and pay e.g. 1000$ to procure the hosting based on that estimate, and the estimate was completely wrong, wouldn't you be very upset?frbitten wrote: For what you wrote wrote that the software supports this number of players, it would only be a matter of hardware needed to accomplish the task. How does developer know how to give hardare estimates for the server to support more than 500 players simultaneously interacting simultaneously in the same world?
I wanted to set up a game server but without an even shallow estimate I can not analyze the costs.
Look, most of us run a server for 10-30 or so people max, and we've had good success with hosting services that are in the 5$-20$ price per month range(I'm not even sure what most others are paying). Note that's not even an estimate, it's just a statement that it works for us, and might not work for you. If you want better estimates for numbers so large that we've never seen them, there is no way that you can or should rely on the few data points right now that exist, because they are entirely invalid for a scale that is off by a factor 10+.
So, if you want to know if 1000 players on a single server is possible, the only way is to try and properly document it. I'm game, of course, I'd love to see something like this.
- lag01
- Member
- Posts: 321
- Joined: Sun Mar 16, 2014 03:41
- GitHub: AndrejIT
- IRC: lag01
- In-game: lag
- Contact:
Re: Server Perfomance
By my observations, 0.4.17 minetestserver consumes 20-80KiB/s per player. (But possible that number increases when more players are online)
So 1000 players would require 20 MiB/s (170 Mbit/s) of outgoing connection as bare minimum.
So 1000 players would require 20 MiB/s (170 Mbit/s) of outgoing connection as bare minimum.
- rnd
- Member
- Posts: 220
- Joined: Sun Dec 28, 2014 12:24
- GitHub: ac-minetest
- IRC: ac_minetest
- In-game: rnd
Re: Server Perfomance
"So 1000 players would require 20 MiB/s (170 Mbit/s) of outgoing connection as bare minimum."
The problem is not linear in number of players:
1. sending player data: If you have n players walking around spawn area, server needs to send player position/view angles/what did player do?... of the rest of n-1 players to each player, thats n*(n-1) = O(n^2) packets to send. This part is nonlinear, and scales very badly (2x more players, 4x more data, 5x more -> 25x more data, 10x more -> 100x more data )
2. sending map data: for each player you have to send some constant of data (on average). This part is linear.
So even 100 players walking all near large spawn area might be way over head of current networking implementation. I remember there was 10+s lag with 70-80 players on just test. On minecraft they must have some really optimized programming to be able to get : https://hypixel.net/ - "Join 19,680 other online Players"
The problem is not linear in number of players:
1. sending player data: If you have n players walking around spawn area, server needs to send player position/view angles/what did player do?... of the rest of n-1 players to each player, thats n*(n-1) = O(n^2) packets to send. This part is nonlinear, and scales very badly (2x more players, 4x more data, 5x more -> 25x more data, 10x more -> 100x more data )
2. sending map data: for each player you have to send some constant of data (on average). This part is linear.
So even 100 players walking all near large spawn area might be way over head of current networking implementation. I remember there was 10+s lag with 70-80 players on just test. On minecraft they must have some really optimized programming to be able to get : https://hypixel.net/ - "Join 19,680 other online Players"
1EvCmxbzl5KDu6XAunE1K853Lq6VVOsT
-
- Member
- Posts: 11
- Joined: Thu Jul 12, 2018 00:16
- GitHub: frbitten
- IRC: frbitten
- In-game: frbitten
Re: Server Perfomance
1000 simultaneous players is a very complex task to be performed. Games that achieve this perfoamance are very rare.
The only thing that I participated in something like this is EVE Online where I participated in a fight with 2500 players at the same time. But the lag was absurd.
I am a developer and I understand the difficulty associated with doing something on this scale. My post was more out of curiosity what would be the known limits of the Minetest server.
For the comments is between 50 and 80 players simultaneously. Remembering that online players are not the same as simultaneous players. For they may be in places far from the servant that do not quarrel with one another.
I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.
Thanks for the feedback.
The only thing that I participated in something like this is EVE Online where I participated in a fight with 2500 players at the same time. But the lag was absurd.
I am a developer and I understand the difficulty associated with doing something on this scale. My post was more out of curiosity what would be the known limits of the Minetest server.
For the comments is between 50 and 80 players simultaneously. Remembering that online players are not the same as simultaneous players. For they may be in places far from the servant that do not quarrel with one another.
I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.
Thanks for the feedback.
- Linuxdirk
- Member
- Posts: 3250
- Joined: Wed Sep 17, 2014 11:21
- In-game: Linuxdirk
- Location: Germany
- Contact:
Re: Server Perfomance
Forget that. It is impossible for Minetest to handle 500 players. Not even 50 without horrible lag.frbitten wrote:I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.
Re: Server Perfomance
The problem with minetest is lack of multithreading. This way, it does not scale well to the strengths of modern CPUs.
Play on Linuxworks server. We have got lots of trains.
My mods (Industrial Livestock Farming, Farebox and Faregate, Moving Walkways, Atm)
My mods (Industrial Livestock Farming, Farebox and Faregate, Moving Walkways, Atm)
- bhree
- Member
- Posts: 169
- Joined: Tue Jun 19, 2018 07:45
- GitHub: bhree
- IRC: bhree
- In-game: bhree
- Location: Alone In The Wild Land
- Contact:
Re: Server Perfomance
And if minetest support something like mpirun then scaling across multiple servers allow players to almost unlimited number. But that may need radical design and another development team.
-
- Member
- Posts: 3476
- Joined: Sun Jul 12, 2015 22:51
- GitHub: BBmine
- IRC: BBmine
- In-game: Baggins
- Location: USA
Re: Server Perfomance
Server hardware isn't everything. If you have 100 players at spawn, each individual player will have player information transmitted to them about the other 99 players. That's a lot of bandwidth for the clients!
Re: Server Perfomance
Making minetest scale across multiple cores or even across nodes isn't as simple as bolting on something like MPI. That would likely accomplish the opposite, and make performce drastically degrade.bhree wrote:And if minetest support something like mpirun then scaling across multiple servers allow players to almost unlimited number. But that may need radical design and another development team.
True multi node scaling of an event driven environment like minetest where nearby players interact would have to be designed from scratch. Even then there are limits as to how many players close together you could handle. And then, if you truly manage to scale up, your lag would also increase due to the extra care that is needed.
I'm not convinced by the discussion that any of this type of work is needed. I'd rather see some actual experiments with data first to see how good/bad it actually is. And given that most responses on this thread are of the "I used a lump of clay to run a server and it sucked" type, I'm entirely pessimistic that people understand that performance work requires solid data to begin with, and ends with solid data.
Note: The game code really matters a lot. If you program your game code well (the Lua part) you can remove most of the heavy bottlenecks if you're careful. But if you grab a handful of random mods that look cool and don't look at the internals, you're going to have a bad time. Running a high quality server requires significant coding skills.
- bhree
- Member
- Posts: 169
- Joined: Tue Jun 19, 2018 07:45
- GitHub: bhree
- IRC: bhree
- In-game: bhree
- Location: Alone In The Wild Land
- Contact:
Re: Server Perfomance
For single threaded system, the app itself is already awesome, simplest way just push the best hardwares and connection. Push the clockspeed, aquire RAM as much as possible, best internet connection. SSD over HDD. And because we are working with large database then enterprise grade DBMS should be applied. Each process is allowed to access its own core, thanks to the OS. Something like remote media server using http on another core will boost the performance. CPU of 4 cores is enough of the fastest clockspeed and if someone can afford half TB of RAM. Nothing beats that.
MPI is not the only multiprocess solution but simpler and cheaper to deploy, albeit the complexity in software design.
MPI is not the only multiprocess solution but simpler and cheaper to deploy, albeit the complexity in software design.
That will be the problem of every client, usually solved with better hardware and connection. I'm not in a capacity of questioning the software.Server hardware isn't everything. If you have 100 players at spawn, each individual player will have player information transmitted to them about the other 99 players. That's a lot of bandwidth for the clients!
- sorcerykid
- Member
- Posts: 1875
- Joined: Fri Aug 26, 2016 15:36
- GitHub: sorcerykid
- In-game: Nemo
- Location: Illinois, USA
Re: Server Perfomance
What? JT2 frequently had 50 or more players with a max_log of just 0.12 seconds. Fixer has even commented on several occasions about the unusual performance compared to other survival servers.Linuxdirk wrote:Forget that. It is impossible for Minetest to handle 500 players. Not even 50 without horrible lag.frbitten wrote:I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.
I'm not going to give away all the optimization tricks I used. But off the top of my head, I'll just say that I spent a great deal of time rewriting various portions of the engine to address scalability and efficiency concerns (two of the major bottlenecks being the ABM Handler and authentication handler, but some of the other builtin Lua routines and mods like "tnt" were vastly reworked as well). I also implemented smart-caching mechanisms for node digging and placement to streamline protection and privilege checks. And the world directory for the production server is configured to use tmpfs.
All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.
Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)
Re: Server Perfomance
At the risk of going offtopic, but we're still talking about an open source project here:
The player community on JT2 isn't large enough to sustain all of the Minetest development. That needs many more good quality servers. You'd be helping yourself by helping other server operators run a higher quality shop.
That's unfortunate, I wish you would. Everyone could benefit from that.sorcerykid wrote:I'm not going to give away all the optimization tricks I used.
Please consider contributing some of the knowledge and methods back to the community instead.sorcerykid wrote:All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.
Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)
The player community on JT2 isn't large enough to sustain all of the Minetest development. That needs many more good quality servers. You'd be helping yourself by helping other server operators run a higher quality shop.
-
- Member
- Posts: 11
- Joined: Thu Jul 12, 2018 00:16
- GitHub: frbitten
- IRC: frbitten
- In-game: frbitten
Re: Server Perfomance
can you give details of the hardware used for that number of players?sorcerykid wrote: What? JT2 frequently had 50 or more players with a max_log of just 0.12 seconds. Fixer has even commented on several occasions about the unusual performance compared to other survival servers.
I'm not going to give away all the optimization tricks I used. But off the top of my head, I'll just say that I spent a great deal of time rewriting various portions of the engine to address scalability and efficiency concerns (two of the major bottlenecks being the ABM Handler and authentication handler, but some of the other builtin Lua routines and mods like "tnt" were vastly reworked as well). I also implemented smart-caching mechanisms for node digging and placement to streamline protection and privilege checks. And the world directory for the production server is configured to use tmpfs.
All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.
Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)
-
- Member
- Posts: 11
- Joined: Thu Jul 12, 2018 00:16
- GitHub: frbitten
- IRC: frbitten
- In-game: frbitten
Re: Server Perfomance
Does the server code not use multiple colors in any case? Control of connections (socket), login, etc. Is it all running on the same thread?sofar wrote: Making minetest scale across multiple cores or even across nodes isn't as simple as bolting on something like MPI. That would likely accomplish the opposite, and make performce drastically degrade.
True multi node scaling of an event driven environment like minetest where nearby players interact would have to be designed from scratch. Even then there are limits as to how many players close together you could handle. And then, if you truly manage to scale up, your lag would also increase due to the extra care that is needed.
I'm not convinced by the discussion that any of this type of work is needed. I'd rather see some actual experiments with data first to see how good/bad it actually is. And given that most responses on this thread are of the "I used a lump of clay to run a server and it sucked" type, I'm entirely pessimistic that people understand that performance work requires solid data to begin with, and ends with solid data.
Note: The game code really matters a lot. If you program your game code well (the Lua part) you can remove most of the heavy bottlenecks if you're careful. But if you grab a handful of random mods that look cool and don't look at the internals, you're going to have a bad time. Running a high quality server requires significant coding skills.
Re: Server Perfomance
s/colors/threads/, I assumefrbitten wrote: Does the server code not use multiple colors in any case? Control of connections (socket), login, etc. Is it all running on the same thread?
The server engine does use threads but it's limited to a few easy to split up things. The only significant thing that is running in a separate thread is mapgen, essentially.
This isn't necessarily bad. For instance, putting socket handling on a different core isn't a way to make socket handling faster. In fact, if you move low level packet handling to a different CPU, you actually decrease the network performance, because now the data may need to be copied from core to core and there will be a significant amount of cache misses for every single packet. If you process packets (ipv4, UDP, socket level etc) entirely on the same core, you'd avoid that problem. This is why benchmark professionals often "pin" related software tasks all on a single core so that those cross-core overheads are eliminated.
A lot of menial tasks also don't make sense to split up. Your hot path is going to be world events. Even if you handle a login event per second, it's still peanuts compared to hundreds of server ticks with player actions and block updates. It just has to be not a burden on overall performance. You have 50ms - 100ms per server tick, and that is a lot of time to handle the occasional login event.
- rubenwardy
- Moderator
- Posts: 7026
- Joined: Tue Jun 12, 2012 18:11
- GitHub: rubenwardy
- IRC: rubenwardy
- In-game: rubenwardy
- Location: London, United Kingdom
- Contact:
Re: Server Perfomance
Minetest uses threads for sending and receiving on sockets.sofar wrote:This isn't necessarily bad. For instance, putting socket handling on a different core isn't a way to make socket handling faster. In fact, if you move low level packet handling to a different CPU, you actually decrease the network performance, because now the data may need to be copied from core to core and there will be a significant amount of cache misses for every single packet.
Here are a list of threads on a minetestserver:
- Server - Main thread
- Emerge-0 - Map generation
- ... Emerge-1 ... Emerge-2 ... etc
- ConnectionSend - sending
- ConnectionRecei - receive
- CurlFetch - not sure. Announcing to server list?