In-Memory Key-Value Databases (ncache/redis/homebrew)

Well, I guess the only way to be sure is to test them out anyhow.

I shall report on my findings, I just finished installing a Centos7 vm and I'm compiling Redis 5 as I type. I'm also going to install the free/open version of NCache and see how they compare for code complexity/use.

The testing will have to wait 'til morning time though(almost 1am here).

Well, shoot, compilation and installation was about as smooth as just about anything I've ever had to compile myself on linux.. ever.. Good signs for Redis already.

I have continued this to my project blog:

https://www.gamedev.net/blogs/entry/2265641-in-memory-key-value-databases-redis-initial-evaluation/

If you wish you can follow along from there, I'll be evaluating NCache next.

Redis is great, as long as you don't need to scale past a single machine per game instance. If you don't need persistence, you could just use pub/sub and leave it at that.

Another option for pub/sub is ZeroMQ, btw. It's decidedly non-persistent, and you may need to build your own central broker that actually lets everyone else rendez-vous to pub-sub, if you don't want everybody to connect to an existing server.

But, more importantly: Why do you have multiple different servers that need to know player location? Why not build a single server process? The constant factor overhead in trying to build a "scalable" solution that distributes frequently-changing CPU-decided data (like physics simulations) across multple machines is gigantic. Game servers are not web applications. How about putting all code that needs player location onto a single machine, and just read the locations when you need it? Once you run out of vertical scalability on that machine (at 1000 or 10000 or 100000 players) then shard your world.

47 minutes ago, hplus0603 said:

Redis is great, as long as you don't need to scale past a single machine per game instance.

What makes you say that exactly?

47 minutes ago, hplus0603 said:

Another option for pub/sub is ZeroMQ, btw. It's decidedly non-persistent, and you may need to build your own central broker that actually lets everyone else rendez-vous to pub-sub, if you don't want everybody to connect to an existing server.

I will look into it, thanks!

47 minutes ago, hplus0603 said:

But, more importantly: Why do you have multiple different servers that need to know player location? Why not build a single server process?

I'm building a massive persistent online world in a "single shard" environment.

47 minutes ago, hplus0603 said:

The constant factor overhead in trying to build a "scalable" solution that distributes frequently-changing CPU-decided data (like physics simulations) across multple machines is gigantic.

Indeed it is gigantic. But not quite as.. Only the position relay server needs subsecond resolution. It or one of its clones will speak to the client(s) directly and will be populating this data into the database for other servers to reference.

47 minutes ago, hplus0603 said:

How about putting all code that needs player location onto a single machine, and just read the locations when you need it? Once you run out of vertical scalability on that machine (at 1000 or 10000 or 100000 players) then shard your world.

There will be no sharding in that sense. I'm using a single data warehouse to hold all of the non-persistent data so that I can run clusters of functionally specialized servers. The primary goal of which is to allow a shared reality to propagate across an internet sized network.

Check out my project blog if you want, I would love more feedback/suggestions/critiques.

My experience from massive simulations is that any attempt to "overlay" simulations (not clustering them geographically) will fail to scale, because the communications overhead (networking) is orders of magnitude higher latency and lower bandwidth than NUMA RAM. Giant persistent worlds are totally possible, and have been built for 20 years, but they all end up sharding by geography and replicating entities across shard borders in one way or another.

If you don't need any kind of physical simulation or physically-based rules (enemy targeting, spells, meelee combat, etc) then you can probably go with a fully sharded "web style" network topology, but then, what would your game be? The world can only see so many Hearthstones or Clash of Clans.

Anyway, I say Redis is great, because I've used it for a number of different use cases, and I've seen several other companies use it, and it's great as long as you stay within a single machine. Once you need a cluster of machines, many of the things that are good about Redis, such as transactional cross-key updates, stop working. If all you use is pub/sub, and you shard the pub/sub key space across separate Redis instances, that can scale higher, but it has the draw-back that you get NxM network connections -- each interested subscriber process needs to connect to each Redis instance, and because both Redis and subscribers scale by number of players, it ends up growing as O(N-squared) albeit with a very small "k" value.

Redis is less great at persistence, because once the data store gets big, you need machines with tons of RAM, and forking the Redis process to persist it will lock up the kernel for seconds at a time if you have enough RAM. You can somewhat alleviate this by sharding across tons of small Redis instances (say, 32 GB each or less) but then you instead have a bigger "M" problem for connections. Use a real database for persistent data.

Awesome, killer info!!

Thanks! Yeah, I guess there are a lot of different ways to use the word shard when talking about this stuff.. haha I assumed it was mostly a term to describe the over-all effect of the experience. In reality I guess my functional segmentation is actually a type of sharding all on its own. So, my game universe will be a single shard experience, but the servers are sharded every way you could possibly imagine.

In order to avoid the static geographic clustering that you mention, I'm attempting to devise a system that is distributed and dynamic. Instead of moving players from shard to shard as they move through the world, the "shards" will move with the players, merging and separating from the shards of other players on the fly(by proximity). It's all still highly theoretical and mostly only makes sense to me when my eyes are closed. haha

My primary purpose for THIS system is to centralize transitory data so that other systems, shards if you will, can access it within a reasonable time frame(before it falls out of recent history, the short-term memory for my universe). Without this, I'm going to have multiple clusters of servers spitting data at each other left and right, regardless if the receiver really needs it or not... Not happy making.

Thanks again, I really appreciate the info!

@hplus0603 Have you looked at Riak at all? It kind of "looks" like a more scalable version of Redis to me..

http://docs.basho.com/riak/kv/2.2.3/

I would be very interested to hear your take on it if you have one. It has a memory-only back end option and is built to be clustered/redundant in even a minimal installation(3-4 servers). I'm going to give it a spin next I think and see how it goes.

Thanks again!

You talk of multiple servers, and you also talk as though you are a single individual.

Pick a solution that fits your actual problem. People pick something that is high performance for a different industry but terrible for games, or that scales to support thousands of machines when they won't ever need it.

Consider your game at its most successful, but while still small enough to not hire anybody. How many transactions per second will it need to support?

Realistically, for many games the peak rate is a single digit number of tiny transactions per second. Even if the game were to become successful and make enough money to start hiring additional programmers, the peak rate for thousands of users can reach tens of transactions per second.

Typically games don't realistically need clusters of servers until they're also large enough to have large teams of developers. When they grow to that point it is easy enough to hire somebody to write that code. If your game is small enough you can't hire people to write and maintain databases, then it is also small enough that you don't really need that technology.

And if you're the person hired to add that to a game that is large enough to require such a cluster, you likely don't need that kind of tech until you are reaching a thousand transactions per second or more. If you're in that boat, you'll have time to do a full evaluation of many technologies and probably wouldn't be asking here.

43 minutes ago, frob said:

You talk of multiple servers, and you also talk as though you are a single individual.

Oh, if I had some better multiple personality jokes... zipzoombang!!!

I sense just a little bit of callus and skepticism in some of the replies I get to my questions these days.. haha. I'm fine with that.

At the moment, yes I am a single individual, and yes I am writing multiple servers, 6 at the moment. I have a partner joining in the project soon, he's also very capable, but not really a programmer.

43 minutes ago, frob said:

Pick a solution that fits your actual problem. People pick something that is high performance for a different industry but terrible for games, or that scales to support thousands of machines when they won't ever need it.

A solution that has a capacity to scale, that exceeds the need, is what I'm trying to pick. I'm simply trying to find something that will(eventually) scale, this kv database is just a tool to keep things in sync. In fact it's just a really big centralized Dictionary(with extra functionality), not unlike the Dictionaries and other KV types I'm already using in my server code.

43 minutes ago, frob said:

Consider your game at its most successful, but while still small enough to not hire anybody. How many transactions per second will it need to support?

Realistically, for many games the peak rate is a single digit number of tiny transactions per second. Even if the game were to become successful and make enough money to start hiring additional programmers, the peak rate for thousands of users can reach tens of transactions per second.

Indeed, I'm not building a global network of game clusters right now, but I am attempting to build my system in a way that can scale up to..etc. As well as being reusable for future gaming projects or expansions on this one. And I would like to start with a lot of extra capacity.

43 minutes ago, frob said:

Typically games don't realistically need clusters of servers until they're also large enough to have large teams of developers. When they grow to that point it is easy enough to hire somebody to write that code. If your game is small enough you can't hire people to write and maintain databases, then it is also small enough that you don't really need that technology.

The game environment I've created is already about as massive as I could imagine, so in actuality I'm just trying to match my server architecture to the capacity of my game environment. It's hard to imagine trying to populate a world full of things without the capacity to keep track of it all.

And, my entire background is in enterprise systems engineering, I can build and maintain servers and server clusters, I can build and maintain databases, this is stuff I do without it feeling like work. I've also been programming in one language or another since I was a child and I'm almost 40 now. So, I'm leveraging my knowledge and experience and building my dream project from the ground up. Since I can write that code, there's no need to hire somebody else to do it.

43 minutes ago, frob said:

And if you're the person hired to add that to a game that is large enough to require such a cluster, you likely don't need that kind of tech until you are reaching a thousand transactions per second or more. If you're in that boat, you'll have time to do a full evaluation of many technologies and probably wouldn't be asking here.

I am doing a full evaluation, and no I'm not being paid for it, the first part of that process(for me anyhow) is asking around...

I'm not here to waste anybody's time, I'm providing rudimentary benchmark results for my evaluations(as I linked above) over on my project blog, in case anybody else gets crazy wild and wants to play with big tech too.

If anybody else has some experience with THIS(in-memory scalable KV databases) technology I would love to hear about it. But for the most part I think I've decided on Redis as my "currently workable" solution. This is a pretty simple technology when it comes right down to it, and most of the players are so cross-pollinated that it's hard to tell their open source versions apart. The enterprise versions are where they really differ, but I'm not really interested in those product offerings right now anyhow.

Thanks Again!

Good to know. Too often that's a place where people make some serious mistakes, investing extended time and effort to plug in an architecture designed for millions of distributed transactions per second when a very simple data store is more than adequate.

Very often this type of storage is best pronounced:
std::map< std::string, std::string >

And if you must back it up with a database, run an upsert on it every few minutes.

For a simple server that's good for many gigabytes of information, assuming you're on a 64-bit system like your server probably is.

2 minutes ago, frob said:

std::map< std::string, std::string >

Unless you live in c# land like I do. haha, then that just looks vaguely like some c I used to know... How's that song go.. "Like somebody I used to know.."

lol

"had your friends pick up your records and changed your number..." bwahahaha

ear worm GO!!

On a more serious note, if these databases ever exceed 1gig in size, there's certainly a problem. I think I'm mostly just having trouble explaining how I intend to use these things I ask about. People see my rep number and think I'm running full speed into a brick wall. In effect it's going to be more of a messaging system, with ram level persistence. Because I don't like the idea of messaging directly between machines as that requires WAY more code and way more traffic than centralizing the data ever could, plus that wouldn't be scalable unless I plugged in or wrote something like ZeroMQ... Bleh..

It isn't about the rep number, it's about how frequently it is brought up.

People come in asking how they can integrate a solution that handles terabytes per second for their networking solution that would take a large team multiple work-years to implement, when they're unlikely to ever produce a finished project.

They come in asking about "what is the best one", looking at online reviews. Perhaps best for a large business system distributed across cities, but not for small games.

On 10/31/2018 at 12:57 AM, Septopus said:

I'm building a massive persistent online world in a "single shard" environment.

Hmmmm.... You sound like you're working on the exact same thing I am. I think I'm just approaching it from a different angle.

52 minutes ago, Gnollrunner said:

Hmmmm.... You sound like you're working on the exact same thing I am. I think I'm just approaching it from a different angle.

Indeed. Except, you are performing magic, and I'm playing with legos. Maybe slightly magical legos, but still. Lol

@frob Yeah, I can see that. That's pretty much what most of my posts look like too. Haha. To be honest I expected far more criticism and far less actual "community" than I have found here. Guess that's why I keep posting.

6 hours ago, Septopus said:

Indeed. Except, you are performing magic, and I'm playing with legos. Maybe slightly magical legos, but still. Lol

I wouldn't necessarily say that. In some ways you are farther along than me. Also you seem reasonably competent an you have a lot of enthusiasm. That should take you a long way if you stick with it.

Riak has a lot higher overhead per transaction than Redis. It's a different kind of beast -- more of a document database (like Amazon Dynamo, or Cassandra/Scylla,) than a network-attached-RAM server. Redis is more like memcached, although with persistence. But still not durability! Redis will checkpoint, and you will lose anything changed after a checkpoint when you crash. (It also has other modes, but they too have a variety of problems.)

If you need actual persistency, just use MySQL or perhaps Postgres. Or one of the high-performance reliable key/value databases like rocksDB or foundationDB if you don't need advanced schemas/indexing. (just promise me you won't use MongoDB -- it promises many things, and ends up failing anyone who tries to scale a business and evolve an application with it.)

Regarding shards that move with players: Two players may want to go different directions, so the only way you can avoid having to move players between shards would be to have one shard per player. There are network topologies that work exactly like that -- the original military DIS standard, as well as the over-engineered HLA replacement, are structured like that. They end up bottlenecking on the network, typically at much fewer simulated entities than you could squish into a single process if you used static geographic sharding.

Btw, I can see how the word "shard" is overloaded. In systems architecture, "vertical sharding" is when you separate different functional areas to different systems, that you can scale individually. "horizontal sharding" is when you can distribute the load within a single system onto many different machines (in games, typically geographically.) What players of MMORPGs call a "shard" is generally called an "instance" in systems architecture -- you spin up multiple full copies of whatever your system (game) is, and let players choose one instance to interact with. Instancing is kind-of like another level of horizontal sharding, except not really, because the workload across instances is not interchangeable; a datum (player character) only makes sense within the instance.

Awesome info again @hplus0603 Thanks! More or less the same conclusion I came to regarding Riak. More than I need/want.

Redis so far has done nothing but make everything I've touched with it a little(or a lot) better.

I've disabled all of it's checkpoint and persistence capabilities, it's basically just a highly supercharged memcached(preliminary tests of memcached in the same environment agree). With pub/sub that works amazingly, well enough that I'm completely trashing all the inter-server netcode I WAS writing. It's really quite brilliant, and everything I was hoping it could be. I'm kicking myself hard for avoiding centralizing the data sooner. I'm currently using it entirely asynchronously to store short-term game-state data and for its phenomenal pub/sub capabilities to keep my servers in sync.

By shards that move with the players, I mean something more along the lines of (brace yourself) semi-authoritative clients. Everybody builds game servers like mainframes, and I know why that is. I'm not arguing the merits of a fully authoritative server architecture(I'm still building one, sort of), but I can't help but see most of it's faults too and think that there are ways to use the network/system resources more efficiently and more importantly, improve the quality and interactivity of the simulation by making all simulation dependent decisions as close to where the player makes theirs as possible.

I see it as a trust-but-verify type of scheme with a somewhat decentralized authority, and not one that cares exactly what's being rendered @xyz in every players sim at every single moment. More what is Feasible to have been(within the rules) or Should Have Been(because a server sent an NPC to get them) rendered near the position they said it was at the time they said it was... I really have no idea how it's going to work out, so far so good is all I can say.

Ultimately, I envision a game where a lot of the content is AI and/or otherwise dynamic and interactive,yet remains fully operational even if all the other real players have been lag-dropped from your game session. You can go about exploring the world and/or making changes as you like, even if your lag is so terrible you can't see any other real players. A system that can filter the game down to its most basic operational state for almost anybody, playing almost anywhere. Possibly replacing lag-inaccessible player characters with non-networked NPC analogs that derive behaviors from the actual players themselves. I got ideas...

But that's why I need all the super-fast yet temporary storage that is accessible from all of my various servers. It will act like a round-robin-database that only keeps a certain window of history ready to re-game cheats(verify/approve client decisions), and for other general behavior analysis and misc. uses by the games AI.

At all costs I'm trying to avoid writing code that plays with time. I'm not going to add time to time stamps so they look like they didn't happen in the past, so that my server simulation can allow a loot drop to happen in the future.. or some other bull.. haha I'm not even sending time stamps with my data. I'll simply check the immutable historical records to see if what the client said happened(within the tested client latency window of the claim's arrival) vibes well enough with the record, and if so, approve the loot drop that should still be waiting on other processes client side anyhow(like a more realistically elongated death/destruction sequence).

yeah, most of this I'm just putting into words right now, so if it seems a bit unpolished... haha