I'm mainly a full stack web developer and never had problems with big (in size) repository projects, so I could just push everything on GitHub without any problems.
Now I decided to make some games and I was wondering how should I manage my repository. Well probably the idea is to add the game assets directory into the .gitignore file, but how do I share the rest of the project with the other developers? How are we going to stay in synch with game asset changes?
babaliaris said:
so I could just push everything on GitHub without any problems.
just wondering… have u found a problem doing the same thing for your game project ?
babaliaris said:
How are we going to stay in synch with game asset changes?
really wondering…how did u stay in synch with your web project ?
ddlox said:
babaliaris said:
How are we going to stay in synch with game asset changes?
GitHub says:
In addition, we place a strict limit of files exceeding 100 MB in size.
https://docs.github.com/en/free-pro-team@latest/github/managing-large-files/about-git-large-file-storage
GitHub and Git in general is designed for text. Text deltas, text compression, and so on.
A distributed version control where everyone gets a copy of everything in history works well as the typical text is a few megabytes compressed.
Large Git repositories reach many megabytes. It's rare but sometimes big systems hit a gigabyte, needing manual approval and tripping safety precautions. Clones take ages.
This model is horrible for games.
Games are almost all data. The entire code base can be smaller than a single image. The entire code history of a AAA game is likely smaller than the source of the game's startup movie clips.
Not even considering history, source data for large games is often better measured in terabytes than gigabytes. My last project we had a 10TB volume for the working copy of the latest built (not source) data. Artists work with only their photoshop docs, which themselves can sometimes hit hundreds of megabytes when layers are thrown around. Source audio uses lossless compression and audio streams reach gigabytes quickly. One small change results in tremendous data changes, especially when the only diff tool assumes diffs are text, rather than images, compressed audio, or compressed video.
A distributed version control where everyone gets a copy of everything in history is awful.
People have hacked together workarounds where the version control systems like Git or Mercurial, which has no actual history, instead just a history of where the data (hopefully) resides. If people are amazing with storage and data persistence you might even be able to reproduce an old build, like the ones generated a few days or weeks back.
Subversion is sometimes used by novice groups who do not realize SVN keeps both a pristine copy and a huge body of metadata on each machine, and the work is often done by the clients rather than the server. So even a few gigabytes ends up with new disk drives for everyone on the team.
The industry mostly uses Perforce. For small projects there is a free license, I think up to five people but it was up to 20 at one point. It is designed from the beginning around large data assets. It is more server intensive, but allows you to get the entire history of the entire project, allows limiting views rather than the “everyone gets everything” model, and can be configured to hold tremendous volumes of data in archive, as many terabytes or petabytes as the company wants to maintain. Access control also allows contractors and third-party developers as big or small a view as needed.
If you want, it has an interface that let's programmers map the tiny sections of code - - tiny compared to the assets - - and access the mapping via Git. I have only worked with a few people who wanted to actually do that, or they had tools that needed it, but they are the exception.
I don't think a centralized system like Perforce has much to offer to the solo developer. The strength of such a system is that the server (plus the backup locations) needs to store the full history, but if you're working solo, you only have very few computers anyway. (And if you're not working solo, then you use what your team lead tells you to use.)
In the meantime, do you actually have gigabytes of source data? Sure, AAA games can easily exceed that, but if you're working solo, you're not working on AAA games. Plenty of indie games can comfortably fit in a git/mercurial/fossil repository.
Thanks for your advice! We are currently 2 devs and one tester/idea suggestions. We are in the planning phase right now, we didn't start the project, this is why I was wondering how to organize the repo of the project. I don't think that the game will end up being too big, but I think it will reach 3-4 GBs.
a light breeze said:
In the meantime, do you actually have gigabytes of source data? Sure, AAA games can easily exceed that, but if you're working solo, you're not working on AAA games. Plenty of indie games can comfortably fit in a git/mercurial/fossil repository.
This may have been the case for many older indie games (and remains the case for smaller/2D/lo-fi games), but some indies were pushing into the tens of gigabytes of source data a decade ago, and I'm afraid that in the current day and age, we're often talking more about hundreds of gigabytes, if not terabytes.
Free-to-use game engines like Unity or Unreal give modern indies access to fancy things like PRB renderers and HDR. A single 4K HDR texture is 135 MB uncompressed, and a standard PBR material uses 4 texture layers. That's over half a gigabyte just for the source textures of a single material.
Want to record a trailer for your game? YouTube supports streaming 4K video, which runs ~50 mbps, or over a gigabyte for a standard 3 minute trailer. The uncompressed source video for editing runs about 90GB/minute.
And so on…
