Contract Complete at Electronic Arts

Published September 25, 2023
Advertisement

I just finished my contract position at Electronic Arts. It was supposed to only be for a year, but my team liked my work a lot and found a way to extend my contract by another three months. This is my post mortem reflection on working at EA.

I worked on the UI team as a remote worker on the west coast. Our core team was located in Orlando, Florida but we had team members spread all over the world. We had a hand full of people from Romania, a few people from other parts of Europe, some people from Canada, Las Vegas, LA, and a few other places. It was truly a geographically distributed team. It was a mixed blessing. Most of the team meetings needed to be coordinated around time zones, particularly times which could accommodate west coast time and eastern european time zones. That often meant I had to be awake and ready for stand ups at 8am local time, or even 7am in some cases. I have to admit, I'm a night owl, so usually I would wake up for those meetings with five hours of sleep. I don't know about you, but when I am tired, I have to fight off brain fog and try to sound coherent. Sometimes coffee helps, but I found what worked best is to write up a paragraph or two the night before on what I needed to say, and then I would just read my paragraph out loud to the zoom call. It worked particularly well when I needed to be precise with my descriptions on technical stuff. Most of the time, you could pretty much guarantee that someone somewhere in the world was awake and working. So, for a night owl like me, if I needed to run something by someone, I could reach out to someone on slack in Romania. Or, during the day, I could reach out to someone in Florida. This accessibility was pretty awesome.

The team I worked with at EA was awesome. Everyone was super helpful and supportive of each other, and if you needed help, the responsiveness was usually within minutes. It has arguably been the best team I've had the pleasure of working with in my whole career. When I started my contract, they put me through an on-boarding process and assigned me an on-boarding buddy. The on-boarding process to get your environments setup and running, to find the learning documentation, tutorial videos, and just general undocumented technical knowledge, is all a lot to tackle. The on-boarding buddy is there to help guide you along and keep you from getting blocked. What they don't tell you is that your on-boarding buddy is someone who never goes away, so if you ever get lost, confused, stuck, or blocked on anything, you can reach out at any time and get help. On boarding buddy not online or available? Ask anyone else on the team and you get the same enthusiastic support. I don't know if this culture was unique to the team I was on or if its widespread all throughout EA, but it will be the high bar I measure all future work teams against. I'm a bit sad that I don't get to continue working with such great people.

On the technical side, the staff was full of super smart people too. I often felt that I was the least smart person on the team and that I had a lot to learn from everyone else. It would be tempting to fall into the cognitive trap of imposter syndrome with such a vast array of talent, but I had to remind myself that I too brought my own unique gifts, talents and experience to the team. We had one guy in Romania who was an absolute wizard at HLSL and working close to bare metal on the GPU. He had render doc mastered, knew how to squeeze every fraction of a millisecond of performance out of the GPU, and pretty much was a rendering performance god. Another guy in Romania was also a similar god of low level architecture and implementation of the rendering and sim update sections of the engine. He was also one of the most helpful people on the team (not to diminish everyone else's helpfulness). A lot of the time, engineers would talk and I would hear english words being spoken, but it all sounded like technical gibberish to anyone who wasn't deeply immersed in the particular area. “Runtime data bindings”, “Imperative Render”, “Dynamic state stream”, “Flux capacitor hyper vane tuning”, etc. Trying to keep up with the terminology is tough, especially when people invent new terminology to describe the thing they invented. I can say without any doubt, the engineers working at EA are world class smart. If I was going to complain about anything though, I'd say that sometimes the engineers are too smart for their own good and they'll get in their own way by introducing unnecessary complexity or over engineering things. It's something I try to avoid as well by following the principle of Occam's Razor and the idiom “Let's keep it simple for now, we can always add complexity later”. The trend of over engineering and over complicating things seemed to be a recurring problem for the Frostbite engine (in my humble ape brain opinion). It wasn't overtly obvious, but the problem of over complication presented itself in subtle ways – workflows were more complicated than they needed to be, too much clutter, some weird code hacks, etc. On my end, I did my best to fight this by constantly asking myself “Is this as simple as it can be? Can I remove anything further to simplify this? Does the workflow make sense for a newbie to the system and their expectations?”

I came from a background of working within Unreal Engine 4. When I first started, I thought Unreal Engine was the best game engine in the world (I still do). I couldn't understand why EA would still want to use their own in-house game engine (Frostbite) when Unreal Engine exists and has such generous terms of use and state of the art tech. I initially thought Frostbite was inferior to Unreal Engine, that it was a generation behind in tech, and why would EA continue spending tens of millions to continue funding the further development of Frostbite when they could just use the superior Unreal Engine, pay a 5% royalty, and save tens of millions in engineering costs? Was there anything that Frostbite did better than Unreal Engine? Or, was I just biased to favor Unreal because of my many years of experience in it and I was just generally inexperienced and ignorant about the capabilities of Frostbite? I decided I didn't know enough about Frostbite to make an informed assessment and that I should give it a fair shake. Now, having worked on Frostbite for 15 months, I think both engines are pretty comparable. There are some features in Frostbite which are super awesome and Unreal wishes they could have – like, you can have your game runtime up and running and you can live-edit the code and scene and see the changes happen live in your runtime. There is a lot of other state of the art tech within Frostbite which no other engine has, and I honestly think it puts Frostbite on parity with Unreal Engine. If Frostbite could make their workflows more streamlined and their editor UI less cluttered, it would give Unreal Engine a run for its money. But, let's assume that Frostbite is a little behind Unreal Engine in state of the art tech, and ask the big question “Why would EA want to continue using Frostbite instead of switching over to Unreal Engine?”

I think there's a handful of factors at play here:
1. Frostbite is a legacy engine coming from DICE and the Battlefield franchise. It existed before Unreal Engine 4 was available, so game teams at EA were already invested in using Frostbite for their next launch titles. You almost NEVER switch game engines in the middle of a project.
2. Game developers at EA are very experienced with using Frostbite. Learning a new game engine would have significant ramp up time for a team to get equally proficient. That's a pretty significant cost to the business.
3. EA leadership has invested tens of millions into developing Frostbite. Frostbite is well entrenched. Switching to a different engine would throw all of that investment away.
4. EA doesn't tell their game teams what engines to use, the game teams choose a game engine for themselves. Some teams at EA actually use Unreal Engine! EA will continue supporting Frostbite as long as game teams are using it.
5. Game teams using Frostbite have access to the Frostbite engineering teams. Do they need an engine feature currently not supported by frostbite? Then ask the engine teams to implement that feature (this happened several times on our team). The game engine engineers are there to support the game developers. Using a different engine could mean that dedicated engineering support isn't as responsive, but it could also mean less engineering support is required since an engine is already feature complete.
6. Some of the Frostbite tech is better than anything else publicly available on the market. It's a competitive advantage.

But….. Frostbite is closed source and not publicly available. That means any engine features or supported capabilities need to be developed in-house. Want VR support? You're going to need to build that API yourself and add compatibility with all the hardware devices yourself rather than having a third party help you develop support for their hardware platform. It's kind of a double edged sword.

Overall, I thoroughly enjoyed working at Electronic Arts. They're a great company. I'm sure some people have strong feelings about EA in one way or another, but I can say my experience with them has been overwhelmingly positive. At the end of my contract, my EA manager notified me of a full time opening on the team and suggested that I should apply for it. Me and one other guy applied. I think the other guy was already a full time employee working on another game team and had about 3 years of in-house experience. Between me and him, it was a very close competition, but EA ultimately ended up favoring the candidate with more in house experience. While it's disappointing to me personally, objectively I would have made the same decision and went with the other guy. Thus ends my time at EA.

For now, I am unemployed and looking for what's next in my career. I am leaning towards full time employment rather than more contract work. The uncertainty of my future is surprisingly anxiety inducing.

3 likes 5 comments

Comments

patheros

Thanks for sharing such detailed thoughts. I'm really happy to got such a wonderful experience and I hope you find someone as fulfilling soon!

September 26, 2023 12:26 PM
dpadam450

I assume you worked with Will Dull? He's easy to work with. UI is maybe different setup and Will has been there a hell of a long time. I worked at Orlando for almost 3 years, pre-Frostbite. I came back after about 5 years away and quit in 6 months. Frostbite is terrible from all angles and the Madden code is a complete disaster from my experience. Most of my team on rendering (about 10 people) left within a year after I did. I could never get the game to run, everything was slow. I had issues like tracking down who owns weather settings and how those funnel into shaders……extremely complicated and the person who setup the “blueprints” had spider webs. Gross. Lots of times we integrated things from FIFA and they broke all our stuff and wasted time tracking repeated bugs brought on by other teams. One example: in FIFA they added mini stadiums or something for a mode and they needed “projected decals”. Some guy checked that box to be default. This means that extra data was stored so that things like in Battlefields instance: bullet decals, not sure what FIFA used it for. Our memory for all stadiums almost doubled because it required some kind of 65kb alignment. Every single trash can prop model and texture. I spent like 3-5 days on that alone. I never really accomplished anything. Sometimes I would load into the “main FB branch - devNA" which was I believe game agnostic to test things out. Things wouldn't build, break, build times were horrible. Getting into the game was very slow after latest. So many times different game modes were just completely broken as well. I knew a LOT of people from my prior years, and almost nobody knew the answers to why this or that, and I'm talking basic things again like why I can't boot the game or some other weird things. I have nothing good to say about Frostbite and the CEO that greenlit the idea all games should share code because it's cheaper, they wasted like 80k on me to accomplish nothing and I would ask 20 some people that couldn't answer things. They threw all their expertise away, so use tech built in Sweden. The amount of turnover there, and then in the USA teams (that are already second hand knowledge), just terrible experience. I worked with good people there though. Every day was frustrating. I'm now remote on NBA2k and I have no meetings, standups, no stupid job matrix tracking you like a Best Buy employee, and the tech is SO much easier to follow and read. We literally load tons of animations and settings from excel. No spider web b/s junk in some random editor file that one guy knows about with abstracted weird ways to set if rain is on. I was always a generalist at EA covering many areas of code so I knew a ton of people and it's sad that almost none could help me and I would spend up to 3 days sometimes fighting Frostbite just to freaking work, or load something. Andrew Wilson's decision as a non-engineer to throw out all expertise on Madden and jump to some random Swedish code was is why I hate CEO's that have business degrees running software companies.

September 28, 2023 11:06 PM
slayemin

@dpadam450 I don't know a Will Dull. Maybe he was before my time? I was pretty tunnel visioned working just on my tasks. I spent very little effort trying to produce any games or content in Frostbite. I was never a fan of the editor UI, but that was beyond my control. My internal philosophy always was “Frostbite might be a cumbersome engine to use, but I will make every effort to make what I build the easiest to use.” I thought that the engine behavior should always match end user expectations. In that spirit, I found and fixed some weird bugs that were hackish work arounds. ie, the arc progress bar would snap to 360 degrees if you set it to 351 degrees, so you couldn't actually represent those progress values until I fixed it. I was a bit surprised nobody else had flagged it as a bug. It was probably hidden because people often used rounded end caps which would cover up the discontinuity. There were a lot of little things like this that I found and fixed, and eventually I got it perfect. So, at least that part of the engine is 100% bullet proof.
The main dev-na branch for frostbite is always going to be the cutting edge of engine features. A lot of it is developmental (hence the name)… so, teams will submit their change lists to dev-na and some bugs will slip through and break the build or crash the build farms. Shit happens. When this happened to me, I'd just wait an hour and then pull latest again, rebuild, and it'd usually be fixed. You could also just always pull the last known good if latest is too unstable. Anyways, dev-na builds weren't rock solid, so internally we would create hardened release builds where any outstanding bugs would be fixed, no new features would be added, and eventually that release build would be issued out to game teams. It would be a solid and stable build game dev teams could rely on.
The thing that annoyed me the most was that the dev-na build of frostbite would use a content example game license, which would have a library of super useful utility functions. You'd build autotests which use these utility functions, but then discover later that those utility functions are a part of the example game content, and it gets stripped out on the build farms since the build farm uses no example content. The utility functions don't exist, so your test might run fine on your local machine, but it breaks on the build farm. Super annoying. The utility functions seem like they should be a part of the base engine rather than a part of any sample content! I often scratched my head wondering "…why?"
But yeah, I wouldn't mind continuing to work ON frostbite to make it better, but as it is now, I'm not so sure I'd want to work WITH frostbite to make a game. It would just be extremely frustrating. Maybe it's me? I don't know how game teams do it, but maybe they're also just silo'd to their area of expertise and have become experts in their niche area of frostbite and have become super proficient at using it. It does mean that the learning curve and ramp up time for new joins is going to be very high and EA can't really afford attrition or lay offs without incurring significant project delay. I think EA could benefit from spending 6 months streamlining the content pipeline workflows.

October 02, 2023 10:41 PM
dpadam450

And not to just sound negative, but read the post-mortem about of BioWare making Mass Effect. They said FB destroyed their game. Respawn is smart enough to make their Star Wars games and Titanfall in Unreal. ( I think Titanfall was Unreal(?) ).

September 28, 2023 11:07 PM
slayemin

@dpadam450 Yeah, I read that public post mortem. I think they compared using frostbite to getting into a knife fight inside of a phone booth? The main thing to remember is that when Mass Effect was developed, Frostbite was probably a lot more rough around the edges at the time as well. In 2023, it's gone through many iterations of improvement and it's gotten better and more stable. It's still got a lot of rough spots to smooth over, but I would argue it's probably a lot better now than it was during the mass effect days and the same criticisms probably wouldn't be as valid today as they were years ago. I think using Unreal Engine is the right call – it's publicly available, so its easier to find and hire expert unreal engine game developers who can hit the ground running and have minimal ramp up time. With that in mind, it makes unreal devs a little bit more fungible than frostbite devs. The other perk is that Unreal Engine source code is available to third parties to work on, so NVidia, Meta, Intel, etc can all tune and contribute to the engine capabilities, performance and stability using their domain expertise.

October 02, 2023 10:52 PM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Profile
Author
Advertisement
Advertisement