Probably not much I can add to this discussion, but it is interesting. I would say that there are two ways game development can be hard. I also have a third point.
The first is not being able to implement your vision due to crappiness of stuff, as a programmer, I find lots of times I want to do something, but because someone had to make it overcomplicated, I can''t do it. API''s like OpenGL are moving in the right direction. Many people reinvent the wheel... and do it poorly. I have messed some with COM objects, an example of something which is so frustrating it prevents good programming(personal example, some people like COM).
The other way it is hard really hacks me off. I had a guy who wanted me to "teach him how to make a 3d engine". So I started with some basic math(BTW, the math skills are only requisite when you''re rolling your own 3d engine, it helps elsewhere, but not as much as clear thinking). Turns out he had been programming three weeks. I suggested he invest some skills in learning programming, this horribly insulted him, as he felt that he should just be able to do it, or I should give him code(which I did, but he couldn''t figure out how to use it), or that in general, the world should somehow provide him with what he wanted. The process should be lubricated as much as it possibly can, but to some its like saying, "I want to write great literature, but I don''t want to bother with actually having to write the book". Suck it down.
My last point(after the flamebait) is that eventually some kinds of programming are refined until the creation of application specific tools are the penultimate expression of the requirements. Perl is an example for text processing. For the old-school text adventure games, I understand it has reached that stage of refinement. For the rest of gaming? Nowhere close, but I would ask you to consider the mod community and Unreal Script, these are the beginnings. Sure there is always a need for better graphics and sound, but at some point, an avenue has reached a stage of refinement where the specialization involved makes the creation of certain tools a neccessity. In fact, I would suggest games will arrive at such a solution in a hybrid fashion. The parts of games which are not fully explored(graphics) will be written in general programming languages with API''s, and the rest of the game will be written in a refined language that will allow the rest of the game features to be expressed concisely.</p>