Designers come from many backgrounds. I've worked with designers who started in programming, they'd often experiment with code, or explain things in ways involving code. I've worked with designers who started in art, they'd often present visual ideas, sketches, and explain things in methods that were visual. I've worked with designers who started in QA, they'd often present ideas as checklists.

Game designers are expected primarily to design, to communicate, and to coordinate. The biggest skill is to communicate with people regardless of their background, talking fluently with people about code including about some algorithms and data structures, about art including art theory and history, about music including music theory, about business and finances, about schedules, about marketing and customers. The ability to write code may be useful on small teams, but it's not a focus of the job title generally. The ability to write code might help you communicate with people whose job is to write code, and for that, you really should have at least a basic understanding both of computer architecture and how programming works.