System memory vs Video memory performance issue

Hi I have a little performance problem with my game i''m creating. My game consists of a map and a bunch of players with weapons. The players should be able to blow holes in the map during the game and when they do small pieces of the map should fly away and bounce on the map. It looks a little like worms but is not turn based and will in the future have support for multiplayergames. My problem is how to store my map surface since i have to both read from it to check if a piece will bounce or not and write to it to be able to make the holes after the weapons. Since i need this i have to Lock() it. When i store my surface in the memory of the videocard i can draw the map very fast but the reading and writing is slow. Thus when i store the map in the system memory the drawing is slow but the reading and writing it quick. Does anyone have a third suggestion of how i can make this go faster? You might think that the time is so little that it will not matter but trust me it will when you have a lot of objects checking the map to bounce or not. And my game has alot of objects cause it is a true action game help needed /Dogen