One of the things I liked about Diablo 2 was its heavy emphasis on combat and skill-building and even cooler (to me, at least) was the system that was behind it all. Simple on the surface, but complex and rewarding underneath. Understanding how it worked allowed you to create character builds that didn't conform to the stereotypes of their classes (the enchantress, for one). One of the things that has always puzzled me was how the designers came up with the equations that the battle system and skill system both use (calculating damage dealt and received, chance to block, AR and DR, attack speed, etc). Just exactly where do they start? Did they draw up equations first and then design the whole battle system around those?

I'm no expert on battle systems, but I've designed simpler systems before and I would imagine the general idea is the same.

Basically you'd want to establish your stats first. What sort of stats you plan to have. What type of health would you plan to use. Hearts? HP bar? Numbers? Is there a defense stat? Do you use a strength stat to determine attack power or do you use weapon attack power + strength + energy level... etc. Once you've decided on how your stats will work, it's merely a matter of coming up with formulas for attack that are balanced and work well.

There is no set formula to use for a damage calculation... you base the calculation on the stats that you've come up with. It all depends on how complex you want your combat system. You could have it as simple as "player has 5 hearts. each time they are hit they take a heart damage. if they have a shield they take half a heart damage" or you could have it complex like "player has 1234 health points. enemy attacks for 100 damage. player has 300 armor points. the formular for damage taken based on armor points is 'armor points/10' from 100 enemy attack subtract 'armor points/10' or '300/10' or '30' so 100-30 = 70 for 70 total damage to player.

That's still a really primitive combat formula. But you can get the idea.

As noted, you can base a battle system on anything. But it is evident that the designers of Diablo II's combat system built up their combat system from some basic principles:

1. You kill things by reducing their health to zero.
2. You can kill things in two ways: weapons or spells.
3. Both for interest and realism, weapons need to hit to cause damage. A weapon attack is more likely to hit if the wielder is skilled (offensively), less likely if the defender is skilled (defensively). They could have made shields simply add to defensive skill here, but decided to give them a chance to prevent damage outright - this gives shields some interesting advantages/disadvantages.
4. Any damage-causing effect is of a type (magic/fire/physical etc.) which is reduced by % resistance of the appropriate type. This allows for a much larger degree of strategy in equipping a player (a necessity in a game based on loot). If there was just one damage type and resistance type, the game would be far less interesting.

The actual numbers they used weren't really based on hardcore mathematics or anything. You can see from the patch notes they had to make some big changes to offensive and defensive ratings to balance things out. I'm sure you could come up with something similar.

Well, as a longtime Paper and Pencil RPG player, I would also say that another basic tenent of a "battle system" is the idea of a check.
Basically, a check is an equation that goes like this:

Player's stat (for example,defense) + appropriate modifier (if he has some kind of magical anti-harm amulet) VS enemy's stat (attack) + appropriate modifier (for example, the enemy's sword's Speed attribute).
IF Player's roll > Enemy's roll, then Player takes 0 damage/enemy misses.

From here, you can go anywhere. You can keep it this simple, or make it really complex, like adding the ability for the player to counterattack if the enemy is off-balance, having a number of different attacks that do different types of damage or have special effects, the ability to string together combos, etc. However, if you are doing something along the lines of Diablo or D&D, this kind of check will probably the basis for your system.

Yeah, as far as I know Blizzard developed Diablo II ground-up and just experimented with ways of working stats and equipment and tweaked each individual one until it was as fun as possible.

And the best way to emulate their design technique is to do your own experimentation. I guarantee that a system nothing like Diablo II's could still be incredibly fun in a combat-focused RPG, it's just a matter of figuring out a unique solution to each individual problem that comes up.

I'd say, contrary to Konidias's idea of starting with stats, start with concepts. What's a fun action, and how might it be modified? For example, the Zelda series' ubiquitous boomerang. They had a set of ideas already: you can swing your sword, and have a secondary battle item. What might an item do? Freeze an enemy. Would it be more fun to throw something to freeze them, cast a spell to freeze them, or smack them with something to freeze them? Well, we already have a close range weapon, and spells cost magic potion, which is too big a drawback for that simple effect. Simple answer? Throw something. Pull up a wikipedia list of thrown weapons, look for a cool one, or just start sketching, and all of a sudden you have a new way for your character to interact, and it's fun, all you need to do is tweak some things (does it deal damage, how much compared to a sword, how far can it fly, how long do things stay frozen? can this be upgraded?). Those tweaks are where your stats come from, and they're essentially deduced from seeing how actions are related to each other: if I can swing a sword and shoot a gun, do I want them to upgrade at the same time/pace? base them on a dexterity stat for % to hit, with damage ratings on the equipment. do I want the player to be forced to choose between the two? Strength affects sword damage, dexterity affects gun damage, and you only get a certain amount of points to distribute between the two. This way you don't have any extraneous stats, whereas if you had written in strength and dexterity and then decided that damage is decided by dexterity and weapon damage, strength is now either a vestigial remnant like Final Fantasy VI's useless MDEF stat, or you have to invent a purpose for it, which isn't how you should be thinking, because you're adjusting to a system when the system had no purpose, instead of having a purpose and putting a system behind it.

I've designed things myself that started from stats, and you inevitably have to do huge amounts of revision once you realize what players are actually going to want to do in the game, and sometimes end up with useless stats or underpowered stats because they had no purpose in the first place. Some players like numbers, but there's no reason to give them numbers to play with that will waste their time or will not affect outcomes in ways that they can see.

The genius of Diablo II is the pacing: levels help you noticeably in overcoming the dungeons and certain enemy types, so your choices matter, and they come just frequently enough that they're worth striving for. This isn't because they wrote down a bunch of words, put numbers next to them, and then figured out how they relate. It's because they tried adding one action at a time, and if it didn't make the battles more fun, they cut it. Once they had something fun, they tweaked its existing stats until it was a viable option but didn't reduce the viability of other options, and then it was a system in equilibrium with no unnecessary bits to muddy the calculation and no single component that wasn't fun (mind you, it's not a perfect game, and a few things slipped through like the energy stat being underpowered, but it has far less of these flaws than many comparable dungeon-crawlers, and they clearly put a ton of time and creativity into differentiating the classes and giving players additional customization options that were actually worth using).

One example of combat system design fail is World of Warcraft. All the monsters have fundamentally the same abilities and behavior, which means the player is not using different strategies for different monsters, boring. This contributes to the ridiculous amount of macroing among the game's players, because each character has a maximally-effective attack pattern that remains the same for several levels, and when it does change only changes a little. My guess would be because there's a lot of difference between how the abilities of different classes work, and with more complicated and varying monster behavior they couldn't balance the monsters to be equally difficult for different classes. Also the leveling system within each class is a narrowing type, where a character's build makes only a subset of that classes' range of spells available to each character. And most builds are only good for one one play style: soloing, tanking, dps, or healing, and pvm or pvp. So that makes the game feel even more boring and restrictive - you're stuck doing the same thing all the time, possibly dependent on whether you have an appropriate mix of other builds at appropriate levels online and willing to run dungeons with you.

All this also causes problems in WoW's pvp - certain classes are clearly superior to others at particular level brackets, the difference in combat ability between characters 10 levels apart is so big that there's almost no point trying to team pvp if you're in the lower half of your bracket. Leveling causes yet further problems because it essentially splits each class into 3 subclasses, some of which no appropriate gear exists for, which is a huge problem at higher levels because gear contributes a large factor to a character's stats.