Ray tracing-how to setup those coefficients?

Your question is a bit unspecific, but I'll assume you mean the following:

Ka - coefficient of ambient reflection
Kd - coefficient of diffuse reflection
Ks - coefficient of specular reflection
Kr - fresnel reflection coefficient
Kt - fresnel tranmission coefficient

All should be in the range [0, 1)

Think of the first three as multipliers on the strength of the reflection, for instance, when calculating diffuse illuminaion:

diffuse = lightColour * dot( normal, lightDirection ) * Kl;

Kr and Kt are calculated from the fresnel formula. Most forms of the fresnel formula will give you Kr, and from this Kt= 1 - Kr.

For the fresnel formula you can either use the physically based version (google), or Schlick's representation, which is an approximation that looks roughly the same for dielectric materials, but is much faster to calculate (again google for it).

Thanks, that makes it clearer.

But just for testing the algorithm, can I just set those number casually( without considering the fresnel formula). I mean can I simple set ks=0.3,kd=0.8,ka=0.4, kr=0.2,kt=1-kr=0.8. Is there some rule about the relationship among them. For example, the sum of ks and kd should not be over 1, or kr should have some relationship with ks and kd?

Why these number could not be 1? You mentioned[0,1).

Sorry again for these trivial questions, but I googled and got no clear anwser about these.

Well you really want to have a value per-wavelength, i.e. a colour. You don't want to have any of them equal to 1 because that would be a perfect reflector and such a thing doesn't exist in nature.

Drop Ka completely: don't use "ambient" illumination: it's a filthy hack.

Kd and Ks should add up to less than one.

What brdf are you trying to implement anyway?

I googled a little again. As you mentioned and I thought, Kd and Ks should add up to less than one. But in this file which made me confused again:
http://www.renderman.org/RMR/Publications/arman/shaders/material.h
there are several materials with kd+ks>1.0.

Actually what I am trying to simulate are just cornell boxes as I mentioned in the other thread. I want to render two scenes, one has two diffuse blocks in it, and the other has a mirror ball and glass ball.

I just used the classical ray tracing to simulate the diffuse scene, the roof is totally black and I am wondering if it is correct. So I played with those coefficents a little bit and aroused these trivial questions.

Thank you again.