Incorrect picking when using glPointSize()

Because it is what is rendered into the picking region. That's the points and nothing but the points. Instead draw it solid for the picking, then redraw it as points for display.

Selection in OpenGL works by determining whether the object intersects the clip volume or not. Basically, the object is transformed by the modelvew and projection matrices, and if the object is still within the view volume, a hit is recorded.

No fragments are rasterized, only transformed and compared against the clip volume. However, line width and point size are rasterization parameters, and therefore does not affect selection. What happens is that the point's coordinates are transformed, and even though the point may be rasterized where you perform selection due to the point size, the point's center can fall outside the selection region.

To solve this you need to increase the size of the picking region so that it's large enough to include the center of the point even if you click not on it's center, but still on a rasterized pixel. So if you have a 30 pixel large point, you need a 30 pixel large picking region aswell.

You said "if I instead use GL_POLYGON...". So my assumption is that you are drawing the vertices of a polygon as points and wondering why you can't select the polygon by clicking somewhere logically on the interior of the polygon.

You have a picking region defined by the gluPickMatrix call. That region is a square 5 pixels on a side. The first two parameters are the lower left corner of that region. So assuming (x,y) is the mouse coordinates then you have to click within 5 pixels to the left and/or below a pixel of the point. Assuming you drew in white with a black background that would be the white pixels forming the point and none of the black pixels that are logically part of the polygon since you didn't draw those pixels.