It's difficult. It all depends on your climate control settings. The cooler the air running through your glove box, and the higher fan speed, the better. In cool weather, if your AC is barely or not being used, the glove box will be no good for cooling.
In cold weather your glovebox 'cooler' will be a heater!
If you want a strong cooling effect, set your temp and fan speed manually for really cold air.
Since there's a separate climate control for both the driver and the passenger, why not have a separate setting for the glove box as well? (Sorry for the obvious, dumb question that warrants a simple explanation)