Whether your speedo (and thus odometer) reads accurately depends on whether your overall tire diameter changes. The standard tires on my '04 V8 are 235/50-17, which yields an overall radius of roughly 21.7 inches, so to maintain that radius, 20" wheels could only have a 1.7" sidewall.
According to the Tire Rack Web site, the lowest profile tires they carry are 25s, but at that aspect ratio with a 1.7" sidewall, we get a tire width of 170mm. If you go with a reasonable width (like a 235 or wider), you're necessarily going with a taller sidewall, which changes the overall radius.
The overall circumference, as we all remember from high school geometry, equals two times Pi times the radius. That means for every revolution of the wheel, the car goes 2*Pi*r units down the road. For my stock wheel/tire combo, that means my car goes approximately 136 inches down the road for every revolution of the wheel. If I go to a 235/25-20 tire, the car goes approximately 140 inches down the road per revolution--as a result, the car travels farther per unit of time, so the speedo reads low.
So, while it is theoretically possible to go to a 20" wheel and not need to recalibrate your spedo, in the real world, if you go to a 20" wheel and don't recalibrate, your speedo will read low.