It can really be a lot colder than it is now.
At a temperature of -40 °C, a person can walk outside without clothing for a maximum of three minutes without serious consequences. Bare skin exposed to cold below -60 °C will freeze immediately. Cells then die immediately. Anyone who goes out naked will die almost immediately.
Wind can make it feel much colder than it actually is. -20 °C in a storm can be just as dangerous as windless and -40 °C.
How high can you survive on a mountain? How long can you last without food and drink? What is the maximum weight a person can get? Curious about more limits of your body? You can read it this month from page 28 in Santé.