We could certainly survive with a brain the size of a cat...
Having a large brain improves our chances of survival exponentially, which is particularly important for us, as we tend to rely on our intelligence* rather than our physical abilities to keep our species going. Our ability to utilize abstract and causative thought helps us to develop tools and plan ways to prevent life-threaten situations.
On the other hand, our physical abilities aren't particularly impressive. We can't run very fast, our teeth and nails are too dull to make for effective weapons and our muscles aren't as strong as those of most other mammals, or even primates. Our senses of smell and hearing aren't all that great, especially compared with any of the canine species. We can see fairly well in daylight, but our night vision is terrible. Our larynx is too low in our throats to prevent choking, we're very susceptible to disease, our bones break fairly easily, and our internal organs have a nasty habit of going haywire on us. Hell, we don't even have enough room in our mouths for all of our teeth, which is why most people have to get their wisdom teeth pulled.
It's not too difficult to see why having larger brains is crucial to our species survival. Without our intelligence, it's fairly unlikely that we could survive as a species. We rely mostly on brains, not brawn.
* When I say intelligence, I mean it in the "ability to think abstractly" sense, not the "Einstein" sense. People like the person who was quoted demonstrate on a daily basis that humanity has A LOT of room for improvement in the intelligence department.
