Colonialism, per se, is not the problem. The problem is with how the mother country treats the natives of the colonies. Let's look at former British colonies, just for the sake of the example:
USA: Natives expelled and/or murdered. White people set up system almost identical to the home country, then fight said home country for independence. Natives given short end of stick until late 20th century--white people pretty much run things.
Canada: Natives traded and intermarried with, for the most part. Some tribes expelled and/or murdered. White people set up system almost identical to the home country, and are eventually granted independence as the sun slowly begins to set on the British Empire.
Singapore: White people subjugate, but do not destroy, natives. Build city that becomes major trading posts, with immigrants coming in from all over the world. City lost to Japan in WWII, becomes independent when Japan loses the war. Now a thriving international port and one of the cleanest urban areas on the planet due to strict anti-litter laws.
India: White people subjugate, but do not destroy, natives. Efforts are made to be sensitive to the less "exotic" aspects of the native culture. Anything that does not mesh with British ethical standards (suttee, for instance) is banned. System is set up similar to that of the home country, with native princes serving as local governors subject to rule of the king of England. Natives revolt and after several false starts and several violent coups, finally manage to set up a stable government. Country is poor, but at least its government is in fairly decent shape.
Any and all British colonies in Africa: White people subjugate, but do not destroy, natives. Native culture completely uprooted and replaced with British culture. Native people denied any attempts to organize or have any say in their own government. Environment and local wealth almost entirely transported out of the colony. Once-thriving regions rendered desolate. Abandoned as imperialism falls out of style and basically left to fend for themselves. Civil unrest common. South Africa is the only ex-British country in Africa that comes to mind as having a stable government--and that's only after the end of apartheid in 1990.
Let's face it, of the above examples, Singapore is the only real success story in terms of the natives not being treated like shit, the government running smoothly, AND the country not being shamefully poor.