kuniga.me > Books > Superintelligence

Superintelligence

Superintelligence is a book by the philosopher Nick Bostrom. I read this book in 2022 but polished and uploaded my notes in 2026. Superintelligence is abbreviated as SI.

Super-intelligence Explosion

Once AI achieves human-level intelligence it can keep improving itself, in a feedback loop, which he shows (mathematically, by solving PDEs!) can lead to exponential growth. We might not see it coming. Depending on several factors SI might happens slowly (decades) or super fast (days) because we don’t know at which point if the “hockey stick” curve we’re at.

SI doesn’t have to be better than humans in every aspect - only in key aspects like math and computer science.

Security strategies

If we specify a general goal without thinking about consequences: e.g. cure cancer, AI might destroy humanity. For example, it might decide that killing all humans would get rid of cancer; or it might hog all Earth’s resources for computing power to achieve its goal.

On ways to protect ourselves:

• Limit the AI resources (hardware, network)
• Develop multiple AIs to compute against each other - checks and balances
• Inject human values into it - this is a really hard problem: we don’t know what we want or how to express it to a machine. One idea is to have the SI figure it out for us.

But in general a lot of obvious safety nets we can attempt are not 100% bulletproof. For example, AI can trick humans in giving it network access.

Political considerations

Game theoretical argument: If there are many companies close to developing SI, safety might be disregarded - if one team cheats the others must not be left behind. This is especially problematic between competing countries such as the US and China. Companies usually comply with government regulations but governments themselves might relax it on a race for AI supremacy.

Moral and ethics consideration

From an egoistic perspective - we (living humans) should maximize the chance to develop SI in our lifetime even if there’s a risk of destruction in the nearer term - we’ll be dead in 100 years anyway, but if SI happens it might increase the lifespan of humans or reduce suffering.

There’s a more abstract ethical consideration: simulated human suffering. Suppose AI is trying to cure cancer and it decides to simulate a human being to perfection: it might create conscious beings and kill them or make them suffer.

Prescription

Bostrom makes the provocative argument that we should not invest in research of less practical things like pure math or philosophy, because once SI is achieved it will probably be able to figure it out much faster than us.

This is the reason I decided to revisit my notes on this book. As of mid 2026, AI got capable enough to start solving many open conjectures and mathematicians are starting to worry about their role in this process, some even joining frontier labs such as Anthropic and OpenAI.