The technological singularity: inevitable progress or likely extinction?
By Kike Gandia · AI & Security · 11 min read
The idea of a technological singularity, a point at which artificial intelligence irreversibly overtakes human intelligence, has become a recurring theme in the media, at conferences and in the scientific literature. This article examines the real foundations of that hypothesis, the technical challenges involved and the existential risks that could follow if we fail to properly address issues such as goal alignment, model opacity and computational ethics. From DeepMind to ChatGPT-4o, we trace AI's accelerated journey from narrow, single-purpose systems toward entities with potentially general capabilities.
From Breakout to adaptive autonomy: reinforcement learning
In 2015, DeepMind unveiled an algorithm called DQN (Deep Q-Network) that could learn to play Atari video games with no prior knowledge of their rules. This milestone, published in Nature, showed that an AI system could learn directly from high-dimensional sensory input using end-to-end reinforcement learning. The system received only the screen pixels and the game score, yet it outperformed all previous algorithms and reached a level comparable to that of a professional human player across 49 different games
What was truly significant about this experiment was not just the performance it achieved, but the system's ability to develop emergent strategies. In Breakout, for example, the agent discovered on its own the strategy of digging a tunnel through one end of the brick wall so that the ball could bounce around behind it and knock the bricks out from the back. This kind of “algorithmic creativity” was not explicitly programmed; it emerged from the optimization process, showing that AI systems can come up with solutions their own creators did not anticipate.
The DQN algorithm introduced several innovations that, for the first time, made it possible to combine deep neural networks (DNNs) with reinforcement learning (RL) in a scalable way. Chief among them was “experience replay,” a neurobiologically inspired mechanism in which the system was trained during the learning phase on samples drawn from a store of past episodes. This feature proved critical to the system's success: disabling it caused performance to degrade severely.
AlphaGo and AlphaZero: artificial intuition
The real qualitative leap came with AlphaGo, which in March 2016 defeated South Korean Go champion Lee Sedol by a resounding 4-1 in a historic match. Go was considered a near-impossible challenge for AI because of its extreme combinatorial complexity, which rules out the kind of brute-force approach Deep Blue used in chess.
The first version of AlphaGo, known as AlphaGo Fan, used two deep neural networks: a policy network that produced move probabilities and a value network that evaluated positions. The policy network was first trained through supervised learning to accurately predict the moves of human experts, and was then refined through policy-gradient reinforcement learning.
The most impressive breakthrough, however, came later with AlphaGo Zero, introduced in October 2017, which dispensed with human data entirely. Unlike its predecessors, AlphaGo Zero was trained exclusively through reinforcement learning by self-play, starting from random play, with no supervision and no human data. After just three days of training, AlphaGo Zero surpassed the version that had beaten Lee Sedol, winning 100 games to 07.
Google then developed AlphaZero, a generalized version of AlphaGo Zero that could play Go, chess and shogi using exactly the same algorithm. With just 24 hours of training, AlphaZero reached superhuman level in all three games, defeating the strongest specialized programs of the day: Stockfish in chess and Elmo in shogi.
This paradigm shift, from systems trained on human examples to systems that teach themselves from scratch, marks a new frontier in artificial intelligence and suggests that the limits of human knowledge no longer constrain what these systems can achieve.
- 2015 — DQN learns Atari without the rules
Pixels and score only: the system learns from high-dimensional sensory input and discovers the Breakout tunnel strategy on its own. - March 2016 — AlphaGo defeats Lee Sedol
4-1 against the South Korean champion in a game considered out of reach for brute force. - October 2017 — AlphaGo Zero does without humans
Trained only through self-play starting from random games, within three days it beats the version that defeated Lee Sedol 100-0. - Later — AlphaZero generalizes
The same algorithm reaches superhuman level in Go, chess and shogi after 24 hours of training, beating Stockfish and Elmo.
Two years to go from learning a video game to beating the best program at three different games.
The Clever Hans effect and the illusion of understanding
We must avoid anthropomorphizing AI's achievements. The so-called “Clever Hans effect” — named after the famous horse that appeared to do arithmetic but was actually responding to subtle, involuntary cues from its trainer — reminds us that systems can look intelligent without any genuine understanding. Many of today's AI systems, large language models included, do not understand the world the way humans do; they correlate patterns at massive scale.
DQN systems, for example, can achieve superhuman performance at Atari games, yet they have no conceptual grasp of what a “paddle” or a “ball” is. They work by optimizing mathematical functions that relate states, actions and rewards, without forming semantic representations equivalent to human ones. Likewise, AlphaZero can master chess, Go and shogi, but it does not “understand” these games in the human sense
Mistaking statistical correlation for semantic understanding is one of the most fundamental errors in interpreting advances in artificial intelligence. The distinction is not merely philosophical: it has crucial practical implications for assessing the real limits, risks and capabilities of AI systems.
What do we mean by artificial general intelligence (AGI)?
The technological singularity is closely tied to the concept of Artificial General Intelligence (AGI), a term describing hypothetical systems capable of performing any intellectual task a human can, and even outperforming humans at it. The concept, however, remains ambiguous and contested.
The technological singularity is defined as “the hypothetical advent of artificial general intelligence,” a point at which “algorithms, computer networks or robots may be able to design or build computers or robots better than the ones that already exist.” The theory suggests that repeating this cycle of improvement could trigger an “intelligence explosion,” in which machines would design ever more powerful generations of themselves.
The concept was first mentioned in 1957 by the mathematician John von Neumann, who suggested that “the ever accelerating progress of technology and changes in the mode of human life give the appearance of approaching some essential singularity in the history of the race.” Later, in 1965, I. J. Good developed the idea by describing an “intelligence explosion,” in which machines slightly surpassing human intellect could improve their own designs, setting off a cycle of self-improvement.
The technological singularity is characterized as “a theoretical scenario in which technological growth becomes uncontrollable and irreversible, culminating in profound and unpredictable changes to human civilization.” This phenomenon would be driven by an artificial intelligence that surpassed human cognitive abilities and could improve itself autonomously.
The term “singularity” in this context borrows from mathematical concepts denoting a point where existing models break down and continuity of understanding is lost. The theory suggests these advances could unfold so quickly that humans would be unable to foresee, mitigate or stop them.
Emerging risks: systemic failures and catastrophic errors
The history of technology shows that even the most sophisticated systems can fail catastrophically because of seemingly trivial errors. A textbook case is the failure of Ariane 5 flight V88 on its maiden launch on June 4, 1996. The rocket, carrying four European Space Agency research satellites worth millions of dollars, self-destructed 37 seconds after liftoff because of multiple software design errors.
The technical problem lay in legacy code, originally designed for Ariane 4, that lacked proper protection against integer overflow. Ariane 5's flight path differed from its predecessor's, producing higher horizontal velocity values. As a result, a data conversion from a 64-bit floating-point number to a 16-bit signed integer overflowed, triggering a hardware exception. To keep the system's processing load within the required 80%, the programmers had protected only four of seven critical variables against overflow, relying on assumptions that held for Ariane 4 but not for Ariane 5.
The incident, regarded as one of the most infamous and costly software bugs in history (more than $370 million lost), is a perfect illustration of how small inaccuracies in complex systems can have disproportionate and devastating consequences.
- 37 s
- for Ariane 5 to self-destruct on its maiden flight
- 64 → 16
- bits: the conversion that overflowed and caused the hardware exception
- 4 of 7
- critical variables the programmers protected against overflow
- $370M
- lost: one of the costliest software bugs in history
A trivial flaw in a complex system, in numbers.
In the context of advanced artificial intelligence systems, especially those capable of self-improvement and autonomy, such failures would no longer be isolated incidents but systemic, potentially irreversible risks. A general AI system with poorly specified goals or implementation errors could, in theory, escalate these problems to levels that would be impossible to contain once deployed.
The goal alignment dilemma
One of the fundamental challenges in developing advanced artificial intelligence safely is the alignment problem: how do we ensure that an AI system's goals and behavior stay aligned with human values and intentions, even as these systems become increasingly autonomous and complex?
As research on the AI alignment problem puts it, “alignment is defined as the congruence between the goals of AI systems and human values and goals.” A lack of proper alignment not only raises ethical challenges but could lead to significant existential risks.
Three key factors make this problem urgent: “the speed of technological progress, global competition in AI development, and insufficient public and academic awareness of the issue.” In extreme scenarios, “an advanced, misaligned AI could pose an existential threat to humanity.”
- Pace Speed of technological progress
- Capabilities are growing faster than the methods for verifying them can mature.
- Race Global competition in AI development
- Whoever slows down to align loses ground, so the incentives push against caution.
- Attention Insufficient public and academic awareness
- The problem is debated in narrow circles while deployment is already happening at scale.
The three factors that make the alignment problem urgent.
The challenge becomes especially critical when we consider AI systems capable of self-improvement, like those theorized under the concept of the technological singularity. A system designed to optimize a seemingly benign but underspecified objective function could develop unexpected and potentially dangerous behaviors in its drive to maximize that function.
This problem is not merely theoretical. Even in today's AI systems we see unwanted behaviors emerging from poorly specified objectives: from recommendation algorithms that maximize engagement by promoting polarizing content to generative AI systems that produce toxic or misleading output when they are given the wrong incentives.
Instilling values in a machine is a monumental task. There is no universal philosophical consensus on what the good is. Kantian, utilitarian and deontological traditions offer divergent and sometimes incompatible views.
The problem gets harder when we consider advanced AI systems such as AlphaGo Zero, which develop their own internal representations from first principles without human intervention. These systems show that an AI can discover strategies and concepts that took humans generations to develop, but they also raise questions about how to ensure those discoveries are aligned with values humans would actually want.
While systems like DQN can optimize to maximize a game score, optimization in the real world involves complex trade-offs between diverse and often conflicting values. Should a medical AI prioritize longevity over quality of life? Should a resource allocation system weigh economic efficiency above social equity?
These questions, which moral philosophy has debated for centuries without reaching definitive consensus, now demand operational, algorithmically precise answers. The challenge is not only technical but deeply philosophical: we need to translate abstract normative concepts into concrete computational specifications, a process riddled with ambiguity and uncertainty.
Where are we heading?
Progress in artificial intelligence is moving at a dizzying and accelerating pace. If we trace the path from the first DQN systems that learned to play Breakout, through AlphaGo beating the world Go champion, to AlphaZero mastering multiple games with no prior human knowledge, we can see a marked acceleration in what these systems can do.
This progression fits the notion of the technological singularity as a point where “intelligent machines would design ever more powerful generations of machines, developing an intelligence increasingly superior to that of humans.” As I. J. Good warned in 1965, the first of these improvements may be small, “but as the machine becomes more intelligent, it could set off a cascade of self-improvement and a sudden surge of superintelligence.”
The critical question is not merely technical but sociotechnical: how do we govern these technologies? Who decides which values get programmed in? Which regulatory frameworks constrain them? In 2025, we find ourselves in a race against time to develop effective governance mechanisms before AI systems reach levels of autonomy and capability that make them hard to control.
As research on the alignment problem notes, “given how quickly the technology is evolving, it is crucial to address the alignment problem immediately and in a coordinated way to mitigate potentially catastrophic consequences.” That urgency only grows when we consider that AI systems are moving ever closer to what we might describe as artificial general intelligence.
The technological singularity is not a single future event but an acceleration curve that has already begun. From DQN's first successes in Atari games to AlphaZero's achievements in complex strategy games, we are witnessing rapid progress toward ever more capable and autonomous AI systems.
The real challenge is not whether we will build systems that surpass general human intelligence, but whether we will know how to manage the consequences. Alignment problems, the risk of systemic failure and ethical questions are not minor technical hurdles; they are fundamental challenges that call for urgent, coordinated solutions.
As a technological society, we must champion scientific exploration and innovation. But we must also stress the urgent need for robust, interdisciplinary and global governance of these emerging technologies. We cannot hand the future of humanity over to systems that, by their very nature, have no intrinsic understanding of the value of life, freedom or human suffering.
Neither enemy nor savior. Artificial intelligence will not inherently be our enemy. But nor will it automatically be our savior. It will be whatever we decide — or fail to decide — to build. And that responsibility, for now at least, remains entirely ours.
Related service
AI and LLM penetration testing
Topics
#AI #AlphaGo-and-AlphaZero #DQN #DeepMind #Clever-Hans-effect #Intelligence-explosion