Smart Until It's Dumb - Emmanuel Maggiori

Oct 22, 2025

Author: Emmanuel Maggiori, PhD
Title: Smart Until It's Dumb
Edition: Applied Maths Ltd. 2023, 200 pages (eBook)
Language: English

Since the release of ChatGPT in 2022, the hype around artificial intelligence knows no limits: all sorts of companies attempt to outdo each other in implementing and praising AI in their products. Research and projects in AI receive billions of dollars from governments and investors. Not a single day goes by where the media do not report about some new financial record in the AI sector or a new breakthrough in AI research or philosophizes about the consequences of AI mass adaption for work and society. Nothing seems to be impossible: AI art, AI based medicine, AI vehicles, AI which makes those who are long dead speak again, AI which writes Code by itself, maybe even develops itself and it seems to be only a matter of time (and processing power) until AI is equal to human intelligence, if not superior.

Yet there have been critical voices from the very beginning, casting wellfounded doubt on the narrative of a highly efficient AI which is already known to replace human specialists and is on an unstoppable ascent towards omniscience. One of those voices is Emmanuel Maggiori, PhD , who is anything but a technophobic Luddite, but an expert in the field of artificial intelligence who knows what he is talking about: Maggiori not only holds a PhD in Computer Science, but can also point to his academic work in the areas of deep learning, machine learning and artificial intelligence as well as his long-term work experience in the tech industry, both in start-ups and large companies like Expedia and Vodafone.

In the book at hand, with the subtitle “Why artificial intelligence keeps making epic mistakes (and why the AI bubble will burst)”, the author dissects some of the common myths that have sprung up around AI, explains how current AI works, what it can and what it cannot do and which developments can be realistically expected.

History, foundations and mechanics of current AI

“Machines will be capable, within 20 years, of doing any work that a man can do.” ~ Herbert A. Simon, 1960, p. 14

The first two chapters provide an overview of the history of AI development as well as today’s state of the technology. We learn that hype and skyhigh expectations have been constant companions to AI ever since the emergence of computers. Again and again, phases of enormous interest and towering financial investments into AI have alternated with phases of disappointment, sobering up and stagnation.

For example, already in the 1960s and 1980s both governments and private industry in the USA, but also in Japan, had invested millions and billions into AI reasearch, enticed by the promise of imminent breakthroughs like autonomous vehicles or systems for language recognition and automatic translation. However, when it became clear that the technology was not capable of fulfilling these hopes, both investments and scientific interest dwindled to a minimum during the so-called “AI winters” of the 1970s, 1990s and 2000s.

The renewed interest in AI, which emerged during the 2010s, is directly related to the triumph of a new, highly successful methodology called machine learning. In a way that is understandable for lay persons, Maggiori explains the basics and the workings of this methodology, upon which most of current AI rests. He also portrays how it shaped the way of working and everyday work of AI developers/data scientists as well as further concepts of machine learning, like reinforced learning and supervised learning. In the same vein, he breaks down in an understandable way how deep learning works, a specialized methodology for the most complex AI applications and the basis of ChatGPT, deep fakes and image generation.

In this context the author also shows how some common utopic visions as well as apocalyptic warnings are equally absurd and simply not possible, given the workings of current AI. For example the vision of an AI that can learn by itself, completely independent from human involvement, or the scenario of an AI that comes to the conclusion that humanity should be exterminated for reasons of efficiency.

Nevertheless Maggiori points out some real and existing dangers of machine and deep learning. They are less apocalyptic in nature but oftentimes the result of faulty - or questionable - data as basis of AI models, or of undiscovered bugs, they stem from elusive and hard to comprehend deep learning models or from the fact that existing models can be fooled and derailed in shockingly simple ways. The following chapter goes into details about these problems.

Artificial confusion

“If you will, current AI is a sophisticated way of cheating.” ~ p. 71

In the third chapter, about which Maggiori himself says that it “may be the most important of the book” (p. 10), he lists a number of cases where AI produced results that were not very intelligent but very confused. Moreover, the reader does not only receive some entertaining tips how to completely confuse ChatGPT with some innocuous questions, but also an explanation for this performance: namely that current AI is not able to understand anything, does not have a mental model of the world and of the relations of things towards each other and instead solely works based on calculations of statistical probability.

Maggiori explains what this means precisely and what AI is accordingly good at, in which areas it is prone to errors and which approaches are simply impossible to implement with current AI. He points out that AI models, lacking an understanding of patterns, will oftentimes forcibly create patterns where there are none, as a result of how machine learning and deep learning work.

This jumping to erroneous conclusions can become a serious problem, but statistically less likely occurences are where AI gets completely lost. Based on the example of autonomously driving cars it becomes clear how even tiny deviations from “learned” situations can lead to devastating consequences. For example, an inconveniently placed traffic cone or a sticker on a traffic sign is enough to completely incapacitate even advanced models of autonomous vehicles.

Overly optimistic statements, like that by Elon Musk, who divined in 2015 that fully autonomous vehicles would become a reality within three years, are rebuked by the explanation of the fundamental limitations of modern AI. According to Maggiori, dreams like these are not possible to realize with the current state of AI. Furthermore we are reminded that continuous progress in AI reasearch is by no means a foregone conclusion and that game-changing new discoveries do not inevitably have to happen within a few years.

Shams and pretense

“Well, we cannot say that we cannot build this. We said a year ago that this was possible, so now we can’t say it’s not.” ~ Anonymous product manager, p. 106

In chapters 4 and 5, the author relays some of his personal experiences from AI projects in the corporate world and in scientific research respectively. Maggiori points out some typical approaches within the corporate world which reliably lead to AI projects ending up in disaster: it begins with knowing the solution before knowing the problem. Companies are caught up in the general excitement or fall prey to the fear of missing out (FOMO) and decide: “We want to do something with AI, too!” What that something is and if AI is really the best solution and creates most value for the intended use case, are questions of secondary importance.

Moving on, it is not uncommon that fundamental errors in AI models remain unnoticed, everything seems to work just fine, but in reality the results are consistently faulty and misleading. As soon as the first minimal successes are in view, there is also an eagerness to jump-start entire AI teams from one day to the next and hiring experts, without knowing who is needed and how many people are needed for the AI project at hand.

The most impressive - and entertaining - part of Maggiori’s field reports are those episodes that relate to the stage of concealment, cover-up and wilful deception within a company, after at least the developers have come to the realization that their “AI” is not working, and will never work. However, management and other departments cannot be brought in on this fact if the members of the AI team do not want to risk their jobs. If eventually the truth is uncovered, those who were deceived resort to cover-ups and deception as well, instead of admitting to the public that they and their company did not notice years of wasted working hours, resources and money.

Maggiori paints a similarly sobering picture of the world of scientific research. The author relates from personal experience, that the results of reasearch projects are often “touched up”, which is not only open knowledge in the scientific community but also sometimes explicitely advised. For example, problematic results are being ignored while a single impressive achievement is presented as the overall level of performance of an AI model, or models are developed which have mastered a specific data set but are otherwise worthless, or successes are announced in overly sensationalist language, in order to generate attention and financial funding.

Ghost in the machine?

“I know a person when I talk to it. It doesn’t matter whether they have a brain of meat in their head. Or if they have a billion lines of code.” ~ Blake Lemoine, p. 131

Finally, the sixth and last chapter takes a look at philosophical questions, such as the origin of consciousness. Maggiori introduces the computational theory of mind (CTM), which holds that ultimately the human brain is nothing more than a computer made of flesh, which runs computer programs. According to this theory it would be possible to extract those programs and also save and run them on other media, so that on the one hand artificial consciousness could arise from computer programs and on the other hand human consciousness could live inside computers and be copied and transferred at will.

Maggiori elaborates on some related, in his own words, “sci-fi scenarios”, introduces classic thought experiments like the “Chinese room” by John Searle and explains why accepting CTM does not only mean that the existence of free will has to be ruled out, but that it would also be reasonable to assume that thermostats are conscious and perhaps lead lives of quiet suffering. Opposing positions are discussed as well, taking both the insights of quantum physics into account as well as the fact that our knowledge about physics and the human brain still remains fragmentary and limited. It becomes clear why the question about the origin of consciousness remains one of the most difficult questions altogether, both from a philosophical and science-oriented point of view.

These sometimes profoundly fascinating, sometimes decidedly absurd considerations, theories and thought experiments are not simply presented for the joy of abstract musing or idle philosophic speculation. The author emphasizes that they imply tangible and grave ethical consequences and furthermore define the limitations and possibilities of AI development. For example, if consciousness can not be produced out of matter, then the development of an AI with consciousness is as impossible as every AI fanatic’s dream of artificial general intelligence (AGI): an AI that is equal to the human brain, that can understand every intellectual task and perform every kind of work.

Unsurprisingly Maggiori is not able to give a definite answer to these questions, instead he invites his readers to give these questions some thoughts of their own. So while it is not possible to conclusively say whether an AI with consciousness and AGI are even possible, Maggiori ventures to predict that even if that were possible, it would happen “not anytime soon” (p. 162), given our limited understanding of several topics crucial to these matters.

Final verdict

Given that the ubiquitous and unbridled hype around AI still reigns strong, this book offers a much-needed corrective. Here, an expert explains, in language accessible to lay persons, the current state of AI research, where and how AI is used, how it works and on what foundations it rests, what it can do and what it cannot do. We receive a realistic picture that shows both impressive successes as well as worrisome failures. The potential and inherent limitations of AI are displayed and it becomes clear that all that glitters is not gold, that not every hype should be followed blindly and unreflected and that not every grandiloquent announcement by a corporation or by science should be taken at face value.

Maggiori’s writing style is never dry, boring or lecturing, but vivid and entertaining to read, like I already mentioned in my previous reviews of his book Siliconned. Difficult abstract theories and questions are presentend in an accessible way, but of course they cannot be treated in a comprehensive and exhaustive way, which is also not the goal of the author by definition. It should also be mentioned that Maggiori again succeeds in entertaining the reader with personal anecdotes and bizarre episodes from his career and in eliciting head-shaking as well as laughter.

On account of the previous cycles of AI hypes and “winters” which are portrayed in the book on the one hand, and on the account of the unrealistic expectations in AI and the way it is approached by businesses and science, sometimes in negligent ways, sometimes in ways which cross the border to fraud on the other hand, it comes as no surprise that Maggiori does not think that the current AI hype will end well either. While he expects machine learning and deep learning to continue making progess and being increasingly adopted, he rejects expectations of miracles and spectacular breakthroughs.

The inevitable hard clash with reality, economic fallout included, as well as the application of AI that is based on wrong assumptions and applied to fields for which the state of technology is not (yet) suited, understandably cause Maggiori concern. The book closes with a summarizing section of “frequently asked questions” about AI and is on the whole discernibly Maggiori’s attempt to provide some desperately needed clarifications about the topic of artificial intelligence and to call for more realism. In my opinion, he succeedes very well in this endeavour and it can only be hoped and wished for that books like this one and the facts it contains will become more widely known.