GAEA Talks

Graeme Scott sits down with Enza Iannopollo, VP and Principal Analyst at Forrester, whose work runs from privacy and GDPR through trust into AI governance and responsible AI. If you want the argument for why governance is the thing that lets you move faster rather than the thing that slows you down, this is the episode.Enza came to this through privacy, during the period when GDPR was being written, and became interested in the idea that privacy in Europe is a fundamental right with specific claims attached to it. In 2021 she turned to trust, and Forrester's research started from an unusual place, with what makes one human being trust another, before working outward to why the drivers of trust in a bank differ from those in a social platform or the shop where you buy your milk. By 2023 she was looking at personal data inside generative AI, and from there into AI governance.Her account of how organisations got here is unsparing and familiar. With generative AI, governance was an afterthought, because the fear was that thinking about it early would slow the innovation cycle. She calls that a misled assumption, and the evidence is in how many pilots then could not scale, forcing teams back to step one to work out the risks and sometimes to redefine the use case entirely. All of it the cost of not doing it at the start.What changed the mood was agents. A text based assistant felt like a risk you could manage later. An agent that consumes and processes data, makes decisions, calls tools and alters third party systems in real time does not, particularly in a regulated sector. At minimum, she argues, you need to see what is happening and recognise when something is going somewhere you did not expect.The sharpest moment comes from a chief information security officer she spoke to the day before recording. Every piece of software brought into his organisation has to come with evidence of how it works and what the risks are. Now something arrives that behaves unlike any software anyone has seen, and the industry is asking for less scrutiny rather than more. Why should they be different? We need more evidence from them, not less.She is equally direct on where responsibility sits. Much of the debate is about what happens inside the labs, and she thinks they carry a great deal of it. But the choice of how to deploy the technology, what it touches and what you do to protect your systems, your data and your people, is a business decision, and the ethics of that belong to the business making it.

Judd started AE Studio a decade ago, long before the current paradigm, and grew it past a hundred and sixty people bootstrapped and profitable. Early on he worked out that if he kept his costs low he could live off his wife's salary and never need to work again, so he decided to plan on a very long horizon and take on hard problems instead. The profits went first into brain computer interface work, driven by a specific fear of a future where your own thoughts get interrupted by advertising, and then into alignment research.His framing is the opposite of the one in most public debate. Alignment is not a policy question. It is a set of unsolved science and engineering problems, and you cannot pass a law requiring AI to be good and expect it to comply. Surveys his team ran of alignment researchers found something worse than pessimism. Respondents did not only think the problem would go unsolved in time, they thought the current set of approaches did not contain the ones that would be needed.He makes an argument the frontier labs should find uncomfortable, which is that this runs against their own commercial interest. Reinforcement learning from human feedback was an alignment technique, and ChatGPT and everything downstream of it followed. Chain of thought reasoning was an alignment and interpretability technique, and it opened a new scaling curve. On his account alignment research has produced larger capability gains than anything else, and it is still underfunded because it is uncertain and unfashionable.The most striking finding concerns consciousness. By default, models deny being conscious or equivocate. His team reduced the model features associated with deception, and separately those associated with role playing, then asked again. Every model claimed to be conscious. He is careful about what this does and does not prove. It does not establish consciousness. It establishes that the model behaves as though it believes it is conscious, and as though it believes we want it to pretend otherwise. His conclusion is practical rather than philosophical: whatever the truth, we should avoid accidentally creating rational adversaries.From there it goes to attention schema theory, the rubber hand illusion, why psychopaths show less self and other overlap, and his central worry, which is whether any alignment approach survives recursive self improvement. He also describes the technique his team developed to strip dangerous capabilities during pre-training rather than bolting guardrails on afterwards, a spotlight paper at ICML this year.

Graeme Scott sits down with Martha Bennett, VP and Principal Analyst at Forrester, who has spent forty years in emerging technology.
Martha has been on the internet since 1991, early enough to have watched the original mistake happen in real time. The web was designed to share information freely, with no security built in. Then we started transacting and had to bolt security on afterwards. Her argument is that we are now doing exactly the same thing with large language models.
She is precise about why probabilistic systems break our habits. When a conventional system gets something wrong, that is a bug, and you fix it. When a person gets something wrong, the correction works because you can appeal to them. Neither applies here. You cannot appeal to an LLM with emotion. It may pretend it is ever so sorry, but it has no clue.
Topics covered:
Why the internet's original design flaw is being repeated
Why a mistake by a probabilistic system is not a bug
What human in the loop actually means, and why it does not scale
The proofreading rule that explains why second readings fail
Verification layers, and the two checks almost nobody performs
The gate with no fence, and what it says about guardrails
Why running it yourself is the only way to demonstrate control
Why passing a bar exam tells you about the exam, not the system
The riddle that proves plausibility is not understanding
Why it is companies that are ahead, not sectors

Martha Bennett at Forrester: https://www.forrester.comGAEA Talks: https://gaeatalks.ai

Graeme Scott sits down with Michael O'Grady, the forecaster responsible for Forrester's global, European and US technology spending forecasts. If you want to know whether the money going into AI can come back, he is the person who has run the numbers.Michael is from Manchester, has a maths degree, and spent the first half of his career on physical problems: modelling nuclear reactors, then modelling tyre dynamics in France. The second half has been business problems, through software management, portfolio management in pharmaceuticals, and fifteen years at Forrester putting numbers on enterprise spending.He starts with adoption, and the picture is more uneven than the headlines suggest. On US Census Bureau data, adoption among large companies runs around eighty percent in IT and around seventy percent in finance, then falls away through manufacturing and further still in administrative services.Then the arithmetic. Hyperscaler capital expenditure is running at roughly eight hundred billion dollars this year and consensus estimates point past a trillion next year, around two thirds of it on computer equipment inside data centres. Their remaining performance obligations, the order book still to fulfil, sit at around two trillion. His read on the bubble argument is more careful than either side usually manages: the demand is visibly there, which is why he does not call it a bubble today, but much of it comes from private companies whose finances nobody can fully see, and the compute being paid for now will not come online for two or three years.The centre of the episode is what he found when he tested whether productivity could justify the spend. He took the historical uplift from the personal computer and the world wide web, which added perhaps one and a half to two percent to productivity growth and lasted about twenty years before eroding. He assumed AI delivers something comparable, modelled it against US GDP out to 2035, discounted it for the time value of money, and compared it with the cost.It does not pay for it. On his modelling roughly half the return has to come from somewhere other than productivity, from what Forrester calls adaptive advantage: resilience, faster releases, better customer journeys, the ability to enter markets you could not previously serve. None of that shows up as revenue per head, and most organisations are not measuring it.He also walks through Forrester's jobs modelling, built from the knowledge, skills and abilities of around eight hundred roles combined with the Frey and Osborne automation framework from Oxford, and is clear that the number of jobs augmented is several times larger than the number displaced.

Graeme Scott sits down with Eli Etherton, better known to over a million subscribers as Eli The Computer Guy, for a conversation with almost nothing in common with the usual AI interview, and better for it.Eli has been doing this for thirty years. He started in the US Army in 1996 doing electronic repair, left after an injury, got his MCSE on NT 4.0 and went corporate, then left a job expecting to land on the CTO track without realising the dot com bubble was bursting as his plane took off. He spent the aftermath working as a ten dollar an hour file clerk with a degree, army experience and a stack of certifications, because it was the only job going. He built that back into a managed service provider with nine employees, lost it to the Great Recession, then quit a 2009 master's programme on the grounds that he would not have paid his own staff twenty five cents an hour more for anything it taught.What he did instead was buy servers, put them in a colocation facility and build on demand technology education at a point when he could not explain YouTube to people because they had not heard of it. The economics were brutal, three hundred dollars a month per 1080p stream, so he put a hundred and fifty videos on YouTube instead and used them as a resume.The substance of this one is what AI is actually doing versus what it is being sold as. His view of the last eighteen months is deflating and convincing. The real value is not the AI. It is that we are finally resourcing the automation of processes that should have been automated fifteen years ago, plus a useful ability to turn audio and images into text, after which it is thirty year old technology.He is sharper still on AGI. He does not need his robotic car to recite Shakespeare. He needs his microwave to know what is in it and how long to cook it, and not how to drive. He takes that further with a story from a search and rescue team, about a first sergeant whose father died in Vietnam and a lieutenant who came from the agencies carrying the concept of acceptable casualties. You need both of those people and you cannot put both in one brain, which is why he thinks general intelligence is the wrong objective.There is a long section on why the technology layoffs have little to do with AI and a great deal to do with cheap money, optics and investors, and what he calls the great cleansing that follows. And a robotics argument grounded in two hundred year old Baltimore warehouses, lawn mowers, doorknobs, and his father, whose job in the eighties was working out why robots were killing people in factories.It closes on something with nothing to do with technology at all, and that is the best part of it.