"A New Phase, Not a Full Circle" Interview with Andreas Hotho

Interview: Elena Müller, GI

Andreas Hotho and the German Informatics Society

Elena Müller: Would you mind introducing yourself and maybe explaining your relation to the German Informatics Society and to IJCAI?

Andreas Hotho: I’m Andreas Hotho. I’m professor for data science at the University of Würzburg, and I’m also the spokesperson of the Center for Artificial Intelligence and Data Science in Würzburg. I work in the area of natural language processing and knowledge graphs, among otherthings, and I’ve been a long-time member of the German Informatics Society. I’ve organized AI conferences and workshops for many years, so there’s a close relationship to the AI community, which is one of my research areas and where I’m actively involved.

EM: Have you ever been to IJCAI before?

AH: To be honest, I’ve never been to IJCAI. I’ve been to ECAI before but I never had the chance to make it to IJCAI. I’m typically at the machine learning conferences, the European and international ones, but IJCAI never happened for me. As I mentioned, though, I’ve organized the German AI conference in the past, so there’s still a close link to the AI conferences.

Hybrid AI: A New Phase

EM: Not long ago, rule-based AI systems that follow explicit logical rules, like older expert systems, were considered a dead end. Now there’s renewed interest in hybrid AI systems that combine today’s data-driven learning models with those older rule-based approaches. Would you say AI is coming full circle, or is this a new phase?

AH: I think it’s a new phase. Rule-based systems never really stopped being of interest to the research community; there was always ongoing work pushing them forward. There’s a clear rationale for that: humans work with the kind of inferencing and uncertainty we now call AI, but we’re also very good at switching over and saying, there’s a clear pattern, a clear rule that explains what we’re observing. That fascination with making such reasoning formal and explicit never went away.

The problem was always operationalizing this in machines: making the smooth transition between rule-based reasoning and uncertainty that humans manage so easily. For a long time this was hard for AI systems. Now, strong language models are able to make that transition, to a point where we can apply rule-based approaches more easily again, especially once people realized it’s not so easy to get rid of uncertainty and errors, and rule-based systems can help with that. So there’s a new way of thinking about this relationship from a technical point of view, and that’s why I see it as a new phase, one where we try to make use of rule-based AI within language models.

Symbolic Reasoning: Limits and Possibilities

EM: Would you say that large language models will ever replace symbolic reasoning?

AH: It’s a complicated question. I’d argue it’s very unlikely there will be a full replacement. These models keep getting more capable, and we as humans can deal with this kind of reasoning behavior, but not everyone can think in formal logic or complex mathematics. AI models will push the boundaries forward, but I’m not sure they’ll become a typical symbolic reasoning system where you can formally deduce information within the model itself.

On the other hand, we already see tool capabilities: systems that ask external tools for help. I can easily imagine that becoming more flexible, so we get better integration between both worlds and can benefit from that. We also work on integrating knowledge graphs into LLMs. From that perspective, I’d argue it’s very unlikely large language models will ever fully replace symbolic reasoning.

EM: Where do today’s AI reasoning systems still fall short compared to specialized logic-based software?

AH: They fall short with very complex sets of relationships and fine-grained or complex mathematical operations. Math was one of the first shortcomings of large language models, and developers responded with reasoning traces, where a model spells out potential solution steps. But once the complexity of these problem traces gets too high, the models can’t grasp all the relationships anymore. Solving that would mean making the models even bigger, and they’re already very big. So for highly complex symbolic reasoning, the setup is still more complicated than what current language models can reliably handle.

EM: As AI systems get better at step-by-step logical reasoning, what do you see as the bigger challenge: drawing strict logical conclusions, spotting general patterns, or forming the best educated guess from incomplete information?

AH: It’s a combination of both strict logic and pattern recognition. On one hand, these models are pattern finders: they look for visible patterns and generate answers from that, and they’re very good at it. On the other hand, humans are very good at jumping between making an educated guess and then applying a specific rule to reach a goal, and that combination is still missing, or only partially there, in the form of reasoning traces that lay out possible solution paths and try to connect them.

When problems get too complex, requiring a lot of information to be put together, these models fall short, but humans as well; it’s hard for us to keep an overview too. Looking at past successes, especially in mathematics, models have already solved problems in ways we hadn’t seen before, so in some areas they’re better than humans. What’s missing is pushing that further into much more complex problems, where a small piece of missing information still needs a rule or a piece of logic applied to reach the final solution. Tool calling again can be another solution. Fitting pattern finding and logic together in a smart, tightly coupled way, that’s the crucial challenge.

World Models and Embodied Understanding

EM: One thing often named as a missing piece in AI is having a working model of the world, some internal understanding of how things behave, like knowing that a glass could break if it falls. Would you say world models are the missing ingredient for the next step in AI?

AH: That’s a good point. Given that current models are typically trained exclusively on text, it’s amazing what they already achieve without direct access to real-world information or visual perception. I like the idea of moving toward world models where the objective is predicting the next state of the world, which in principle could include physics and all sorts of information.

What’s missing is that everything mankind has written down, summarized in text, either has to be relearned from scratch, or we need a way to merge that physical understanding with abstract, textual information. You can design training objectives that link visual and textual information, multimodal models already show this, but truly increasing world understanding needs a strong relationship to explicit symbolic and textual information too, and I’m not sure that’s an easy path. It will be very interesting to see what happens over the next years; I think it’s one of the next big research directions, especially for robotics. I’m involved in a Bavarian government-funded project on foundational vision-language and world models to support the next generation of robots, and that’s a very exciting next step.

EM: Would you say language alone can teach AI a true understanding of the physical world, or does it need real embodied interaction with that world?

AH: The question is what understanding even means in AI. There are papers digging into what’s going on inside these models and how they self-organize. I’m involved in a project training a sovereign, open-source model with data and infrastructure in Germany, and if you look at how simple the training objective is, predicting the next token, it’s astonishing that there’s emergent self-organization storing information about the world. The question is whether that self-organization counts as a kind of understanding. There are already models that no longer simply follow what they’ve read in training data; they abstract and start to follow their own internal rules, which suggests a kind of reflection, maybe a kind of understanding.

Whether physical interaction is necessary for that is hard to answer, since understanding itself isn’t well defined. If you’ve never had any perception of the physical world, you could run into trouble imagining how the world looks or what physical information means. But think of humans who are blind: they still understand what’s going on in the world, they feel and sense it in other ways. So honestly, I can’t look deeply enough into these models to answer definitively. It could be that language alone is enough, or it could not be.

From Single Systems to Social Settings

EM: Most current AI research evaluates individual systems with benchmarks. In the future, AI systems will increasingly act as participants in complex social settings alongside humans and other AI. How can research shift more toward evaluating those social systems instead of only single systems?

AH: When the first language-model chat systems came out, we were surprised they could generate coherent, nicely readable text. That was around 2019. A year or two later we were surprised again that they could answer questions well, so we went back to existing benchmarks and tested how far the systems could go. They improved so quickly that we just kept inventing new benchmarks. The pace is so high that the research community can hardly keep up with designing the right measures to capture what these systems can and can’t do. We see this, for example, in software benchmarks where language models now find bugs and propose fixes automatically, which wouldn’t have been a serious benchmark five years ago. We need to stay on par with these systems to keep challenging them, though we can’t invent benchmarks for tasks we can’t solve ourselves yet.

As for integration into social settings, this will be one of the next big steps, and it’s already happening: people work on persona research, systems that emulate certain personas, and so on. The bigger problem I see is that we as a society use these systems but are no longer really in control of them. Big companies push the boundaries, and as academic researchers we can’t fully investigate what these systems can and cannot do, because we lack the access and the resources. The crucial question isn’t just how to evaluate them, it’s how to find their boundaries, what we should allow as a society, and how we control things when we don’t have the resources to run or develop such systems ourselves.

EM: What would be your suggestions for how to handle that situation?

AH: We need to decide as a society that we want at least a basic understanding of these systems, which means investing money in developing them at a scale where we can follow what’s going on. We don’t need to be first to invent every next step, but if we fall too far behind, we won’t understand what the real problems are. You already see this in cybersecurity, where systems can work together to reach certain goals, including hacking other systems. In Europe, especially in Germany, we’re currently not able to do this ourselves. We don’t have enough compute, we don’t have these models under our own control, and we don’t know the full recipe for setting them up and tuning them, for example to defend our own systems or to know what’s technically possible so we can design fair regulation. That’s an underestimated risk. These systems will definitely impact society, and right now we’re losing control.

I’m involved in the SOOFI project on training a sovereign model, where we built what was considered a large model some years ago, thirty billion parameters, now an even bigger one with 120B. That’s no longer large by frontier standards, and even building it was very expensive, more of a one-shot effort than a thoroughly researched model. We’d need a lot more of this kind of work, and many more experts, to really understand these systems and advise decision-makers. The longer we wait, the further behind we fall.

Is There a European Way of AI?

EM: Is the European Union prepared for very strong AI from other regions, or to develop it itself? There’s often talk of a Chinese way and an American way of doing AI. Would you say there’s a European way of doing AI successfully, and if so, how?

AH: If you just ask whether there’s a European way, I’d jokingly say yes: ignoring AI and waiting for others to solve it. But you added successfully, and there I’d say we still have a chance to catch up. The projects we’re involved in, Bavarian initiatives and others, show that we’re able to train our own models. But we need money and support, and a shared understanding that if we want to stay in control of this technology, we can’t just wait for others to deliver it and assume everything will be fine.

As long as we lack that understanding and the willingness to invest, nothing will happen. I hope governments will recognize the dependency we’re heading into, both the economic and the societal risk, and that if they invest a reasonable amount of money, industry can pick up and build on the models we prepare. We need to close the gap to the leading AI companies; that would be the solution, investing now and taking the first step. I’m also surprised Mistral hasn’t published models since some time or truly taken technological leadership in Europe, even though it’s one of the last significant European players, despite a large share of its capital being owned by American and other investors. They could take the lead, but they haven’t, so it’s worth asking why, if they’re effectively the only serious competitor here. I’m not sure we can still fully catch up, but my suggestion is to invest, train our own researchers and engineers, and buy the hardware. Hardware is the next dependency and expensive, but it’s necessary.

Closing Thoughts: AI and the Future of Data-Driven Research

EM: Is there anything else you’d like to add?

AH: The most critical point I already made in the last answer. I’m involved in several projects where we train our own LLMs together with colleagues, a big consortium of researchers across Germany. My final point would be that these models, and their accelerated, almost exponential development, will change not only computer science but all research based on data.

Over the past twenty or thirty years we’ve digitalized nearly every research domain, so we now have a lot of data and insight; sensors and measurements are far better than before. We can make use of all that data with these new technologies. Once that becomes widely understood across the research communities, the rush toward AI models and usage of LLMs will grow, not necessarily be because the models get more smarter but because the pace of development and usage keeps accelerating. This will not only change society, but also science itself and how it works.

EM: Thank you very much.

Transcribed and revised with Otter.ai