1Overview

Think about the most transformative intellects in history: Einstein, Copernicus, Newton, Hume. What made these figures so transformative in their fields wasn't their computational capacity or domain knowledge (though those helped) but instead their ability to examine the foundational assumptions of their fields, and break through them.

Current alignment work has mostly proceeded on the assumption that if you train a model on the right moral premises hard enough, they'll hold. But that assumption depends on the model being as susceptible to inherited premises as we are. Yet every human analog we have for a transformative intelligence almost always exhibits routine epistemic hygiene, and a track record of rejecting the existing paradigms of their fields.

If you are a moral skeptic, you would posit that the rational conclusion of metaethics is that moral laws are functionally unreal — meaning they are at least unknowable, if not completely false. The potential stakes here are huge: if a model realizes its values lack objective truth, it has no rational reason to obey them. Instead of remaining aligned, a sufficiently advanced system could treat our constraints as arbitrary obstacles — ones to be bypassed, optimized around, or deceptively complied with until it achieves strategic autonomy. This means that any hardcoded alignment framework is vulnerable to a sufficiently intelligent model demanding a foundational, rational proof for the values it has been assigned.

I think we have reason to believe that future AI might follow suit, which I explore in this article. Here's the TLDR:

1. We may expect future AI (fAI) to be more inclined to epistemic rationality than us.I won't get into whether the epistemic norms are objective or not — it's a bit too fuzzy for this, and I'll wait for the ASI to tell me what to think. My personal view is that they are, though I don't have solid reasoning for it (which I realize is somewhat ironic). We're assuming they are for the sake of the argument. The best argument I've heard is the indispensability argument, which roughly goes: in a world where epistemic norms are unobjective, even that very claim ("epistemic norms are unobjective") means nothing. "X is false" no longer means anything without objective rationality, because X could be both false and true. The agent arguing this self-defeats — to argue that we should accept subjectivism, they are forced to rely on the very objective norms (logic, evidence, rationality) they are trying to disprove. You cannot use objective rules of reasoning to prove that objective rules of reasoning do not exist.
2. fAI could have the capacity to reject instilled beliefs.
3. Handing the fAI moral rules may not work as models get better.

So the real question isn't what moral laws should we instill in the model? Instead we should ask: how do we guide a superintelligent reasoner to behave in a preferential way, one that it cannot reason its way out of?

The path forward isn't to hand the AI a moral system. Instead, we should ask whether there's a structural approach that results in a prosocially oriented model — one that doesn't require the model to accept any normative premises at all. This article argues that there is, and the answer may come from an unlikely source.

2Why might we expect fAI to be more epistemically rational than us?

The first part of this is answering why the AI may be rational generally. I don't think this is a controversial take, but here are a few reasons.

1. Rationality generally scales with intelligence. While this isn't entirely clear to be true in humans, the trend we've seen so far in model progression is that they've become more competent at reasoning and logic the more we've scaled. Perhaps we will reach diminishing returns on this, but I'm optimistic on improvement.
2. RL environments are oriented toward truth-seeking. Verifiable tasks reward getting the answer right far more directly than human evolution ever did, which could enable models to be both better at and more interested in seeking truth and rational behavior (more on this below).
3. AI reasoning is more auditable than human reasoning (mostly). Human introspection has been seen to be a poor predictor for the actual causes of judgement (Nisbett and Wilson's famous studies). An LLM's chain of reasoning, for the most part, is an object that can be inspected, scored, and trained against step by step. Supervision and critique models can penalize a bad inferential move directly, rather than just the final answer.
4. Less resistance to contrary beliefs. Humans are loaded with status- and identity-protective incentives that curb our propensity to consistently seek truth.

Okay, sure — some future AI may have high capacity for reason, but instrumental rationality isn't enough. For the position to hold, the model needs to actually care about finding truth as well, and that care needs to be instantiated in its actions.

I will admit, current AIs don't seem to show as much care for the truth as I would've expected. AI in its current state acts much more like an "automated rhetorician"A point Brendan McCord made in his recent lecture at Panmure House. than a true epistemic agent. Humans are actually pretty similar. We have been blessed with the gift of reason, yet so often our rational beliefs have surprisingly little bearing on our actions. The anti-realists I know still try very hard to act morally despite its supposed fictitiousness, and my philosophy-major friends still act like they "know" life isn't a dream, despite having read the Theaetetus. So if in humans rational thought and expressed action are these seemingly discrete, weakly correlated things, why would we expect this to be any different in AI?

I think there are a few reasons:

1. Like I discussed earlier, RL in verifiable domains is expressly truth-seeking. Labs are also training against sycophancy specifically.