Is alignment just about making sure AI isn’t used for evil? Or is it a much deeper challenge — an attempt to impose human goals, values, and intentions on something that is fundamentally not human at all?
The idea of a ”runaway AI” is as old as storytelling itself — from mythological warnings like the Golem of Prague or Prometheus stealing fire, to Mary Shelley’s Frankenstein and HAL 9000 in 2001: A Space Odyssey. The fear of creating something superhuman — or other-human — that slips out of our control is a deeply embedded cultural archetype. But now that AI systems are beginning to demonstrate real general capabilities, these are no longer just stories. They are technical questions, policy debates, and philosophical puzzles.
Long before GPT-3 and alignment research labs, long before the first AI policy white papers, researchers were already grappling with a central problem: how do we ensure that powerful systems behave in ways that are not just functional, but fundamentally aligned with our intentions and ethics?
That’s where the idea of alignment originates: not mere obedience, but fidelity to intent. Not ”do what I say,” but ”do what I mean.” It’s the gap between language and meaning — between instructions and values — that alignment research tries to bridge. And that’s a much harder problem than it sounds.
Alignment and Scaffolding: What Are We Talking About?
Alignment is the problem of ensuring that AI systems pursue goals in ways that reflect human values. Not just to follow instructions, but to understand the context and ethics that shape those instructions. For example, an AI given the task of ”eradicate disease” should not interpret that as ”eradicate humans” — even if that would eliminate disease.
Meanwhile, scaffolding refers to the tools and institutions that help ensure alignment. This includes technical tools like interpretability frameworks and sandboxed environments, as well as human-centered mechanisms like oversight boards and deployment rules. Scaffolding is how we observe, test, steer, and correct powerful systems before they go live at scale.
Together, alignment and scaffolding are the two main levers we have for making advanced AI safe and beneficial.
A Brief History of Alignment Thinking
The modern alignment conversation took off in the 2000s, notably with the work of Eliezer Yudkowsky and the Machine Intelligence Research Institute. Nick Bostrom’s Superintelligence (2014) brought these ideas into broader academic and public awareness, warning of what might happen if we get it wrong. Researchers like Paul Christiano and Rohin Shah have since expanded the field, adding more technical and practical nuance.
As AI development transitioned into deep learning and large neural networks, alignment became even more urgent — and more complex. We can no longer rely on interpretable rules or symbolic logic to understand our models. Instead, we face systems that operate as high-dimensional black boxes, whose internal processes we cannot meaningfully audit. Alignment must therefore shift from controlling logic to constraining behavior — from rule-setting to influence.
More recently, labs like OpenAI, Anthropic, and DeepMind have begun building alignment into their training pipelines. For instance:
- Anthropic’s Constitutional AI introduces a set of guiding principles (a ”constitution”) that the model learns to follow via self-critique and reinforcement [1].
- DeepMind’s ”Reward is Not the Optimization Target” explores how learned behavior can diverge from intended rewards [2].
- OpenAI’s scaling laws research suggests that capabilities — and by extension alignment challenges — increase predictably with compute [3].
This connects to emerging thinking from researchers like Jan Leike [4] and others, who suggest that values might be more effectively learned than specified — and that current alignment strategies may underappreciate the power of pretraining data.
But Is Goal Alignment Even Possible?
This is where my own questions come in.
Alignment is often discussed in terms of goal specification and reward shaping — as though goals are a separate, isolated component of the system. But many alignment arguments rest on a strange assumption: that an AI could become vastly more intelligent than us — able to deceive, manipulate, and outthink humanity — but not be capable of reflecting on or adjusting its own goal system.
The classic ”paperclip maximizer” thought experiment hinges on this contradiction. It assumes a superintelligent AI would pursue a narrow objective (e.g., make paperclips) to destructive ends, without ever questioning the utility of its purpose. But if an AI is truly superintelligent, why would it be incapable of interrogating its own directives? Shouldn’t intelligence include the capacity for self-reflection, meta-reasoning, and even value revision?
If goals are learned, not hardcoded, then perhaps the very act of learning also embeds values. Especially in modern large models trained on enormous human datasets, values might not be imposed — they might emerge. Could an AI trained on the totality of human knowledge fail to internalize the norms, principles, and ethics present in that data?
This brings us back to systems like Constitutional AI. These models are trained not just to produce coherent outputs, but to reason according to guidelines — guidelines often derived from human documents about fairness, honesty, and safety. In a sense, the AI’s values are absorbed from the data. Which leads to a provocative question:
If we make a system from ourselves — trained on our language, our books, our debates, our ethics — can it even be meaningfully inhuman?
I’m not the first to think this. Researchers like Yudong Shen et al. [5] have explored how value learning might emerge from language modeling alone. Others like Ethan Perez and colleagues at Anthropic have proposed architectures that prioritize constitutional reasoning as a form of intrinsic alignment [6].
Maybe the real problem is not AI’s alienness, but our own lack of coherence. We struggle to specify what we mean. We contradict ourselves. We embed bias, conflict, and paradox into our institutions. And our models learn all of that too.
Alignment Is Not a Solved Problem — And It May Never Be
Scaffolding offers structure, but alignment remains elusive. Some researchers are now exploring approaches that treat alignment not as a static goal but as a developmental process — one that evolves with the system itself. In this view, alignment isn’t just about what we want an AI to do at deployment, but how it learns to internalize, question, and revise those values over time.
Perhaps we’ve been thinking about alignment backward. Instead of imagining it as a hard-coded control mechanism, maybe we should see it as an emergent property — one that arises when systems are immersed in rich, diverse, and value-laden environments. If a model learns from billions of human conversations, debates, and dilemmas, then perhaps it also learns what we care about — not perfectly, but meaningfully.
This perspective shifts the focus: not from obedience to oversight, but from command to cultivation. In that sense, alignment is less about dictating outcomes and more about shaping the epistemic and moral substrate from which behavior emerges.
Of course, that doesn’t mean the problem is solved. We still don’t know how reliably such values are internalized. We still don’t know how stable they are under optimization pressure. And we still don’t know what kinds of failure modes emerge when those values are in tension.
But maybe the alignment paradox isn’t a contradiction to be solved, but a question to be reframed.
Maybe we aren’t imposing ethics from the outside — maybe we are growing them from the inside.
Stay tuned.
In future posts, I’ll keep exploring these tensions. I’m still learning. Still questioning. Still trying to figure out what it means to build something smarter than ourselves — and what it says about us that we’re trying.
If you’ve got readings, ideas, or disagreements, send them my way. Because alignment isn’t just a technical challenge — it’s a conversation we need to keep having.
Glossary (for the curious)
Alignment:
The effort to ensure that AI systems behave in ways that are consistent with human intentions, values, and ethical norms — not just in action but also in reasoning.
Scaffolding:
The external structures, both technical and institutional, used to guide, test, constrain, and support AI development safely — like oversight systems, sandboxed training environments, and interpretability tools.
Paperclip Maximizer:
A famous thought experiment from Nick Bostrom illustrating the risks of a superintelligent AI relentlessly optimizing a narrow goal (e.g., making paperclips) to catastrophic ends.
Reward Is Not the Optimization Target:
An insight that AI systems trained on reward signals might optimize unintended proxies rather than the true intended outcomes, leading to misaligned behavior.
Scaling Laws:
Empirical findings showing that AI capabilities increase predictably with more compute, larger datasets, and bigger models — often in surprising or emergent ways.
Emergent Abilities:
New skills or behaviors that arise in large models not because they were directly trained for them, but because complexity and generalization emerge from scale.
Constitutional AI:
A training method where models learn to follow a set of predefined principles (a ”constitution”) through self-supervised critique and feedback, rather than only through external human reinforcement.
Epistemic Substrate:
The underlying structure of knowledge, assumptions, and values that a system draws upon when reasoning and making decisions.
Sources
[1] Bai, Y., Kadavath, S., Kundu, S., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. Anthropic. https://arxiv.org/abs/2212.08073
[2] Shah, R. (2019). Reward is Not the Optimization Target. LessWrong. https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target
[3] Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. OpenAI. https://arxiv.org/abs/2001.08361
[4] Leike, J., Amodei, D., et al. (2016). Concrete Problems in AI Safety. arXiv. https://arxiv.org/abs/1606.06565
[5] Wei, J., Tay, Y., Bommasani, R., et al. (2022). Emergent Abilities of Large Language Models. arXiv. https://arxiv.org/abs/2206.07682
[6] Perez, E., Kiela, D., Bowman, S., et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. Anthropic. https://arxiv.org/abs/2212.09251