Highlights From the Cerebral Valley AI Summit
Lessons learned from an amazing summit hosted by Newcomer and Volley
This week, my colleague Jon Turow and I had the chance to attend the Cerebral Valley AI Summit hosted by Newcomer and Volley. The one-day summit brought together an incredible mix of top AI/ML talent, visionary CEOs, and some of the most cutting-edge AI companies. The energy at the conference was electric, with thought-provoking discussions and inspiring talks. Eric Newcomer captured many key insights in his excellent recap which you can read here, but we thought it would be fun to share a few of our own takeaways in the below. Hope you enjoy!
1. Agents: The Next Major AI Paradigm
As Jon wrote earlier this week, "The AI agent revolution feels stuck in first gear, but innovation has been bubbling beneath the surface — innovation that will unlock AI agents across a much broader set of domains." Jon wrote about software and infrastructure-level innovations beneath the surface that are unlocking a new wave of agent innovation, and described 4 types of agents we expect to see in 2025.
At Cerebral Valley, Scale AI CEO Alex Wang expanded on this topic and called attention to a longer-term challenge to AI Agents becoming true collaborators: we lack the right kind of data for agents to understand and replicating human decision-making processes. This requires a fundamentally different approach to data collection and model optimization.
As Wang pointed out, the internet has "shockingly little data" of humans carrying out actions and documenting their processes. Even for everyday tasks like booking a flight or completing a complex workflow, we lack the kind of detailed, high-quality documented human process data, we have limited understanding of multimodal interactions. There is also a gap in complex decision-making data, even for seemingly-common tasks. In the long run, our industry will ultimately need better training approaches, observation of human activity, fine-tuning for individuals' personal workflows, and even natural-language process documentation to achieve future step changes in agentic reasoning.
It’s clear that despite these limitations, even the current generation of models can support remarkably capable agentic workflows and applications. Builders do not need to wait. Even if AI reasoning capabilities were to stand still (which they won’t), there are years of innovation ahead in how we architect, compose, and deploy these capabilities.
2. From Pre-Training to Post-Training
During a kickoff fireside chat, Alex Wang discussed how we're hitting a "data wall." Companies have run out of high-quality public internet data for pre-training models. This isn't just a temporary obstacle, but it's forcing a fundamental shift in how we approach AI development.
Training larger language models doesn't automatically result in better performance. The focus needs to be on better data and training approaches, not just scale. In response, companies are pivoting to post-training optimization through several approaches:
High-Quality Specific Datasets: Instead of broad internet data, there's a focus on curated, task-specific data that's more relevant and reliable.
Synthetic Data Generation: Companies are developing ways to create artificial datasets that capture the complexities missing from internet data.
RLHF and Process Data: There's increased emphasis on reinforcement learning from human feedback, where models are refined through targeted interaction rather than raw data consumption.
Multi-Modal Training: Complex multimodal data (combining text, images, actions, and decisions) rarely exists on the internet in useful forms. Companies are now creating controlled environments to capture this type of rich, interactive data.
3. Beyond RAG and In-Context Learning
A panel discussion with the CEOs of Glean, Together, and Coreweave shed light on the way enterprises and app developers will apply AI to their own data in 2025. Retrieval Augmented Generation (RAG) and in-context learning have been exciting developments in the field of AI since about 2020. Those techniques are exciting because they make it possible to apply models to customer data via the prompt, without requiring any fine-tuning or retraining of the models. That's faster and easier to get started from a customer's perspective.
While this is good, there's a problem. RAG and in-context learning assume that the underlying model can reason effectively about the customer's data. But developers and enterprises are discovering that models are best at reasoning on the types of data they were trained on ("in domain"). The further the customer's data is from the training set, the less effective the reasoning will be. This problem is particularly acute with smaller models, which tend to generalize less effectively than their larger counterparts.
Companies like Glean are already demonstrating how fine-tuning can enhance RAG pipelines, particularly in optimizing semantic search and embedding models for enterprise-specific data. As Together AI's CEO Vipul Prakash notes, while RAG works well for simpler needs, fine-tuning is essential for teaching models new tasks that deviate significantly from their pre-trained capabilities.
Expect more app developers and (perhaps a bit later) more enterprises to tune and train their own models in 2025, in combination with RAG, especially for open-ended, complex, or domain-specific tasks where RAG alone proves insufficient.
4. The Zero-Cost Creation Revolution
In a fireside chat with Martin Casado, Martin highlighted how AI is driving the marginal cost of creative production towards zero which is fundamentally reshaping the economics and cost structure across creative industries.
Traditional creative work often cost on average (depending on the quality):
Marketing copy: $100-200+ per page
Product videos: $500+ per video
Background music: $50+ per track
3D renders: $200+ per image
However with AI, these same outputs now cost pennies in compute. After the initial setup, the cost of producing the 100th or 1,000th piece of content is virtually identical to the first. This zero-marginal-cost dynamic has made creative composition the number one dollar-weighted use case for AI. This shift creates two critical challenges within the industry:
The retention problem: How do you maintain value when creation costs nothing?
The customer acquisition problem: How do you stand out when everyone has the same tools?
Companies are responding by shifting value from production capability to creative vision, distribution, and workflow integration. As Casado noted, we're entering an era where competitive advantage comes not from the ability to create, but from knowing what to create and how to reach the right audience.
In Conclusion..
It’s clear that the conversations and innovations from the Cerebral Valley AI Summit will continue to shape the future of AI, and we’re grateful to have been part of it. A couple concluding worthwhile mentions from the summit include:
We heard from Writer’s CTO, Waseem Alshikh, that they are announcing Writer self-evolving models, where models can continuously learn and adapt in real time without full retraining cycles. These models utilize a memory pool to store and recall new information, assign uncertainty scores to unfamiliar inputs to identify knowledge gaps, and integrate new insights into existing knowledge bases.
It's very clear that many of the best AI researchers are now focused on robotics. Many top AI researchers, like Chelsea Finn (CEO of Physical Intelligence, who demoed at Cerebral Valley), are focusing on robotics, but they face a severe shortage of data. Robotics data is far scarcer and more expensive than data for other AI modalities. One estimate suggests we need 40,000 robot-years of data to train a typical-scale transformer and test if scaling laws apply to robotics as hoped. Tessa Lau of Dusty Robotics noted that, to build revenue and customers in the meantime, robotics companies will likely focus on specialized robots optimized for specific tasks before achieving general-purpose robotics.
Huge thanks to Eric and Volley team for hosting an inspiring event!