Building a Biotech Investor (Companion) for Public Stocks

A couple of months ago, my carefully constructed RAG pipeline confidently hallucinated a source that looked correct but was not. The retrieved chunks were relevant, and the embedding similarity scores were high but the output was wrong in a way that would make this ineffective for my customers.

I am an MD and a former biotech investor, who has spent over 6 months trying to build an AI companion that would help me in my personal biotech investing. I love stock picking and today some of my best recent ideas, $CDTX and $NUVL, came from AI-assisted diligence. I thus wanted to build something that could help me reason across the full messy surface area of biotech investing: PDFs of 10-Ks, clinical posters, ClinicalTrials.gov, and graphs and tabular data.

What followed was a tour through nearly every architectural pattern in the modern AI stack. I want to share what I learned, not because I believe that I built the definitive system, but because I think the journey maps onto a larger shift in how we should be thinking about AI tooling in biotech, I arrived through this through much trial and error and an embarrassing number of wasted API calls.

Act I: The RAG Honeymoon

Like everyone, I started with vanilla RAG: chunk your documents, embed them into vectors and store them in your vector database. At query time, the system would retrieve the most semantically similar chunks and feed them to an LLM for generation. LlamaIndex made this trivially easy to prototype.

This worked at first. Simple factual queries returned reasonable answers and therapeutic maps. I could ask for a list of all the Chinese obesity assets or what was Vertex Pharmaceuticals’ R&D spend in Q3, and get a number with a citation.

Then I started asking harder questions: ones that actually are important in biotech investing. Compare this company’s PhII trial to the historical benchmark for this tumor type. This is a multi-hop reasoning problem. The answer lives across several documents: clinical posters, publications, and a reference dataset. It also requires absorption and processing of graphical figures. Vector similarity retrieval doesn't understand that these documents are connected. It retrieves chunks that look relevant based on semantic distance. It has zero understanding of how those chunks relate to each other.

The failures stacked up fast. Cosine similarity found chunks that mentioned the right drug names but from the wrong trial. Context windows filled with five semi-relevant passages when what I needed was one passage from document A and one from document C, stitched together with domain knowledge. The worst part was how confident the LLM was, even when it was wrong or working from stale data.

I was hitting the limitations of flat retrieval meaning the architecture works when a query maps cleanly to a single document and the answer doesn't require reasoning across multiple sources. For anything more complex, you're hoping the model can stitch together disconnected chunks correctly.

The standard playbook for improving retrieval is query decomposition, rerankers hybrid BM25+dense search etc. and these help with conventional information retrieval problems but the core challenge in biotech investing is that the queries are inherently multimodal and evidence weighted. You are looking for the most current clinical evidence across multiple document types, weighted by the hierarchy of evidence. Often the answer is in a figure, not in text. No amount of reranking fixes a retrieval architecture that doesn't understand this.

Act II: The Knowledge Graph Detour

The logical next step was knowledge graphs. If the problem is that vector search doesn’t understand relationships, then model the relationships explicitly. Build a graph where entities (drugs, targets, trials etc) are nodes and their relationships are edges.

Microsoft Research had released their GraphRAG framework. Neo4j had published its GraphRAG Manifesto. The academic literature was encouraging showing both a reduction in hallucinations and a decrease in token usage on financial benchmarks. I built the graph and it was beautiful, but it was also a nightmare to maintain: ontology work alone took weeks. Every time a new asset enters your coverage universe, you’re back to entity extraction and relationship mapping.

Act III: The Filesystem Revelation

I started to realize that my problem was fundamentally a context curation problem, not a retrieval problem. I needed a system that could assemble the right context from the right documents and present them to the model in a readable order.

I built a stigmergic workspace (borrowing from the concept in biology where organisms coordinate through modifications they leave in a shared environment). Each coverage asset gets a structured folder. Inside that folder: the latest 10-K summary (markdown), the most recent clinical data, the competitive landscape and a metadata file that tells the orchestrating agent what’s in each subfolder and when it was last updated.

The retrieval mechanism? grep. And find. And bash scripts that know where to look based on the query structure. This works because the folder structure serves as knowledge graph. And because everything is in plain text and markdown, it’s human-readable, and easily debuggable.

What This Really is: A Deterministic Hierarchical Router

When I replaced a vector database with a folder structure, I built a deterministic hierarchical router, and I am the routing function. he ontology of the folders reflects my mental model of how biotech assets relate to each other. It works because my coverage universe is bounded and because I have enough domain expertise to design a folder structure that maps onto how I actually ask questions.

This is simultaneously the system's greatest strength and its most important limitation. It's a strength because the routing is predictable, auditable, and never hallucinates. It's a limitation because it encodes my understanding at the time I designed it, and because the retrieval mechanism, grep, is syntactically rigid in ways that create real blind spots.

Here's the problem: biotech is riddled with semantic synonyms that grep treats as completely unrelated strings e.g loss of response, progressive disease, PD, treatment failure, lack of clinical benefit, these can all mean functionally the same thing depending on context. ORR, confirmed response rate by RECIST 1.1, and cRR by independent review are related concepts with important distinctions that grep cannot reason about. I can hardcode a synonym dictionary, but that dictionary is brittle, requires constant maintenance.

What this system actually needs is metadata-gated hybrid search: the determinism of the folder structure for routing, layered with a semantic search component that understands biomedical terminology. The folder structure tells the system where to look; the semantic layer understands what to look for once it gets there. The grep-and-folders approach trades recall for precision, and that's an acceptable trade-off only when your coverage universe is small enough to audit manually.

Additionally a folder structured can make single asset deep dives excellent but cross asset comparison structurally difficult. This is where the knowledge graph becomes loading-bearing again. You need some kind of normalized comparison layer: either a graph that encodes relationships across assets, or a periodically regenerated comparison table that tags each asset with tumor type, line of therapy, mechanism of action, and key endpoints so the routing layer can pull the right folders in parallel.

Act IV: The Multi-Model Stack

Now I needed to focus on the quality of what was going into each paper. A multi-model approach became essential. Routing every job through one frontier model was not taking advantage of the dozens of production ready models optimized for radically different workloads. My stack evolved to use different models for different jobs:

1. Gemini for PDF extraction. Google’s Gemini models have become the best tool I’ve found for turning complex biotech PDFs into structured data. Gemini’s multimodal understanding handles the visual layout reasoning that pure OCR pipelines miss. For born-digital PDFs like most sec filings, the extraction is nearly flawless. For scanned clinical posters, it’s good enough to get structured data that I can then verify.
2. Claude for synthesis and reasoning. Once the data is extracted and sitting in the right folder, Claude is my generation model. For the kind of multi-step, nuanced reasoning that biotech investing demands, Claude’s extended thinking and careful hedging aligned well. That said it still struggles with balancing (clinical utility and safety weighting, oppose to producing a generalized summary).
3. Specialized embedding models for the search index. For the parts of my system that do use semantic search (a hybrid search layer that sits alongside the folder structure for broad corpus queries), I’ve moved to domain-tuned embedding models. I used a strong hosted model and added a reranker. Fine-tuning on domain-specific data could close the remaining gap.

One tool I have been exploring is TabPFN, a prior-data fitted network that is purpose-built for small to medium tabular datasets. This made me think it would be perfect for biotech: trials with limited patient numbers. TabPFN approximates Bayesian inference through in-context learning on a transformer trained across millions of synthetic datasets, which means it avoids the overfitting traps that plague conventional models when you have 50 or 200 rows of clinical data.

But there's a fundamental assumption baked into TabPFN that doesn't hold in biotech: it assumes features are independent and identically distributed (i.i.d). Trial data across different companies is emphatically not i.i.d. Two trials will have different enrollment criteria, different endpoint definitions, different censoring patterns, different data maturity, and different standards of data reporting. The distributional shift across sponsors, geographies, and trial designs is massive.

This doesn't make TabPFN useless, but it means you can't feed it raw extracted trial features and expect meaningful output. You need rigorous feature standardization first ideally anchoring to a reference ontology like CDISC, and at minimum normalizing by line of therapy, tumor type, endpoint definition, and data cutoff maturity. That standardization step is itself a nontrivial engineering problem. For tasks like predicting probability of clinical success from extracted trial features, or clustering assets by efficacy profile, TabPFN could fill a real gap, but only after you've done the hard work of making the inputs actually comparable.

Act V: Agent Orchestration with the AI SDK

The glue holding this together is Vercel’s AI SDK. I needed something that could orchestrate calls across multiple model providers with a unified API, handle tool calling, and manage the agentic loop.

The AI SDK solved several problems simultaneously. It’s model-agnostic by design, meaning I can swap providers without rewriting integration code. It handles streaming, which matters when you’re waiting for a complex synthesis across multiple documents. The AI SDK’s stopWhen and prepareStep abstractions made this loop controllable and debuggable, which is critical when you’re dealing with investment decisions and need to understand exactly why the system produced a particular output.

Act VI: Evals (And Why Biotech Breaks Every Framework)

None of this matters if you can’t measure whether it’s working. This was the hardest lesson and the one I resisted longest.

The RAG evaluation landscape has matured significantly. The canonical open-source framework is RAGAS, which pioneered four core metrics: faithfulness, answer relevance, context precision, and context recall.

But here’s what the eval frameworks can’t catch, and what I learned the hard way: a system can score 0.95 on faithfulness and still produce wrong answers if the retrieved content itself is stale for example. Faithfulness measures whether the model is loyal to its sources. It doesn’t measure whether the sources are right.

My eval approach ended up being three-layered. First,I used automated metrics via RAGAS on a curated test set of questions with known answers. Secondly, I ran a freshness check that flags when source documents haven’t been updated past a threshold. Thirdly,, periodic manual review where I compare the system’s output against my own analysis for a handful of recent investment decisions.

But even this three-layer approach misses failure modes that are specific to biotech. in biotech, evidence has a temporal hierarchy e.g. a longer follow-up dataset supersedes the earlier abstract from the same trial. My system needs bitemporal awareness: tracking both when the data was generated and when it was ingested, using those timestamps to ensure the most authoritative source wins

Beyond freshness, you need a structured belief model underneath the retrieval layer. Every investment thesis decomposes into discrete claims, about safety profile, SOC comparison, regulatory precedent, market expectations, etc and each claim should be backed by specific evidence with provenance and a confidence level.

The Obituary That Wasn’t: Why RAG Isn’t Dead

Somewhere during this journey, I started seeing the think-pieces. “Long context kills RAG.” I understand the appeal of the argument. Context windows have gone from 4K tokens to over 1M in under two years. If you can just shove every document into the prompt, why bother with retrieval at all?

I tried this. It doesn’t work for biotech investing. And I would argue it does not work for most domains where the data is dense, dynamic, and the stakes are real.

The most important reason is attention dilution, I found that when you stuff hundreds of pages into the context window, the model’s attention over passages in the middle of the context degrades meaningfully.

Beyond attention, long-context inference is slow in ways that create genuine UX friction. It becomes expensive quickly and the biotech data surface is constantly expanding, and you will never fit your domain's knowledge into a single prompt. The future is agentic: in agentic workflows, you cannot preload every possible data source the agent might need. Retrieval becomes essential.

The Meta-Lesson: Context Engineering

Which brings us to the reframe that ties all of this together. Context engineering has proven to be far more helpful than prompt engineering. The bottleneck is not the model but what you feed the model. The quality, structure, and curation of what enters the context window determines the quality of what comes out.

I needed to optimize for selecting, curating, and assembling the right information for the task at hand. The fact that I eventually replaced a vector database with a folder structure and grep doesn’t mean I abandoned retrieval. It means I found a retrieval mechanism that gave me better precision over a curated corpus. My argument here is that everyone has access to the same frontier models but the advantage is how you organize your knowledge, how you maintain freshness across a coverage universe, and how you assemble the right subset of that knowledge for each specific question.

What I’d Tell My Past Self

If I were starting over, knowing what I know now, I would start with the folder structure. I would build the multi-model pipeline from day one instead of trying to force one model to do everything. And I’d set up evals before I built the first feature.

The irony is that the most sophisticated AI system I could build turned out to look, from the outside, a lot like a well-organized research directory with some very smart scripts, a thin knowledge graph, and a clinical preprocessing layer on top. The sophistication isn't in the architecture. It's in knowing what to put where, and why and being honest about what the system still can not do.

That’s context engineering. And for anyone building AI tools in a domain as complex and consequential as biotech investing, it’s the most important thing to get right.