Step 1 of 4
The Problem
Industry
Legal Technology (LegalTech) — Document Drafting and Review Automation
The legal industry processes an enormous volume of long-form, high-stakes text. Contracts, regulatory filings, litigation briefs, due diligence reports — these documents routinely run to tens of thousands of tokens. Unlike consumer chatbots where a user asks a short question and receives a short answer, legal AI systems operate in a regime where context is everything: a single contract clause only makes sense in light of every clause that came before it. This makes legal AI one of the most demanding deployment environments for large language model inference, and one of the most commercially important. The global legal technology market is projected to exceed $50 billion by 2027, with AI-assisted drafting representing the fastest-growing segment.
Company Profile
Lex Machina Legal Systems (not to be confused with the litigation analytics company of the same name — this is a fictional entity for this case study)
- Founded: 2021, San Francisco, CA
- Team Size: 68 employees — 19 ML engineers, 12 backend engineers, 8 product managers, 11 attorneys serving as domain advisors, and the rest across sales, ops, and design
- Funding Stage: Series B, $47M raised, led by a tier-1 enterprise SaaS fund
- Key Products:
- LexDraft: An AI drafting assistant that completes, continues, and revises contract sections in real time as attorneys type
- LexReview: A batch analysis pipeline that flags risk clauses, inconsistencies, and missing provisions across uploaded contract sets
- LexCompare: A retrieval-augmented generation tool that finds precedent language from an internal clause library and proposes alternatives
The company's flagship product, LexDraft, is deployed as a browser plugin integrated into document editors used by 340 enterprise law firms and corporate legal departments. It serves approximately 4,200 active daily users, predominantly senior associates and partners billing at rates between $400 and $1,200 per hour.
Business Challenge
LexDraft's core loop is generative: a user pauses mid-sentence, and the model proposes a continuation, clause by clause, in real time. The user experience is expected to mirror the feel of a fast autocomplete — the model must respond within roughly 300 milliseconds for the first token and sustain a generation cadence that keeps up with reading speed for subsequent tokens. The underlying model is a fine-tuned 7-billion parameter transformer that the team has hosted on a cluster of NVIDIA A100-80GB GPUs.
Recently, as the company signed several large enterprise accounts — including one Fortune 50 financial services company with a legal team generating documents with average context lengths of 6,000–12,000 tokens — the inference system has begun to crack.
Specifically, the team is observing the following:
-
First-token latency (TTFT) is spiking on long documents. At 2,000-token contexts, TTFT sits around 210 ms, which is acceptable. At 8,000-token contexts, TTFT is regularly exceeding 1.4 seconds, causing the UI to show a loading spinner — a UX death sentence for a tool targeting time-pressured attorneys.
-
Memory exhaustion at high concurrency. During peak hours (9:00–11:00 AM EST, when east coast legal teams start their workday), the serving cluster is crashing with out-of-memory (OOM) errors. Analysis shows that the KV cache for simultaneous long-context sessions is consuming the bulk of the 80 GB VRAM budget, leaving too little room for model weights and activations.
-
GPU utilization is paradoxically low during the crashes. The monitoring dashboard shows GPU compute utilization hovering around 22–35% at the exact moments when latency is highest and OOM errors are occurring. This counterintuitive observation — the GPU is idle yet also overwhelmed — is the central mystery the ML team needs to solve.
The engineering team initially proposed simply adding more GPUs, but the CFO flagged that the current GPU cluster already costs approximately $38,000 per month in cloud compute. Doubling the cluster to solve a problem caused by inefficient memory usage is not an acceptable answer without a rigorous technical justification and an exploration of software-level fixes first.
Why It Matters
The stakes are both direct and compounding:
- Revenue at risk: The Fortune 50 financial services client alone represents $2.1M ARR. Their IT leadership has sent a formal notice that sustained latency issues will trigger a contract review clause. Three other large accounts, totaling $1.8M ARR, are watching closely.
- User trust erosion: Attorneys are a notoriously exacting user base. A single bad experience — a spinning cursor at a critical drafting moment — is enough to break the habit loop that makes the tool sticky. Retention data shows that users who experience TTFT above 600 ms on three consecutive sessions have a 40% higher 30-day churn rate.
- Competitive exposure: Two well-funded competitors (Harvey AI and CoCounsel) are actively pitching the same enterprise accounts. Speed is a top-three evaluation criterion in every sales cycle the company has documented.
- Engineering credibility: The team needs to demonstrate internally that they understand why the system is behaving this way before they can propose a defensible fix. An expensive but misdiagnosed solution (throwing more GPUs at a memory bandwidth problem) would be far worse than the current situation.
Constraints
| Constraint | Detail |
|---|---|
| Latency | TTFT ≤ 300 ms at p95 across all context lengths up to 8,000 tokens; sustained generation ≥ 15 tokens/sec |
| Compute budget | Monthly GPU spend must not exceed $45,000 — a soft cap set by the CFO pending Series C fundraising |
| Concurrency | Must support at least 80 simultaneous long-context sessions per A100 node |
| Data privacy | Client contracts are strictly confidential. No actual client data may leave the company's VPC. All profiling must use synthetic or publicly licensed data. |
| Compliance | SOC 2 Type II and GDPR Article 28 compliance — inference infrastructure must maintain audit logs; no third-party API calls with client data |
| Model constraints | The core 7B fine-tuned model is frozen (the fine-tuning budget for this quarter is spent). Architectural changes to the model itself are out of scope. Serving-layer optimizations only. |
| Deployment environment | Self-hosted on AWS p4d.24xlarge instances (8 × A100 40GB each) behind an internal API gateway; no model-as-a-service providers |