ClauseLens
AI-powered Terms of Service analysis with hybrid retrieval, cited answers, and evidence-based abstention.
At a glance
Mean Q&A faithfulness across 15 evaluation questions
BM25 + dense retrieval + cross-encoder reranking
Fine-tuned, quantized Qwen 2.5 7B inference
See it in action
What needed solving?
Terms of Service agreements are lengthy and difficult to interpret. Users often accept them without understanding important conditions. General-purpose LLMs can also generate unsupported answers without showing the clauses on which their responses are based.
What I built
Built an end-to-end Terms of Service analysis application, including PDF ingestion, hybrid retrieval, evidence-grounded question answering, quantized model inference, frontend and backend integration, cloud deployment, and evaluation.
Overview
ClauseLens helps users understand complex Terms of Service agreements through document analysis and clause-grounded question answering. It retrieves relevant passages, generates answers with inspectable source references, and abstains when sufficient supporting evidence is unavailable. The application uses a fine-tuned, quantized 7B language model with CPU-only inference support.
System architecture
Explore the workflow
Extract text from uploaded PDFs, create structure-aware chunks, and preserve page and section metadata.
Decisions, validation & lessons
Engineering decisions & tradeoffs+
Hybrid retrieval with reranking
Combined dense semantic retrieval with BM25 using reciprocal rank fusion, followed by cross-encoder reranking, to capture both conceptual similarity and exact terminology in legal documents.
Tradeoff: Additional retrieval stages increase complexity and computation. Chunk boundaries can also separate related clauses and exceptions.
Evidence extraction before generation
Separated evidence extraction from answer synthesis to provide inspectable source references and support abstention when sufficient evidence is unavailable.
Tradeoff: Retrieval and evidence extraction errors can still produce incomplete or incorrect answers. Citations alone do not guarantee factual correctness.
Quantized 7B model for CPU inference
Used a legal-task fine-tuned Qwen 2.5 7B model in GGUF format with llama.cpp to support inference without requiring a dedicated GPU.
Tradeoff: CPU inference introduces latency, while quantization and the model's limited context window can affect answer quality.
Validation & results+
Evaluated 15 questions across 5 PDFs from 4 services. Recorded context recall was approximately 0.93, mean Q&A faithfulness 0.88, and answer relevance 0.60. A separate local benchmark recorded 17.2-second mean Q&A latency. These are small offline evaluations, not production accuracy guarantees.
Limitations & learnings+
The model can miss clauses, misinterpret exceptions, or generate unsupported claims despite citations. CPU inference and cloud cold starts introduce latency. Citation accuracy and correct-abstention rates have not been independently measured, and experimental risk labels are not legal judgments.
Explored hybrid retrieval, evidence-grounded generation, fine-tuning, and 4-bit quantization. Evaluated retrieval and answer quality while navigating CPU inference latency, cloud cold starts, and the limitations of citation-based grounding.