Posts

Tilting at Red Windmills

  Modern Communists and Their Quixotic Quest Few literary characters are as emblematic of naive preposterousness as Miguel de Cervantes’ Don Quixote. Caught up in the romantic ideal of chivalry, this doomed but noble knight-errant actually saw windmills swaying in the distance and charged at them, thinking them to be giants. Recent history has given us new and surprising candidates for the spirit of Quixote: namely, the modern communists. Like Quixote, modern communists believe that not only is a perfect, classless society possible, it exists in a form that one can take wings to reach. In this respect, modern communists are a bit like our original knight: they’re chasing a quixotic dream (for the sake of humans) that regularly runs aground on the shoals of reality. Their windmills are the archons of global capitalism; their Dulcinea, a workers’ paradise that can never quite be captured. But, like our original knight, they’re wreaking havoc in the meantime. Behold, then, the Soviet ...

Evaluating LLM and RAG Systems

Image
  Focusing on key Metrics There is no single metric that can fully capture the functionality of Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) systems. In this document, we discuss the metrics that are particularly important when evaluating LLM and RAG systems, both the atomic components and the holistic system. Key Metrics for RAG Evaluation: 1. Faithfulness Definition: Measures the factual consistency of the generated answer against the given context. Break the generated answer into individual statements. Verify if each statement can be inferred from the given context. Scaled to a range of 0 to 1, with higher scores indicating better faithfulness. 2. Answer Relevance Definition: Assesses how pertinent or relevant the generated answer is to the given prompt. Generate multiple variants of the question from the generated answer using an LLM. Measure the mean cosine similarity between these generated questions and the original question. Higher scores indicate b...