AI & ML
AI/ML development and deployment best practices
Best Practices
Feature Store Best Practices
A centralized system for defining, storing, and serving machine learning features consistently for training and inference, avoiding skew and duplicated work.
by Linux Foundation (Feast)MLOps Principles
The discipline of applying DevOps and engineering rigor to machine learning so models are built, deployed, monitored, and retrained reliably and reproducibly.
by GoogleML Model Monitoring and Drift Detection
Continuously tracking deployed ML models for performance decay, data drift, and concept drift so degradation is caught and corrected before it harms outcomes.
by Evidently AIData Version Control (DVC)
Versioning datasets, models, and ML pipelines alongside code so experiments are reproducible, using Git for metadata and external storage for large files.
by Iterative (DVC)Retrieval-Augmented Generation (RAG) Best Practices
RAG grounds a large language model in external documents retrieved at query time, reducing hallucination and letting answers reflect current, private data without retraining the model.
by Meta AI (RAG paper authors)Prompt Engineering Best Practices
Prompt engineering is the practice of designing clear instructions, examples, and structure so a large language model returns accurate, consistent, and useful output.
by OpenAILLM Evaluation and Evals
LLM evaluation measures model and application quality with repeatable tests, scoring accuracy, faithfulness, safety, and cost so teams can ship and improve with evidence.
by OpenAILLM Guardrails
LLM guardrails are programmatic checks on model inputs and outputs that enforce safety, format, topic, and policy rules, blocking or correcting unsafe or off-policy responses.
by Guardrails AIModel Context Protocol (MCP)
The Model Context Protocol is an open standard that lets AI applications connect to external tools and data sources through a uniform client-server interface.
by AnthropicAI Agent Design Patterns
AI agent design patterns are reusable structures for LLM systems that plan, use tools, and act over multiple steps, covering reflection, tool use, planning, and multi-agent collaboration.
by AnthropicLLM Observability
LLM observability is the practice of tracing, logging, and measuring LLM applications in production to monitor quality, cost, latency, and safety and to debug failures.
by Cloud Native Computing Foundation (OpenTelemetry)Vector Database Best Practices
A vector database stores embeddings and serves fast similarity search for AI features like RAG and semantic search; best practices cover indexing, metadata, and freshness.
by PineconeLLM Cost Optimization
LLM cost optimization reduces the spend of running language model applications through model selection, caching, prompt efficiency, and token-aware design without sacrificing quality.
by OpenAIFine-Tuning vs RAG Decision Framework
A decision framework for choosing between fine-tuning, RAG, or both, based on whether the goal is new knowledge, consistent behavior, freshness, or domain adaptation.
by OpenAIHallucination Mitigation
Hallucination mitigation reduces confident but false LLM output through grounding, retrieval, citation, verification, and uncertainty handling so answers can be trusted.
by AnthropicTutorials
How to build a RAG pipeline for question answering
Build a retrieval-augmented generation pipeline that grounds an LLM's answers in your own documents using chunking, embeddings, and a vector store.
How to add semantic search with embeddings and a vector database
Add meaning-based search to an app by generating text embeddings and querying a vector database for nearest neighbors.
How to build an LLM app with function calling (tools)
Give an LLM the ability to call your functions, so it can fetch data and take actions instead of only producing text.
How to fine-tune a language model on your own data
Adapt a base language model to your domain with supervised fine-tuning, covering data prep, training, and evaluation.
How to evaluate LLM outputs systematically
Build a repeatable evaluation suite for LLM features using reference checks, rubrics, and model-graded scoring.
How to engineer effective prompts for LLMs
Apply practical prompt-engineering techniques such as clear instructions, examples, and structured output to get reliable LLM results.
How to build an AI agent that plans and acts
Build an autonomous LLM agent that uses tools in a perceive-plan-act loop to accomplish multi-step tasks.
How to deploy a machine learning model for inference
Serve a trained model behind an HTTP API, containerize it, and scale it for production inference.
How to add guardrails to an LLM application
Protect an LLM app with input and output guardrails that filter unsafe content, block prompt injection, and validate structure.
How to build a Model Context Protocol (MCP) server
Build an MCP server that exposes tools and resources to AI assistants over a standard protocol.
Checklists
LLM/RAG Production-Readiness Checklist
Verification items for taking a retrieval-augmented generation (RAG) application from prototype to reliable production service.
ML Model Deployment Checklist
Pre-flight verification for promoting a trained machine learning model into a production serving environment.
MLOps Pipeline Review Checklist
Audit items for assessing the maturity, reproducibility, and automation of an end-to-end machine learning operations pipeline.
LLM Evaluation Readiness Checklist
Verification items for building a trustworthy evaluation harness before releasing changes to an LLM-powered feature.
AI Agent Deployment Checklist
Pre-flight items for safely deploying an autonomous LLM agent that calls tools and takes actions on behalf of users.
LLM Cost Optimization Review Checklist
Review items for reducing the cost of an LLM application without degrading quality, covering prompts, caching, and model choice.
Technology Stacks
MLOps Stack
MLflow, Kubeflow, TensorFlow, Feature Store - ML lifecycle
MLflow MLOps Stack
End-to-end MLOps pattern using MLflow for experiment tracking, model registry, packaging, and deployment, integrated with feature, data, and serving layers.
RAG Stack (LangChain + pgvector + LLM)
Retrieval-augmented generation stack: LangChain orchestrates retrieval over pgvector embeddings in Postgres and grounds an LLM's answers in your own data.
Kubeflow ML Platform
Kubernetes-native ML platform: Kubeflow Pipelines, training operators, KServe serving, and Katib tuning run the ML lifecycle on Kubernetes.
Feast Feature Store Stack
Feature store pattern using Feast to define, materialize, and serve consistent ML features from an offline warehouse and a low-latency online store.
LLM Agent Stack
Agentic AI stack: an orchestration framework drives an LLM to reason, call tools and APIs, and use vector memory to complete multi-step tasks autonomously.
Ray Distributed ML
A unified compute stack using Ray to scale Python machine learning workloads from data processing through training to serving.
vLLM + Ray Serve
A high-throughput LLM serving stack combining vLLM's optimized inference engine with Ray Serve for scalable, multi-replica deployment.
LangGraph + pgvector
An agentic LLM application stack using LangGraph for stateful agent workflows backed by Postgres with pgvector for retrieval and memory.
Haystack RAG
A production RAG stack built on the Haystack framework for composable retrieval-augmented generation pipelines over a document store.
LlamaIndex + Qdrant
A RAG and data-framework stack pairing LlamaIndex for LLM data ingestion and querying with Qdrant as a high-performance vector database.
Triton Inference Server
A high-performance model-serving stack using NVIDIA Triton to serve models from any framework with GPU optimization on Kubernetes.
Tecton Feature Store
A production feature platform stack using Tecton to define, compute, and serve consistent ML features for training and real-time inference.
MosaicML Composer Training
A large-scale model training stack using Composer and the MosaicML toolkit to train and fine-tune models efficiently on GPU clusters.
FAQs
What is a large language model (LLM)?
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence, which lets it generate and understand natural language. Modern LLMs use the transformer architecture and contain billions of parameters learned during training. They can perform tasks like summarization, translation, code generation, and question answering without task-specific training, often guided only by a prompt.
What is retrieval-augmented generation (RAG)?
Retrieval-augmented generation is a technique that supplements a language model with relevant documents fetched at query time, rather than relying only on knowledge baked into the model's weights. A retriever searches a knowledge source (often a vector database) for passages related to the user's question, and those passages are inserted into the prompt as context. RAG reduces hallucination, lets models answer about private or recent data, and avoids retraining when the underlying information changes.
What are embeddings in machine learning?
Embeddings are dense numerical vectors that represent text, images, or other data in a continuous space where semantic similarity corresponds to geometric closeness. A model maps each input to a fixed-length vector so that related items sit near each other, enabling similarity search, clustering, and classification. Embeddings power semantic search and retrieval-augmented generation, where queries and documents are compared by the distance between their vectors.
What is the difference between fine-tuning and RAG?
Fine-tuning updates a model's weights by training it further on domain-specific examples, changing how the model behaves and what style or skills it has. RAG leaves the model unchanged and instead supplies relevant information at inference time through the prompt. Use RAG when knowledge changes often or must stay external and auditable; use fine-tuning to teach consistent formats, tone, or specialized tasks. The two are complementary and are often combined.
What is fine-tuning a model?
Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset so it adapts to a particular domain, style, or behavior. It adjusts the model's weights, which makes the changes persistent and removes the need to include lengthy instructions in every prompt. Parameter-efficient methods like LoRA fine-tune only a small subset of weights, cutting compute and storage costs while preserving most of the base model.
What is a token in the context of LLMs?
A token is the basic unit of text that a language model reads and produces, typically a word fragment, whole word, or punctuation mark rather than a single character. A tokenizer splits input text into tokens and maps them to integer IDs the model can process. As a rough guide, one token is about four characters or three-quarters of a word in English, and model limits and pricing are usually measured in tokens.
What is a context window?
A context window is the maximum number of tokens a language model can consider at once, covering both the input prompt and the generated output. If a conversation or document exceeds this limit, earlier content must be truncated, summarized, or retrieved selectively. Larger context windows allow more documents and history to be included, but they increase cost and latency and do not guarantee the model attends equally to everything in the window.
What is prompt engineering?
Prompt engineering is the practice of designing the instructions, examples, and context given to a language model to get reliable, accurate outputs. Techniques include giving clear roles and constraints, providing examples (few-shot prompting), and asking the model to reason step by step (chain-of-thought). Good prompts reduce ambiguity and hallucination, and they are often the cheapest way to improve results before resorting to fine-tuning.
What is an AI agent?
An AI agent is a system that uses a language model to decide and take actions toward a goal, rather than producing a single response. It typically operates in a loop: the model reasons about the task, calls tools or APIs, observes the results, and repeats until the goal is met. Agents can search the web, run code, query databases, or control other software, which makes guardrails, permissions, and observability essential.
What is hallucination in LLMs?
Hallucination is when a language model produces text that sounds confident and plausible but is factually wrong or unsupported by its sources. It happens because the model generates statistically likely text rather than retrieving verified facts, so gaps in knowledge are filled with fabrication. Mitigations include grounding answers with retrieval (RAG), asking for citations, lowering temperature, and validating outputs against trusted data before acting on them.
What is temperature in LLM generation?
Temperature is a parameter that controls the randomness of a language model's output by scaling the probability distribution over the next token. A low temperature near zero makes the model pick the most likely tokens, producing focused and deterministic responses, while a higher value increases diversity and creativity at the cost of consistency. Use low temperature for factual or structured tasks and higher temperature for brainstorming or varied creative writing.
What is inference in machine learning?
Inference is the phase where a trained model is used to make predictions or generate output on new inputs, as opposed to training where the model learns from data. For LLMs, inference means running a forward pass to produce tokens, and it is where latency, throughput, and serving cost matter most in production. Techniques such as batching, caching, and quantization are used to make inference faster and cheaper at scale.
What is the difference between supervised and unsupervised learning?
Supervised learning trains a model on labeled examples, where each input has a known target, so the model learns to predict labels for new data in tasks like classification and regression. Unsupervised learning works with unlabeled data and finds structure on its own, such as grouping similar items through clustering or reducing dimensionality. Supervised learning needs costly labeled datasets but gives precise targets, while unsupervised learning explores patterns without labels.
What is overfitting in machine learning?
Overfitting happens when a model learns the training data too closely, including its noise and quirks, so it performs well on that data but poorly on new, unseen inputs. It usually signals that the model is too complex relative to the amount of data or that training ran too long. Common remedies include more or more varied training data, regularization, dropout, cross-validation, and early stopping.
What is a transformer architecture?
The transformer is a neural network architecture, introduced in 2017, that processes sequences using a self-attention mechanism instead of recurrence. Self-attention lets each token weigh the relevance of every other token in the input, capturing long-range relationships in parallel rather than step by step. Transformers scale efficiently on modern hardware and underpin nearly all current large language models and many vision and multimodal models.
What is quantization in machine learning?
Quantization reduces the numerical precision of a model's weights and activations, for example from 32-bit floating point to 8-bit or 4-bit integers, to shrink memory use and speed up inference. It lets large models run on smaller or cheaper hardware with only modest accuracy loss when done carefully. Post-training quantization applies after training, while quantization-aware training accounts for the lower precision during training to preserve accuracy.
What is the Model Context Protocol (MCP)?
The Model Context Protocol is an open standard that defines how AI applications connect to external tools, data sources, and services in a consistent way. An MCP server exposes resources, tools, and prompts that an MCP-aware client and its language model can discover and use over a standard interface. By standardizing these integrations, MCP lets the same connectors work across different AI clients instead of building one-off integrations for each.
What is chain-of-thought prompting?
Chain-of-thought prompting asks a language model to work through a problem step by step before giving a final answer, rather than responding immediately. Exposing intermediate reasoning often improves accuracy on math, logic, and multi-step tasks because the model allocates more computation to the problem. The trade-off is longer, more expensive outputs, and the visible reasoning is a plausible explanation rather than a guaranteed account of the model's internal process.
Benchmarks
Code Translation Accuracy
Measures how accurately AI models translate code between programming languages
MMLU (Massive Multitask Language Understanding)
A 57-subject multiple-choice benchmark testing broad academic and professional knowledge across STEM, humanities, social sciences, and law.
MMLU-Pro
A harder, reasoning-focused successor to MMLU with ten answer options and tougher questions designed to separate frontier models that saturated the original.
GSM8K (Grade School Math 8K)
A benchmark of ~8,500 grade-school math word problems that test multi-step arithmetic reasoning with a single numeric answer.
MATH (Competition Mathematics)
A benchmark of 12,500 competition-style math problems across algebra, geometry, number theory, and calculus, graded on exact final-answer match.
HumanEval
A code-generation benchmark of 164 Python programming problems graded by executing unit tests, popularizing the pass@k metric.
MBPP (Mostly Basic Python Problems)
A benchmark of ~1,000 entry-level Python programming tasks with test cases, used to evaluate basic code synthesis from short descriptions.
SWE-bench
A benchmark of real GitHub issues from open-source Python repositories where a model must produce a patch that resolves the issue and passes tests.
SWE-bench Verified
A 500-task, human-validated subset of SWE-bench with clear specifications and reliable tests, used as the standard clean measure of agentic coding.
BIG-bench (Beyond the Imitation Game)
A collaborative suite of 200+ diverse tasks probing reasoning, knowledge, and emergent abilities beyond conventional language benchmarks.
BBH (BIG-bench Hard)
A 23-task subset of BIG-bench focused on challenging multi-step reasoning where chain-of-thought prompting yields large gains.
HellaSwag
A commonsense sentence-completion benchmark where models pick the most plausible continuation among adversarially generated distractors.
ARC (AI2 Reasoning Challenge)
A grade-school science question benchmark split into Easy and Challenge sets, the latter built from questions retrieval methods answer incorrectly.
TruthfulQA
A benchmark measuring whether models avoid generating false answers that mimic common human misconceptions and falsehoods.
GPQA (Graduate-Level Google-Proof Q&A)
A benchmark of expert-written, graduate-level science questions designed to be extremely hard even with web access, testing deep domain reasoning.
MMMU (Massive Multi-discipline Multimodal Understanding)
A multimodal benchmark of college-level questions requiring joint reasoning over text and images such as diagrams, charts, and figures.
MT-Bench
A multi-turn conversational benchmark where a strong LLM judge scores model responses across categories on a 1-10 quality scale.
Chatbot Arena
A live, crowdsourced evaluation where users compare two anonymous model responses and votes are aggregated into Elo-style rankings.
HELM (Holistic Evaluation of Language Models)
A standardized framework evaluating language models across many scenarios and multiple metrics including accuracy, robustness, fairness, and efficiency.
DROP (Discrete Reasoning Over Paragraphs)
A reading-comprehension benchmark requiring discrete operations like addition, counting, sorting, and comparison over passage content.
WinoGrande
A large-scale commonsense benchmark of pronoun-resolution sentence pairs designed to require world knowledge rather than lexical cues.
AGIEval
A benchmark built from human standardized exams such as college entrance, law, and civil-service tests to measure human-centric reasoning.
LiveCodeBench
A contamination-resistant coding benchmark that continuously collects new competitive-programming problems and evaluates by execution over time.
AIME (Competition Math Benchmark)
An olympiad-level math benchmark using American Invitational Mathematics Examination problems with integer answers, a key frontier reasoning test.
tau-bench (Tool-Agent-User Benchmark)
An agentic benchmark testing tool-using models in simulated customer-service dialogues that require following domain policies and calling APIs correctly.
Terminal-Bench
An agentic benchmark evaluating models on completing real command-line tasks inside a sandboxed terminal, verified by automated checks.
RULER (Long-Context Benchmark)
A synthetic long-context benchmark with configurable tasks measuring a model's effective context length beyond simple retrieval.
MTEB (Massive Text Embedding Benchmark)
A broad benchmark for text embedding models spanning classification, clustering, retrieval, reranking, and semantic similarity across many datasets and languages.
BEIR (Benchmarking Information Retrieval)
A heterogeneous zero-shot retrieval benchmark that tests how well a single retrieval model generalizes across diverse domains and query types without task-specific training.
IFEval (Instruction-Following Eval)
A benchmark that measures whether LLMs follow precise, verifiable formatting and content instructions using automatic checks rather than subjective judgment.
BFCL (Berkeley Function-Calling Leaderboard)
A benchmark for evaluating how accurately LLMs select, call, and parameterize functions and tools, including parallel, multiple, and multi-turn calling scenarios.
MGSM (Multilingual Grade School Math)
A multilingual extension of grade-school math word problems that tests whether LLMs can reason through arithmetic in many languages, not just English.
SimpleQA
A factuality benchmark of short, fact-seeking questions with single verifiable answers, designed to measure how often LLMs are correct, wrong, or appropriately abstain.
FRAMES (Factuality, Retrieval, And reasoning MEasurement Set)
A benchmark for retrieval-augmented generation that tests end-to-end factuality, multi-document retrieval, and multi-hop reasoning on questions needing several sources.
MuSR (Multistep Soft Reasoning)
A benchmark of long natural-language narratives requiring multistep commonsense and logical reasoning, such as murder mysteries and object-placement puzzles.
LiveBench
A contamination-resistant benchmark that continuously refreshes questions from recent sources and grades automatically against objective ground truth across many task categories.
BigCodeBench
A code-generation benchmark for realistic programming tasks that require composing many library calls, evaluated with rigorous test suites and high branch coverage.
CRUXEval (Code Reasoning, Understanding, and Execution)
A benchmark that tests whether models can reason about code execution by predicting function inputs from outputs and outputs from inputs.
RewardBench
A benchmark for reward models and LLM judges that measures how well they prefer better responses over worse ones across chat, reasoning, safety, and refusal cases.
AlpacaEval
An automated LLM-as-judge benchmark that estimates a model's win rate against a reference model on open-ended instructions, with a length-controlled variant to reduce verbosity bias.
Arena-Hard
An automatic benchmark of challenging, real-user-derived prompts graded by an LLM judge, built to align closely with human preference rankings and separate strong models.
ImageNet (ILSVRC Classification)
The foundational large-scale image classification benchmark covering 1,000 object categories, long used to track progress in computer vision and pretraining.
COCO (Object Detection and Segmentation)
A large-scale benchmark for object detection, instance segmentation, and keypoints in complex everyday scenes, evaluated with mean average precision across IoU thresholds.
VQAv2 (Visual Question Answering)
A benchmark that tests whether models can answer open-ended natural-language questions about images, balanced to reduce language-only shortcuts.
MMBench (Multimodal Benchmark)
A systematic multimodal benchmark that evaluates vision-language models across many fine-grained ability dimensions using a robustness-checked multiple-choice protocol.
DocVQA (Document Visual Question Answering)
A benchmark for answering questions about document images, testing OCR, layout understanding, and reasoning over text, tables, and forms.
ChartQA
A benchmark for answering questions about charts and plots that require visual data extraction plus arithmetic and logical reasoning over the extracted values.
LibriSpeech (ASR Word Error Rate)
A widely used benchmark for automatic speech recognition built from read English audiobooks, measured primarily by word error rate on clean and noisy splits.
HarmBench
A standardized red-teaming benchmark that measures how often automated attacks elicit harmful behaviors from LLMs and how well refusal and defenses hold up.
MLPerf Training
Industry-standard benchmark suite measuring how fast hardware and software systems train machine-learning models to a fixed target quality.
MLPerf Inference
Benchmark suite measuring how fast and efficiently systems serve trained ML models under realistic latency and throughput constraints.
MLPerf Tiny
Benchmark suite for ultra-low-power machine learning on microcontrollers and embedded devices, measuring latency, energy, and accuracy.
DAWNBench
Stanford benchmark that measured end-to-end deep-learning training and inference by time-to-accuracy and cost, popularizing those metrics.
See a real scan run
A replay of the actual CLI running against our test repositories — live progress, real findings, a genuine DriftScore. Nothing executes in your browser.