Thoughts on Software Architecture

What is a Large Language Model?

Understanding Inference

Large Language Models Come in Many Sizes and Flavors

Retrieval vs Generative Models

Tokenization: Breaking Text into Pieces

Context Size: How Much Information Can a Language Model Use During Inference?

Modalities: Beyond Text

Provider Ecosystems

Choosing an LLM Provider

OpenRouter

Thinking About Performance

Experimenting With Different LLM Models

Compound AI Systems

Deployment Patterns for Compound AI Systems

Roles in Compound AI Systems

Latent Space: Incomprehensibly Vast

How The Path Gets “Narrowed”

Turning Down The Temperature

Hyperparameters: Knobs and Dials of Inference

Raw Versus Instruct-Tuned Models

Raw Models: The Unfiltered Canvas

Instruct-Tuned Models: The Guided Experience

Choosing the Right Kind of Model for Your Project

Prompt Engineering

The Building Blocks of Effective Prompts

The Art and Science of Prompt Design

Prompt Engineering Techniques and Best Practices

Zero-Shot Learning: When No Examples Are Needed

One-Shot Learning: When a Single Example Can Make a Difference

Few-Shot Learning: When Multiple Examples Can Improve Performance

Example: Prompts Can Be Much More Complex Than You Imagine

Experimentation and Iteration

The Art of Vagueness

Why Anthropomorphism Dominates Prompt Engineering

Separating Instructions from Data: A Crucial Principle

Prompt Distillation

How It Works

Initial Prompt Generation

Prompt Refinement

Prompt Compression

System Directive and Context Integration

Final Prompt Assembly

Key Benefits

What about fine-tuning?

What is Retrieval Augmented Generation?

How Does RAG Work?

Why Use RAG in Your Applications?

Implementing RAG in Your Application

Preparation of Knowledge Sources (Chunking)

Proposition Chunking

Implementation Notes

Quality Check

Benefits of Proposition-Based Retrieval

Real-World Examples of RAG

Case Study: RAG in a Tax Preparation Application Without Embeddings

Intelligent Query Optimization (IQO)

Reranking

RAG Assessment (RAGAs)

Faithfulness

Answer Relevance

Context Precision

Context Relevancy

Context Recall

Context Entities Recall

Answer Semantic Similarity (ANSS)

Answer Correctness

Aspect Critique

Challenges and Future Outlook

Semantic Chunking: Enhancing Retrieval with Context-Aware Segmentation

Hierarchical Indexing: Structuring Data for Improved Retrieval

Self-RAG: A Self-Reflective Enhancement

HyDE: Hypothetical Document Embeddings

AI Workers As Independent Reusable Components

Account Management

E-commerce Applications

Product Recommendations

Fraud Detection

Customer Sentiment Analysis

Healthcare Applications

Patient Intake

Patient Risk Assessment

AI Worker as a Process Manager

Store Your Trigger Messages

Integrating AI Workers Into Your Application Architecture

Designing Clear Interfaces and Communication Protocols

Handling Data Flow and Synchronization

Managing the Lifecycle of AI Workers

Composability and Orchestration of AI Workers

Chaining AI Workers for Multi-Step Workflows

Parallel Processing for Independent AI Workers

Ensemble Techniques for Improved Accuracy

Dynamic Selection and Invocation of AI Workers

Combining Traditional NLP with LLMs

What is Tool Use?

The Potential of Tool Use

The Tool Use Workflow

Include function definitions in your request context

Dynamic Tool Selection

Forced (aka Explicit) Tool Selection

Forcing a Function To Get Structured Output

Execution of Function(s)

Optional Continuation of the Original Prompt

Best Practices for Tool Use

Descriptive Definitions

Processing of Tool Results

Error Handling

Iterative Refinement

Composing and Chaining Tools

Future Directions

Implementating a ReplyStream

The “Conversation Loop”

Auto Continuation

Conclusion

Practical Case Study: Fixing Broken JSON

Considerations and Counterindications

Data Criticality

Error Severity

Domain Complexity

Explainability and Transparency

Unintended Consequences

Personalization

Productivity

Rapid Iteration and Experimentation

Scalability and Efficiency

AI Powered Localization

The Importance of User Testing and Feedback

Generating Copy for User Interfaces

Personalized Forms

Context-Aware Explanatory Text

Defining Generative UI

Example

The Shift to Outcome-Oriented Design

Challenges and Considerations

Future Outlook and Opportunities

Business Need

Key Benefits

Key Patterns

Dynamic Task Routing

Contextual Decision Making

Adaptive Workflow Composition

Exception Handling and Recovery

Implementing Intelligent Workflow Orchestration in Practice

Intelligent Order Processor

Intelligent Content Moderator

Predictive Task Scheduling in a Customer Support System

Exception Handling and Recovery in a Data Processing Pipeline

Monitoring and Logging

Monitoring Workflow Progress and Performance

Logging Key Events and Decisions

Benefits of Monitoring and Logging

Considerations and Best Practices

Scalability and Performance Considerations

Handling High Volumes of Concurrent Workflows

Optimizing Performance of AI-Powered Components

Monitoring and Profiling Performance

Scaling Strategies

Performance Optimization Techniques

Testing and Validation of Workflows

Unit Testing Workflow Components

Integration Testing Workflow Interactions

Testing AI Decision Points

End-to-End Testing

Continuous Integration and Deployment

Chain of Thought

How It Works

Examples

Benefits and Considerations

Mode Switch

How It Works

When to Use It

Example

Role Assignment

How It Works

When to Use It

Examples

Prompt Object

How It Works

Prompt Template

How It Works

Benefits and Considerations

When to Use It:

Example

Structured IO

How It Works

Scaling Structured IO

Benefits and Considerations

Prompt Chaining

How It Works

When To Use It

Example: Olympia’s Onboarding

Prompt Rewriter

How It Works

Example

Response Fencing

How It Works

Benefits and Considerations

Error Handling

Query Analyzer

How It Works

Implementation

Benefits

Query Rewriter

How It Works

Example

Benefits

Ventriloquist

How It Works

When to Use It

Example

Predicate

How It Works

When to Use It

Example

API Facade

How It Works

Key Benefits

When To Use It

Example

Scalability and Performance Considerations

Comparison with Other Design Patterns

Result Interpreter

How It Works

When to Use It

Example

Virtual Machine

How It Works

When to Use It

Example

Behind The Magic

Specification and Testing

Specifying the Behavior

Writing Test Cases

Example: Testing the Translator Component

Replay of HTTP Interactions

High-Level Patterns

Hybrid Intelligence

Adaptive Response

Human-AI Role Switching

Escalation

How It Works

Key Benefits

Feedback Loop

How It Works

Applications and Examples

Advanced Techniques in Human Feedback Integration

Passive Information Radiation

How It Works

Key Benefits

Applications and Examples

Collaborative Decision Making (CDM)

How It Works

Example

Continuous Learning

How It Works

Applications and Examples

Example

Ethical Considerations

Role of HITL in Mitigating AI Risks

Technological Advancements and Future Outlook

Challenges and Limitations of HITL Systems

Traditional Error Handling Approaches

Contextual Error Diagnosis

How It Works

Intelligent Error Reporting

Predictive Error Prevention

How It Works

Smart Error Recovery

How It Works

Personalized Error Communication

How It Works

Adaptive Error Handling Workflow

How It Works

Eval

Problem

Solution

How It Works

Example

Considerations

Guardrail

Problem

Solution

How It Works

Example

Considerations

Guardrails and Evals: Two Sides of the Same Coin

The Interchangeability of Guardrails and Reference-Free Evals

Implementing Dual-Purpose Guardrails and Evals

A

B

C

D

E

F

G

H

Inference

J

K

L

M

N

O

P

Q

R

S

T

U

V

W

Z