Context Weaver: AI-Powered Code Intelligence

Stage 1: AST Parsing with Babel

Why: To search code semantically, you first need to understand its structure. Instead of treating files as plain text, Context Weaver parses them into Abstract Syntax Trees (ASTs)—understanding functions, classes, methods, parameters, and comments as distinct semantic units.

How: Using Babel (the same parser used by ESLint and Prettier), each TypeScript/JavaScript file is parsed into an AST. The parser extracts:

• Functions (regular, arrow, class methods) with signatures and line numbers
• Classes with their methods as separate searchable units
• JSDoc/TSDoc comments explaining what code does
• Import/export relationships showing module dependencies

// Example: Parsing extracts this structure
function validateUser(input: UserInput): boolean {
  /** Validates user input against schema */
  return schema.validate(input);
}

// Becomes this semantic chunk:
{
  filePath: "src/auth/validate.ts",
  chunkType: "function",
  name: "validateUser(input)",
  startLine: 15,
  endLine: 18,
  docstring: "Validates user input against schema",
  content: "function validateUser..."
}

Stage 2: Vector Embeddings for Semantic Search

Why: Traditional keyword search misses semantic similarity. "authentication logic" and "user login validation" mean the same thing but share no keywords. Vector embeddings solve this by representing code meaning as coordinates in high-dimensional space—similar concepts cluster together.

How: Each code chunk (function, class, method) is converted into a 1536-dimensional vector using OpenAI's text-embedding-3-small model. These embeddings capture not just what the code is named, but what it does. When developers search, their query is also embedded, and Weaviate's vector database performs approximate nearest neighbor (ANN) search to find semantically similar code—even if the exact words differ.

// Example: Embedding code chunks
const chunkText = `File: src/auth/validate.ts
Language: typescript
Type: function
Name: validateUser(input)

/** Validates user input against schema */
function validateUser(input: UserInput): boolean {
  return schema.validate(input);
}`;

const embedding = await openai.embeddings.create({
  model: "text-embedding-3-small",
  input: chunkText
});

// Store in Weaviate with vector
await weaviate.insertChunk(chunk, embedding.data[0].embedding);

Stage 3: LLM Q&A with Claude

Why: Finding relevant code is valuable, but developers often need explanation. "How does error handling work?" requires synthesizing information from multiple files and understanding architectural patterns. Large language models excel at this contextual reasoning.

How: When developers ask a question, Context Weaver: (1) searches for the top 5 most relevant code chunks, (2) sends them to Claude Sonnet 4.5 along with the question, (3) streams back a detailed explanation with source citations. The LLM can explain complex logic, trace execution flows, and provide architectural insights that pure search can't deliver.

AST Parsing is Worth the Investment

Initial temptation was to use simple regex to split functions—faster to build but fragile. Babel AST handles all edge cases: arrow functions, class methods, export declarations, decorators. The time investment paid off with robust, production-quality parsing that extracts rich metadata (line numbers, docstrings, signatures).

Batch Everything for Performance

Processing embeddings one-by-one would have taken ~50 seconds for 15 files. Batching 100 chunks per API call reduced this to 2.1 seconds—a 24x speedup. Always check API docs for batch endpoints before implementing sequential processing.

Dogfooding Accelerates Development

Using Context Weaver to index itself during development provided instant feedback on parsing quality and search relevance. Asking it questions about its own code revealed bugs and proved usefulness—if it can't understand itself, it won't understand user codebases.