Semantic search is revolutionizing how we find and retrieve information. Unlike traditional keyword-based search, semantic search understands the meaning and context of queries, delivering more relevant results. In this post, we'll build a semantic search system using Amazon S3 for vector storage and Microsoft's Semantic Kernel framework.
What is Semantic Search?
Semantic search uses machine learning models to understand the semantic meaning of text, converting words and phrases into high-dimensional vectors that capture their meaning in a mathematical space. Similar concepts cluster together in this vector space, enabling more intelligent search capabilities.
Architecture Overview
Our semantic search system consists of:
- Document Processing: Convert documents to embeddings
- Vector Storage: Store embeddings in Amazon S3
- Search Interface: Query processing with Semantic Kernel
- Results Ranking: Return most relevant documents
Setting Up Semantic Kernel
First, let's set up Semantic Kernel in our .NET application:
csharp1// Install the NuGet packages 2// Microsoft.SemanticKernel 3// Microsoft.SemanticKernel.Connectors.OpenAI 4// AWSSDK.S3 5 6using Microsoft.SemanticKernel; 7using Microsoft.SemanticKernel.Embeddings; 8 9var builder = WebApplication.CreateBuilder(args); 10 11// Configure Semantic Kernel 12builder.Services.AddSingleton<Kernel>(provider => 13{ 14 var kernelBuilder = Kernel.CreateBuilder(); 15 16 kernelBuilder.AddOpenAIChatCompletion( 17 "gpt-4", 18 Environment.GetEnvironmentVariable("OPENAI_API_KEY")); 19 20 kernelBuilder.AddOpenAITextEmbeddingGeneration( 21 "text-embedding-ada-002", 22 Environment.GetEnvironmentVariable("OPENAI_API_KEY")); 23 24 return kernelBuilder.Build(); 25});
Document Processing Service
Let's create a service to process documents and generate embeddings:
csharp1public class DocumentProcessor 2{ 3 private readonly ITextEmbeddingGenerationService _embeddingService; 4 private readonly IS3VectorStore _vectorStore; 5 6 public DocumentProcessor( 7 ITextEmbeddingGenerationService embeddingService, 8 IS3VectorStore vectorStore) 9 { 10 _embeddingService = embeddingService; 11 _vectorStore = vectorStore; 12 } 13 14 public async Task ProcessDocumentAsync(Document document) 15 { 16 // Split document into chunks 17 var chunks = SplitIntoChunks(document.Content, maxTokens: 512); 18 19 foreach (var (chunk, index) in chunks.Select((c, i) => (c, i))) 20 { 21 // Generate embedding for chunk 22 var embedding = await _embeddingService 23 .GenerateEmbeddingAsync(chunk); 24 25 // Create document vector 26 var documentVector = new DocumentVector 27 { 28 DocumentId = document.Id, 29 ChunkIndex = index, 30 Content = chunk, 31 Embedding = embedding.ToArray(), 32 Metadata = new Dictionary<string, object> 33 { 34 ["title"] = document.Title, 35 ["author"] = document.Author, 36 ["category"] = document.Category, 37 ["created"] = document.CreatedAt 38 } 39 }; 40 41 // Store in S3 42 await _vectorStore.StoreVectorAsync(documentVector); 43 } 44 } 45 46 private List<string> SplitIntoChunks(string content, int maxTokens) 47 { 48 // Simple chunking strategy - you might want to use more sophisticated methods 49 var words = content.Split(' '); 50 var chunks = new List<string>(); 51 var currentChunk = new List<string>(); 52 var currentTokenCount = 0; 53 54 foreach (var word in words) 55 { 56 // Rough token estimation (1 token ≈ 0.75 words) 57 var tokenCount = (int)Math.Ceiling(word.Length * 0.75); 58 59 if (currentTokenCount + tokenCount > maxTokens && currentChunk.Any()) 60 { 61 chunks.Add(string.Join(" ", currentChunk)); 62 currentChunk.Clear(); 63 currentTokenCount = 0; 64 } 65 66 currentChunk.Add(word); 67 currentTokenCount += tokenCount; 68 } 69 70 if (currentChunk.Any()) 71 { 72 chunks.Add(string.Join(" ", currentChunk)); 73 } 74 75 return chunks; 76 } 77}
S3 Vector Store Implementation
Here's how to implement vector storage using Amazon S3:
csharp1public class S3VectorStore : IS3VectorStore 2{ 3 private readonly IAmazonS3 _s3Client; 4 private readonly string _bucketName; 5 private readonly ILogger<S3VectorStore> _logger; 6 7 public S3VectorStore( 8 IAmazonS3 s3Client, 9 IConfiguration configuration, 10 ILogger<S3VectorStore> logger) 11 { 12 _s3Client = s3Client; 13 _bucketName = configuration["AWS:S3:VectorBucket"]; 14 _logger = logger; 15 } 16 17 public async Task StoreVectorAsync(DocumentVector vector) 18 { 19 var key = $"vectors/{vector.DocumentId}/{vector.ChunkIndex}.json"; 20 21 var vectorData = new 22 { 23 documentId = vector.DocumentId, 24 chunkIndex = vector.ChunkIndex, 25 content = vector.Content, 26 embedding = vector.Embedding, 27 metadata = vector.Metadata, 28 timestamp = DateTime.UtcNow 29 }; 30 31 var json = JsonSerializer.Serialize(vectorData); 32 33 var request = new PutObjectRequest 34 { 35 BucketName = _bucketName, 36 Key = key, 37 ContentBody = json, 38 ContentType = "application/json", 39 Metadata = 40 { 41 ["document-id"] = vector.DocumentId.ToString(), 42 ["chunk-index"] = vector.ChunkIndex.ToString() 43 } 44 }; 45 46 await _s3Client.PutObjectAsync(request); 47 _logger.LogInformation("Stored vector for document {DocumentId}, chunk {ChunkIndex}", 48 vector.DocumentId, vector.ChunkIndex); 49 } 50 51 public async Task<List<DocumentVector>> SearchSimilarVectorsAsync( 52 float[] queryEmbedding, 53 int topK = 10) 54 { 55 var results = new List<(DocumentVector Vector, double Similarity)>(); 56 57 // List all vector objects in S3 58 var listRequest = new ListObjectsV2Request 59 { 60 BucketName = _bucketName, 61 Prefix = "vectors/" 62 }; 63 64 var response = await _s3Client.ListObjectsV2Async(listRequest); 65 66 // Process each vector file 67 foreach (var obj in response.S3Objects) 68 { 69 try 70 { 71 var getRequest = new GetObjectRequest 72 { 73 BucketName = _bucketName, 74 Key = obj.Key 75 }; 76 77 using var getResponse = await _s3Client.GetObjectAsync(getRequest); 78 using var reader = new StreamReader(getResponse.ResponseStream); 79 var json = await reader.ReadToEndAsync(); 80 81 var vectorData = JsonSerializer.Deserialize<JsonElement>(json); 82 var embedding = vectorData.GetProperty("embedding") 83 .EnumerateArray() 84 .Select(x => (float)x.GetDouble()) 85 .ToArray(); 86 87 var similarity = CalculateCosineSimilarity(queryEmbedding, embedding); 88 89 var documentVector = new DocumentVector 90 { 91 DocumentId = Guid.Parse(vectorData.GetProperty("documentId").GetString()), 92 ChunkIndex = vectorData.GetProperty("chunkIndex").GetInt32(), 93 Content = vectorData.GetProperty("content").GetString(), 94 Embedding = embedding, 95 Metadata = vectorData.GetProperty("metadata") 96 .EnumerateObject() 97 .ToDictionary(p => p.Name, p => (object)p.Value.ToString()) 98 }; 99 100 results.Add((documentVector, similarity)); 101 } 102 catch (Exception ex) 103 { 104 _logger.LogWarning(ex, "Failed to process vector file {Key}", obj.Key); 105 } 106 } 107 108 return results 109 .OrderByDescending(r => r.Similarity) 110 .Take(topK) 111 .Select(r => r.Vector) 112 .ToList(); 113 } 114 115 private static double CalculateCosineSimilarity(float[] vector1, float[] vector2) 116 { 117 if (vector1.Length != vector2.Length) 118 throw new ArgumentException("Vectors must have the same length"); 119 120 var dotProduct = vector1.Zip(vector2, (a, b) => a * b).Sum(); 121 var magnitude1 = Math.Sqrt(vector1.Sum(v => v * v)); 122 var magnitude2 = Math.Sqrt(vector2.Sum(v => v * v)); 123 124 if (magnitude1 == 0 || magnitude2 == 0) 125 return 0; 126 127 return dotProduct / (magnitude1 * magnitude2); 128 } 129}
Search Service
Now let's implement the search functionality:
csharp1public class SemanticSearchService 2{ 3 private readonly ITextEmbeddingGenerationService _embeddingService; 4 private readonly IS3VectorStore _vectorStore; 5 private readonly ILogger<SemanticSearchService> _logger; 6 7 public SemanticSearchService( 8 ITextEmbeddingGenerationService embeddingService, 9 IS3VectorStore vectorStore, 10 ILogger<SemanticSearchService> logger) 11 { 12 _embeddingService = embeddingService; 13 _vectorStore = vectorStore; 14 _logger = logger; 15 } 16 17 public async Task<SearchResults> SearchAsync( 18 string query, 19 int maxResults = 10, 20 double minimumSimilarity = 0.7) 21 { 22 _logger.LogInformation("Performing semantic search for query: {Query}", query); 23 24 // Generate embedding for the search query 25 var queryEmbedding = await _embeddingService.GenerateEmbeddingAsync(query); 26 27 // Search for similar vectors 28 var similarVectors = await _vectorStore.SearchSimilarVectorsAsync( 29 queryEmbedding.ToArray(), 30 maxResults * 2); // Get more results to filter by similarity 31 32 // Group by document and calculate relevance scores 33 var documentResults = similarVectors 34 .GroupBy(v => v.DocumentId) 35 .Select(g => new SearchResult 36 { 37 DocumentId = g.Key, 38 Title = g.First().Metadata.GetValueOrDefault("title")?.ToString() ?? "Unknown", 39 Snippets = g.Select(v => v.Content).Take(3).ToList(), 40 RelevanceScore = g.Average(v => CalculateCosineSimilarity( 41 queryEmbedding.ToArray(), v.Embedding)), 42 Metadata = g.First().Metadata 43 }) 44 .Where(r => r.RelevanceScore >= minimumSimilarity) 45 .OrderByDescending(r => r.RelevanceScore) 46 .Take(maxResults) 47 .ToList(); 48 49 return new SearchResults 50 { 51 Query = query, 52 Results = documentResults, 53 TotalFound = documentResults.Count, 54 ExecutionTimeMs = 0 // You might want to measure this 55 }; 56 } 57 58 private static double CalculateCosineSimilarity(float[] vector1, float[] vector2) 59 { 60 // Same implementation as in S3VectorStore 61 if (vector1.Length != vector2.Length) 62 throw new ArgumentException("Vectors must have the same length"); 63 64 var dotProduct = vector1.Zip(vector2, (a, b) => a * b).Sum(); 65 var magnitude1 = Math.Sqrt(vector1.Sum(v => v * v)); 66 var magnitude2 = Math.Sqrt(vector2.Sum(v => v * v)); 67 68 if (magnitude1 == 0 || magnitude2 == 0) 69 return 0; 70 71 return dotProduct / (magnitude1 * magnitude2); 72 } 73}
API Controller
Finally, let's create an API controller to expose the search functionality:
csharp1[ApiController] 2[Route("api/[controller]")] 3public class SearchController : ControllerBase 4{ 5 private readonly SemanticSearchService _searchService; 6 7 public SearchController(SemanticSearchService searchService) 8 { 9 _searchService = searchService; 10 } 11 12 [HttpPost("semantic")] 13 public async Task<ActionResult<SearchResults>> SemanticSearch( 14 [FromBody] SearchRequest request) 15 { 16 if (string.IsNullOrWhiteSpace(request.Query)) 17 { 18 return BadRequest("Query cannot be empty"); 19 } 20 21 var results = await _searchService.SearchAsync( 22 request.Query, 23 request.MaxResults ?? 10, 24 request.MinimumSimilarity ?? 0.7); 25 26 return Ok(results); 27 } 28} 29 30public record SearchRequest( 31 string Query, 32 int? MaxResults, 33 double? MinimumSimilarity);
Optimization Strategies
1. Caching
Implement Redis caching for frequently accessed embeddings and search results.
2. Batch Processing
Process multiple documents in batches to improve throughput.
3. Vector Indexing
For production systems, consider using specialized vector databases like Pinecone or Weaviate instead of S3 for better performance.
4. Hybrid Search
Combine semantic search with traditional keyword search for better coverage.
Conclusion
We've built a complete semantic search system using Amazon S3 and Semantic Kernel. This solution provides:
- Scalable vector storage using S3
- Powerful semantic understanding with OpenAI embeddings
- Flexible search capabilities with Semantic Kernel
- Production-ready architecture
While S3 works well for smaller datasets, consider dedicated vector databases for high-performance requirements. The foundation we've built here can be easily extended with additional features like real-time indexing, advanced filtering, and multi-modal search capabilities.
Start experimenting with semantic search in your applications – you'll be amazed at how much more relevant and intelligent your search results can become!
