I Turned 9,575 Jewelry Comments Into Landing-Page Tests—Without Calling Reddit a Market Survey

Introduction

Traditional market research moves at the speed of spreadsheet exports. You draft questions, wait for respondents, clean the data, and hope the insights translate to conversion. By the time the report lands on the product manager’s desk, the campaign window has usually closed. We needed a faster signal. Instead of designing a survey, we built a pipeline that treats organic community discussion as a continuous data stream. Over a twelve-hour window, we ingested 9,575 comments from niche jewelry subreddits, ran them through a vector clustering pipeline, and fed the resulting themes into a structured hypothesis generator. The output: forty-two landing page variants, deployed automatically, with traffic routed by a Bayesian bandit. This article covers the architecture, the code, and the production trade-offs of turning unstructured social text into measurable growth experiments.

Why This Matters

Growth engineering is no longer a copywriting exercise. It is a data pipeline problem. When you treat community discussion as latent user intent, you remove the bias of leading questions and the latency of manual analysis. Engineers should care because this pattern bridges three production systems: natural language processing, automated experimentation, and real-time traffic routing. It solves a concrete pain point: the bottleneck between insight discovery and variant deployment. By extracting themes directly from how users actually talk about a product category, you generate testable hypotheses that reflect genuine purchase drivers, not focus-group guesswork. The architecture scales horizontally, respects API limits, and keeps human reviewers in the loop only where it matters.

How It Works

The pipeline operates as a continuous extraction-to-experimentation loop. Raw comment streams are ingested asynchronously, normalized for noise, and passed through a dense embedding model. The resulting vectors are clustered using density-based algorithms to surface recurring themes without predefined labels. Each cluster is summarized and fed into a large language model with strict JSON schema validation. The LLM returns structured hypothesis payloads containing headline variants, subhead copy, CTA text, and supporting social proof cues. A variant builder serializes these payloads into lightweight frontend configurations. Traffic is allocated using a Bayesian multi-armed bandit that continuously updates allocation weights based on conversion signals. A feedback evaluator monitors population stability and triggers model retraining when distribution drift exceeds a threshold.

flowchart TD
    R[Reddit Comment Stream] --> S[Async Ingestion Service]
    S --> C[Text Normalizer & Deduplicator]
    C --> E[Dense Embedding Model]
    E --> CL[Semantic Clusterer]
    CL --> H[LLM Hypothesis Generator]
    H --> V[Landing Page Variant Builder]
    V --> TR[Bayesian Traffic Router]
    TR --> LP[Live Landing Page]
    LP --> AC[Event Collector & Metric Store]
    AC --> FE[Feedback Evaluator]
    FE -->|Drift Detected| RT[Model Retraining Pipeline]
    FE -->|Stable| TR
    RT --> E
    RT --> H

ingestion handles rate limits, payload validation, and deduplication. Normalization strips markdown, resolves slang, and collapses near-identical phrasing. The embedding model projects text into a fixed-dimensional space where semantic similarity maps to Euclidean distance. Clustering groups vectors by density, naturally filtering outliers and bot noise. The LLM step is constrained by a Pydantic schema to guarantee machine-readable output. The variant builder compiles the payload into a JSON config that the frontend hydration layer reads at runtime. The traffic router updates posterior distributions for each variant, shifting allocation toward performers without stopping the experiment prematurely. The feedback evaluator tracks conversion rate variance and embedding distribution shift, closing the loop by retraining the clustering parameters when drift exceeds a configurable bound.

Core Concepts

• Dense Passage Embedding: We use a transformer-based encoder to map comments to 384-dimensional vectors. The model captures latent intent rather than keyword matches, allowing comments about “heirloom durability” and “gift longevity” to cluster together.
• Density-Based Clustering (HDBSCAN): Unlike K-Means, HDBSCAN does not require a predefined cluster count. It identifies core points, border points, and noise, which aligns with how organic discussions naturally form around specific pain points or desires.
• Structured LLM Output: Unconstrained generation breaks downstream pipelines. We enforce a strict JSON schema with Pydantic validation, rejecting malformed responses and falling back to deterministic templates when confidence scores drop.
• Bayesian Traffic Routing: Instead of fixed 50/50 splits, we use a Thompson sampling approach. Each variant maintains a Beta distribution for conversion. The router draws samples, allocates traffic proportionally, and updates posteriors as events stream in.
• Drift Detection & Retraining Trigger: We monitor the Population Stability Index on conversion metrics and the cosine similarity distribution of incoming comment embeddings. When drift crosses a threshold, the pipeline queues a lightweight retraining job to adjust clustering tolerances and prompt templates.

Examples & Code Walkthrough

The following module demonstrates the core pipeline components. It is written for production use, with defensive error handling, type hints, and realistic fallback logic.

import asyncio
import json
import logging
import numpy as np
from typing import List, Dict, Any
from dataclasses import dataclass
from pydantic import BaseModel, ValidationError
from sentence_transformers import SentenceTransformer
import hdbscan

logger = logging.getLogger("growth_pipeline")

# 1. Structured Hypothesis Schema
class LandingVariant(BaseModel):
    variant_id: str
    headline: str
    subhead: str
    cta_text: str
    social_proof_hint: str
    confidence_score: float

# 2. Embedding & Clustering Service
class SemanticClusterer:
    def __init__(self, model_name: str = "all-MiniLM-L6-v2", min_cluster_size: int = 15):
        self.encoder = SentenceTransformer(model_name)
        self.min_cluster_size = min_cluster_size
        self._model_cache: Dict[str, np.ndarray] = {}

    def embed_batch(self, texts: List[str]) -> np.ndarray:
        # Cache hits avoid redundant transformer forward passes
        if len(texts) == 1 and texts[0] in self._model_cache:
            return self._model_cache[texts[0]]
        
        try:
            vectors = self.encoder.encode(texts, show_progress_bar=False, normalize_embeddings=True)
            for text, vec in zip(texts, vectors):
                self._model_cache[text] = vec
            return vectors
        except Exception as exc:
            logger.error("Embedding failure: %s", exc)
            raise

    def cluster(self, vectors: np.ndarray) -> List[Dict[str, Any]]:
        clusterer = hdbscan.HDBSCAN(
            min_cluster_size=self.min_cluster_size,
            metric="cosine",
            cluster_selection_epsilon=0.3
        )
        labels = clusterer.fit_predict(vectors)
        
        clusters = {}
        for idx, label in enumerate(labels):
            if label == -1:
                continue  # Skip noise
            if label not in clusters:
                clusters[label] = []
            clusters[label].append(idx)
        
        return [{"cluster_id": k, "comment_indices": v} for k, v in clusters.items()]

# 3. Structured LLM Hypothesis Generator (Mocked for pipeline integrity)
def generate_hypotheses(cluster_samples: List[str], cluster_id: int) -> LandingVariant:
    # In production, this calls an LLM API with system prompt + JSON schema enforcement
    # Here we simulate deterministic fallback + validation
    prompt_context = "\n".join(cluster_samples[:5])
    
    # Simulated LLM response structure
    raw_response = {
        "variant_id": f"v-{cluster_id:03d}",
        "headline": f"Crafted for {cluster_id} buyers",
        "subhead": "Based on community feedback",
        "cta_text": "View Collection",
        "social_proof_hint": "Trusted by collectors",
        "confidence_score": 0.82
    }
    
    try:
        return LandingVariant(**raw_response)
    except ValidationError as err:
        logger.warning("LLM schema validation failed for cluster %d: %s", cluster_id, err)
        # Fallback to safe defaults
        return LandingVariant(
            variant_id=f"fallback-{cluster_id:03d}",
            headline="Timeless Jewelry, Built to Last",
            subhead="Designed with your lifestyle in mind",
            cta_text="Shop Now",
            social_proof_hint="Join thousands of satisfied buyers",
            confidence_score=0.50
        )

# 4. Pipeline Orchestrator
async def run_hypothesis_pipeline(raw_comments: List[str]) -> List[LandingVariant]:
    clusterer = SemanticClusterer()
    
    # Batch size tuned for GPU memory limits
    BATCH_SIZE = 256
    all_vectors = []
    for i in range(0, len(raw_comments), BATCH_SIZE):
        batch = raw_comments[i:i+BATCH_SIZE]
        vectors = clusterer.embed_batch(batch)
        all_vectors.append(vectors)
    
    if not all_vectors:
        logger.warning("No valid embeddings generated. Aborting pipeline.")
        return []
        
    combined_vectors = np.vstack(all_vectors)
    clusters = clusterer.cluster(combined_vectors)
    
    variants = []
    for cluster in clusters:
        cid = cluster["cluster_id"]
        indices = cluster["comment_indices"]
        samples = [raw_comments[i] for i in indices[: