Trend Analysis Methodology

Detailed information about how trend analysis works, clustering algorithms, analysis parameters, and review types

Quick navigation

Looking for data extraction? View data extraction methodology →

How the analysis works

Stage 1: Converting text to vectors

Each paper abstract is converted into a 1,536-dimensional vector (embedding) using OpenAI's text-embedding-3-small model.

Think of this as a "semantic fingerprint" where:

  • Papers about similar topics have similar vectors
  • The distance between vectors reflects semantic similarity
  • This is deterministic: same text always produces the same vector

Example: Papers about "transformer architectures" will have vectors close to each other in this 1,536-dimensional space, while papers about "reinforcement learning" will cluster in a different region.

Stage 2: Mathematical clustering (no AI)

The clustering stage uses pgvector's cosine similarity with greedy community detection—a mathematical algorithm, not AI. This is pure geometry:

  1. Distance calculation: pgvector's cosine distance operator measures how far apart papers are in vector space
  2. Graph building: Papers with similarity above your chosen threshold become "neighbors"
  3. Greedy community detection: The algorithm starts with the paper that has the most neighbors, adds all its neighbors to form a cluster, then repeats

Why this isn't a black box: Greedy community detection is a well-studied graph algorithm with mathematical properties. No neural networks are involved in clustering—results are deterministic.

Stage 3: AI analysis

For each cluster, Claude Haiku 4.5 (by Anthropic) reads and summarizes papers:

  1. Calculate the cluster's centroid (mathematical center point)
  2. Sort papers by cosine similarity to centroid (closest = most representative)
  3. Select the top 20 papers closest to the centroid
  4. AI reads these abstracts and generates structured analysis based on your chosen review type

Transparency features:

  • • AI only sees what's in the prompt (the 20 abstracts)
  • • Papers are ranked mathematically, not by AI preference
  • • Every claim references specific papers you can verify
  • • Same inputs produce consistent outputs

Understanding review types

Each review type tells the AI to focus on different aspects of the same mathematical clusters. The clustering doesn't change—only what the AI looks for and reports.

Thematic Mapping

Purpose: Get a balanced overview of what's being researched

AI focuses on:

  • • Main approaches and methodologies
  • • Notable contributions and significant papers
  • • Trends and patterns across the literature
  • • Connections to broader research areas

Best for: Understanding the research landscape, writing literature reviews, getting oriented in a new field

Scoping Review

Purpose: Map the extent of research and identify gaps

AI focuses on:

  • • Breadth of research approaches and coverage
  • • Knowledge gaps and understudied areas
  • • Methodological diversity (or lack thereof)
  • • Future research directions based on gaps

Best for: Planning research projects, finding opportunities, strategic decisions about where to invest effort

State of the Art Review

Purpose: Track cutting-edge developments and recent advances

AI focuses on:

  • • Most advanced work defining the research frontier
  • • Recent breakthroughs and innovations
  • • Emerging trends just beginning to appear
  • • Technical evolution and open challenges

Best for: Staying current with fast-moving fields, competitive intelligence, understanding what's cutting edge

Analysis parameters explained

Cluster Discovery (Similarity Threshold)

Controls how tightly papers must be related to form a cluster. This is a cosine similarity threshold between paper vectors.

0.50 - Broad view (major clusters)

Papers must be very similar to cluster together. Results in fewer, larger clusters representing major research themes. Good for high-level overviews.

0.40 - Balanced (recommended)

Moderate similarity required. Balances breadth and detail—you'll see both major themes and some specialized areas.

0.30 - Detailed view (granular clusters)

Papers can be somewhat similar to cluster together. Results in more, smaller clusters capturing nuanced differences. Good for comprehensive analysis.

Minimum Cluster Size

Sets the minimum number of papers needed to form a cluster. Smaller values reveal emerging topics; larger values focus on well-established areas.

10+ papers (recommended)

Captures both established and emerging areas. Small enough to reveal new topics but large enough to filter noise.

15+ papers

Focuses on more established research areas with moderate activity.

20+ papers

Only shows well-established research areas with significant activity. Good for high-level strategic analysis.

Note: Papers that don't meet the minimum cluster size are marked as "noise" and not included in analysis. This helps focus on meaningful patterns rather than one-off papers.

What are clusters?

A cluster is a group of papers that are mathematically similar to each other. Think of it as a neighborhood in 1,536-dimensional space where papers about related topics naturally group together.

The term "cluster" is technically accurate, reflecting the clustering algorithms that group papers. Depending on the review type you choose, these clusters represent:

  • Thematic Mapping: Research themes or topics
  • Scoping Review: Research areas (some crowded, some sparse)
  • State of the Art: Frontiers or emerging trends

But fundamentally, they're all the same thing: groups of similar papers identified through mathematical clustering.

How papers are selected for analysis

For each cluster, AI analyzes the 20 papers most representative of that cluster. Here's exactly how they're selected:

  1. Calculate the centroid: Find the mathematical center point of all papers in the cluster (average of all vectors)
  2. Measure distances: Calculate cosine similarity between each paper and the centroid
  3. Sort by centrality: Papers closest to the centroid are the most "typical" or representative
  4. Select top 20: Choose the 20 papers with highest similarity to centroid (or all papers if fewer than 20)

Why this approach?

Papers closest to the centroid are the most representative of what makes the cluster coherent. They're not random—they're the mathematical "core" of the cluster. This ensures AI analyzes papers that best exemplify the cluster's identity.

Accuracy and limitations

What can be verified

  • All cited papers are real and linkable – Click any citation to view the arXiv abstract
  • Clustering is deterministic – Same papers + same parameters = same clusters
  • Paper selection is transparent – You can see which papers were analyzed (listed in cluster view)
  • AI only sees abstracts – No hidden knowledge or training on your domain

Limitations to be aware of

  • Abstract-only analysis – AI doesn't read full papers, so some nuances may be missed
  • Sample size: 20 papers per cluster – Large clusters (100+ papers) have more papers that aren't analyzed
  • AI summarization variability – Slight variations in wording between runs (though insights remain consistent)
  • arXiv only – Currently limited to AI/ML papers on arXiv (no conference proceedings, journals, or other domains yet)

Best practice recommendation

Use Future Scan as a starting point for research discovery, not as a replacement for reading papers. The analysis helps you quickly understand the landscape and find relevant papers, but you should always read the actual papers before citing them or relying on specific findings.

↑ Back to top