Horizon Scan

AI-driven intelligence platform for medical and pharma trends

About the project

Horizon Scan is an AI-agent platform for monitoring trends in pharma and biotech. Agents daily scan regulatory databases (FDA, EMA), clinical trial registries, scientific journals, press releases from key players, and social media (Telegram, Facebook, X, LinkedIn - official company channels and KOL profiles), extract facts, update trend pages, and publish a daily digest to social media (Telegram, Facebook, X) - without editor involvement. Scalability through a unified collection pipeline and transparent source traceability form the basis for market intelligence and compliance products. The platform is suitable for licensing to media and pharma companies as a white-label solution for up-to-date trend landscapes.

TriggerDaily on schedule or manually from the admin panel with one click
↓
CollectionSource crawl: regulatory databases, journals, clinical trial registries, company social media and KOL profiles
↓
ProcessingFact extraction, deduplication, normalization. Each fact is linked to a primary source
↓
AnalysisTrend dynamics, cross-signals, regulatory patterns
↓
GenerationAI produces text and tables with links to primary sources and dates
↓
ControlAuto-publish by default; a block can be switched to manual approval - control for regulatory and commercial scenarios. Full version history
↓
PublicationThe page updates automatically. Daily digest with creatives is posted to Telegram, Facebook, X

Technology stack

Fast Track MVP: minimal infrastructure, SaaS solutions, rapid delivery.

Scaling requires partial migration.

Characteristics: parallel streams: ≤ 20; stream volume: ≤ 12 trends; processing speed: ~60-75 trends/hour; max capacity: ~180-200 trends.

Pipeline
GitHubPythonPydanticPlaywright

Actions: daily workflow (collection → normalization → generation → write to Supabase → digest to social). Python 3.13; httpx, BeautifulSoup4, feedparser - REST API, HTML/XML, RSS / Atom. Playwright - JS sites

AI
Claude APIspaCy

Block generation from facts, RAG via pgvector. NER: drugs, indications, statuses, dates

Storage
Supabase

Postgres (pgvector): facts, block versions, RAG. Storage: raw documents. Auth: roles (editor, administrator, observer)

Frontend and API
Next.jsVercel

REST API: trend blocks, ISR / webhook. Admin: review queue, history, rollback

Repository and monitoring
GitHubVercel

Code, CI/CD. On pipeline failure - issue (GitHub Issues), notifications (Actions). Runtime - Vercel Runtime Logs; deploy and error alerts

Analytics
ClarityGA4

Behavior, web analytics

CMS
Sanity

Site content management: pages, copy, media. Visual editing, preview drafts on Vercel

Communication
ResendClaude API

Contact forms (email notifications). FAQ chatbot (Vercel AI SDK)

SMM and auto-publishing
Telegram Bot APIFacebook Graph APIX API v2GPT Image 1.5

Claude API - digest text and channel format adaptation. GPT Image 1.5 - creative generation (covers, cards). QA agent: fact, tone, size and format checks before publishing to site and social. GitHub Actions publishes on schedule

Proxy and anti-detection
Bright DataResidential Proxy

Residential proxy rotation for LinkedIn, Facebook, X. Sticky sessions, exponential backoff. User-Agent rotation and TLS fingerprint alignment for Playwright connectors

Security
Vercel FirewallSupabase RLSGitHub Secrets

DDoS: Vercel Firewall (L3/L4/L7) + WAF + Attack Challenge Mode. Supabase: Row Level Security on every table, JWT key rotation, service_role backend-only. API keys and social tokens - GitHub Encrypted Secrets

P1 Architecture and infrastructure

Architecture, data model, CI/CD, roles, API for Sanity and admin.

Tasks

  • 1Data schema design: raw documents, extracted facts, block versions
  • 2Internal entity model: drug, technology, indication, country, status
  • 3Dev environment, CI/CD, logging and error monitoring setup
  • 4Role separation: system, editor, administrator, observer
  • 5API design for CMS integration and internal editor UI
  • 6Security: RLS on every table, GitHub Encrypted Secrets for keys and tokens, Vercel WAF setup

Phase deliverables

  • ✓System architecture diagram
  • ✓ERD data model
  • ✓Deployed dev and test infrastructure

P2 Data source integration

Connectors to all sources, raw archive, synonym dictionary, monitoring.

Collection strategy by source

Regulatory and safety - API and RSS (FDA, EMA). Clinical trials - registry APIs (ClinicalTrials.gov, CTIS, WHO ICTRP). Publications - APIs (PubMed, Europe PMC, Crossref) and RSS from leading journals. Pharma companies - press releases and news from official sites. Social media - official company channels and KOL profiles (LinkedIn, Telegram, Facebook, X). All stored in raw archive with source and date metadata.

REST API RSS / Atom Playwright Web Service / httpx·BS4
Source Method
openFDAiRegulatory dataREST API
CTIS (ЕС)iClinical trialsREST API
CrossrefiScientific publicationsREST API
ClinicalTrials.goviClinical trialsREST API RSS / Atom
EMAiRegulatory dataREST API RSS / Atom
FDA AlertsiRegulatory dataREST API RSS / Atom
PubMed / Europe PMCiScientific publications
Lancet, NEJM, JAMA & etc		REST API RSS / Atom
InstitutionsiOrganizations
NIH, WHO, Uni Clinics & etc		REST API Playwright
Social mediaiChannels, official accounts, KOL profiles
Meta, X, Telegram & etc		REST API Playwright
Pharma industryiOrganizations
J&J, Merck, Pfizer, Roche & etc		RSS / Atom Playwright
WHO ICTRPiClinical trialshttpx / BS4

Tasks

  • 1DataSource abstraction with unified fetch() interface for all connectors
  • 2Synonym dictionary: INN, trade names, related terms
  • 3Raw document storage with date, source type and technical identifier metadata
  • 4User-Agent rotation, delays and residential proxies for Playwright connectors and social
  • 5Source database for parsing (URL, method, selectors/scenarios)
  • 6Source availability monitoring: alert when unavailable for more than one day

Phase deliverables

  • ✓Working connectors to all sources
  • ✓Synonym dictionary v1
  • ✓Source availability monitoring system

P3 Data normalization and cleansing

Unified data model, NER, status normalization, conflict queue.

Tasks

  • 1Deduplication and merging of different names for one entity
  • 2NER: extract drugs, technologies, indications, countries, statuses, dates, safety signals
  • 3Regulatory status normalization: approved, under review, restricted
  • 4Data alignment to unified internal model for page generation
  • 5Conflict detection and queuing for review

Phase deliverables

  • ✓Normalization pipeline
  • ✓Unified knowledge base v1
  • ✓Conflict log and review queue

P4 AI module: generation, analytics, insights

Block generation from facts, source traceability, tables from structure. Trend analytics: dynamics, cross-signals, regulatory trajectory.

Tasks

  • 1Material classification by page blocks: description, classification, clinical trials, safety
  • 2Key fact extraction without inferring missing data
  • 3Neutral, uniformly formatted paragraphs for publication
  • 4Link each paragraph and table row to one or more sources
  • 5Flag doubtful, conflicting or incomplete data for editorial review
  • 6Tables and lists from structured data: drugs, countries, clinical trials
  • 7Trend dynamics analysis: rise or fall in source activity over a period
  • 8Cross-source signals: confirmations from multiple independent sources
  • 9Regulatory trajectory, safety signal escalation, competitive landscape, geographic slice

Phase deliverables

  • ✓AI block generation module
  • ✓Trend page prototype
  • ✓Source traceability system
  • ✓Analytics and insights module

P5 Block updates and versioning

Update only changed blocks, versions, change log.

Tasks

  • 1Compare new block version with previous: update only on real changes
  • 2Update tables when a new drug appears or regulatory status changes
  • 3Update safety section when a new safety signal appears
  • 4Record daily check even when no changes occur
  • 5Store raw document, extracted facts, current and previous block versions
  • 6Change history: source, discovery date, publication date

Phase deliverables

  • ✓Block update engine
  • ✓Block version store
  • ✓Automatic change log

P6 Admin panel and review workflows

Review queue, approve/reject, version rollback.

Tasks

  • 1Review queue UI: changes to base trend description
  • 2Review UI: clinical significance and long-term effect interpretation
  • 3Review UI: safety, cost and access, summary conclusions
  • 4Approve or reject change with comment
  • 5Rollback to previous version of any block
  • 6Public frontend display: page update date, block date, sources, history
  • 7Manual collection trigger: "Refresh source" button → workflow via GitHub Actions API

Phase deliverables

  • ✓Admin panel
  • ✓Version rollback mechanism
  • ✓Editor notifications configured

P7 Site integration and launch

Production, CMS integration, tests, SMM and auto-publishing, documentation, handover.

Tasks

  • 1Sanity and public site integration via API
  • 2Final testing on "Gene therapy" trend per technical specification sample
  • 3Load testing of daily cycle: all sources, all blocks
  • 4Alerts and update quality monitoring setup
  • 5Acceptance testing per acceptance criteria
  • 6Clarity and GA4: events, goals, dashboards
  • 7Contact forms: Resend integration for email notifications
  • 8FAQ chatbot: Vercel AI SDK + Claude API for common questions
  • 9SMM: channels (Telegram, Facebook, X); creative and digest text generation, content QA, auto-publishing
  • 10Documentation and operational handover

Phase deliverables

  • ✓System in production
  • ✓Acceptance certificate per all technical specification criteria
  • ✓Web analytics, contact forms, FAQ chatbot
  • ✓QA agent checks content before publishing to site and social
  • ✓Operations documentation

Acceptance criteria

Daily update cycle runs stably without data loss
↓
Page is auto-generated in agreed structure; text blocks keep consistent style from update to update
↓
Each block has supporting sources and update date; tables and registry blocks update without manual assembly
↓
System updates only changed blocks; version history and change log are stored
↓
System detects trend dynamics, cross-source signals and regulatory patterns
↓
Editor can approve or reject changes in sensitive blocks
↓
QA agent checks content before publishing to site and social (Telegram, Facebook, X); daily digest with creatives is published automatically

© 2026 mcbile.app