Pubky Nexus

Pubky Nexus is the production-grade indexing and aggregation service that powers Pubky App’s social features. It transforms decentralized data from multiple Homeservers into a high-performance social graph API, enabling real-time social media experiences at scale.

Overview

Nexus serves as the central bridge between Pubky Homeservers and social clients, implementing the aggregator, indexer, and web server components of the custom backend architecture. By aggregating events from Homeservers into a rich social graph, Nexus provides the infrastructure needed for features like feeds, search, recommendations, and real-time notifications.

⚠️ Note: The Nexus API is currently in active development. The service uses the /v0 route prefix to indicate API instability, and breaking changes may occur as the system evolves toward stability.

Key Features

Nexus continuously ingests events from multiple Pubky Homeservers, building and maintaining a structured social graph in real-time. This enables features like:

Following relationships and friend networks
Post interactions (likes, replies, mentions)
Tag-based content discovery
User muting and moderation

Full-Content Indexing

Rather than requiring clients to locate and query individual Homeservers for content, Nexus indexes and serves content directly. This dramatically improves latency and user experience while still maintaining the decentralized nature of the underlying data. Clients can optionally verify content authenticity directly with Homeservers when needed.

High Performance & Scalability

Built in Rust with carefully optimized data structures, Nexus is designed for speed:

Sub-millisecond response times: Most requests are served in less than 1ms
Constant time complexity: Query performance doesn’t degrade as user base grows
Efficient caching: Redis-based caching layer accelerates common queries
Horizontal scalability: Architecture supports distributed deployment

Nexus supports Social Semantic Graph-based interactions, enabling:

web-of-trust relationship mapping
Content filtering based on social connections
Personalized feed ranking and recommendations
Community detection and trust inference

Graph-Enhanced Search & Recommendations

Leveraging Neo4j for graph database operations, Nexus provides:

Deep relationship queries across the social graph
Recommendation algorithms based on network topology
Tag and content discovery through graph traversal
Influencer and community identification

Flexible Caching Architecture

A sophisticated Redis caching layer ensures optimal performance:

Common queries cached for instant retrieval
Incremental cache updates on new events
Minimal database load for read-heavy workloads
Cache invalidation synchronized with graph updates

Architecture

Nexus is composed of several specialized components working together:

Components

nexus-watcher: The event aggregator that monitors Pubky Homeservers
- Subscribes to Homeserver event streams
- Filters and validates incoming events
- Translates events into social graph updates
- Handles retry logic for failed operations
nexus-webapi: The REST API server (formerly nexus-service)
- Serves client requests via HTTP/REST endpoints
- Implements OpenAPI/Swagger specification
- Handles authentication and rate limiting
- Returns formatted responses to Pubky App frontend
nexus-common: Shared library for common functionality
- Database connectors (Redis, Neo4j)
- Data models and schemas
- Query builders and utilities
- Configuration management
nexusd: Service orchestration daemon
- Manages component lifecycle
- Performs database migrations
- Handles reindexing operations
- Provides CLI for administration

Data Flow

Event Ingestion: The watcher monitors multiple Homeservers, receiving events as they occur (new posts, follows, likes, etc.)
Event Processing: Events are validated, filtered based on configured rules, and transformed into graph operations
Indexing: Processed events update both the Redis cache (for fast queries) and Neo4j graph database (for complex relationships)
API Responses: Client requests hit the web API, which serves data from the optimized indexes with sub-millisecond latency

Technology Stack

Language: Rust (for performance and safety)
Graph Database: Neo4j (for relationship queries and graph algorithms)
Cache Layer: Redis (for high-speed access to common queries)
API Framework: Axum (Rust web framework)
Observability: OpenTelemetry integration (optional)

API Endpoints

Nexus provides a comprehensive REST API for social features:

Live API Access

Staging (latest): https://nexus.staging.pubky.app/swagger-ui/
Production (stable): https://nexus.pubky.app/swagger-ui/

Explore the full API specification, test queries, and view response schemas directly through the Swagger UI.

Key Endpoint Categories

User endpoints: Profile data, follower/following relationships, user search
Post endpoints: Post creation/retrieval, replies, mentions, bookmarks
Feed endpoints: Timeline generation, filtered streams, personalized feeds
Tag endpoints: Tag-based discovery, trending tags, tag streams
Search endpoints: Full-text search across users and content
Graph endpoints: Relationship queries, web-of-trust calculations
Notification endpoints: Real-time notification delivery

Observability & Monitoring

Nexus provides rich observability features for operators:

Database Exploration

Redis Insight: Inspect cached data structures in real-time at http://localhost:8001/redis-stack/browser (local dev)
Neo4j Browser: Visualize and query the social graph at http://localhost:7474/browser/ (local dev)

Telemetry Integration

Optional OpenTelemetry integration for production monitoring:

Distributed tracing across components
Performance metrics and latency tracking
Error rate monitoring and alerting
Integration with tools like Signoz, Jaeger, or Prometheus

Development & Deployment

Prerequisites

Rust toolchain (latest stable)
Docker and Docker Compose (for databases)
Neo4j (graph database)
Redis (caching layer)

Quick Start

# Clone the repository
git clone https://github.com/pubky/pubky-nexus
cd pubky-nexus

# Set up databases via Docker
cd docker
cp .env-sample .env
docker compose up -d

# Run the service (uses default config)
cargo run -p nexusd

# Or run components individually
cargo run -p nexusd -- watcher  # Run event watcher
cargo run -p nexusd -- api      # Run API server

Configuration

Nexus uses a TOML configuration file (default location: $HOME/.pubky-nexus/config.toml). Custom config paths can be specified:

cargo run -p nexusd -- --config-dir="custom/config/folder"

Testing & Benchmarking

# Load mock data for testing
cargo run -p nexusd -- db mock

# Run unit tests
cargo nextest run -p nexus-common --no-fail-fast
cargo nextest run -p nexus-watcher --no-fail-fast
cargo nextest run -p nexus-webapi --no-fail-fast

# Run benchmarks
cargo bench -p nexus-webapi
cargo bench -p nexus-webapi --bench user  # Specific endpoint

Data Migration System

Nexus includes a sophisticated migration manager for handling breaking changes to data structures:

Migration Phases

Dual Write: New writes go to both old and new data sources simultaneously
Backfill: Historical data is migrated from old to new source
Cutover: Reads switch to the new data source
Cleanup: Old data sources are safely removed

Managing Migrations

# Create a new migration
cargo run -p nexusd -- db migration new MigrationName

# Run pending migrations
cargo run -p nexusd -- db migration run

# Clear database (use with caution)
cargo run -p nexusd -- db clear

Use Cases

Nexus enables a variety of social features for applications built on Pubky:

Chronological and algorithmic timelines
Filtered feeds by tags, authors, or topics
Personalized recommendations based on social graph
Real-time updates without polling

Discovery & Search

Full-text search across posts and users
Tag-based content discovery
Trending topics and hot tags
User recommendations based on network proximity

Moderation & Filtering

User muting and blocking
Community-based filtering
Spam detection and prevention
Custom content policies per instance

Analytics & Insights

User reach and influence metrics
Content engagement tracking
Network growth analysis
Community detection and clustering

Deployment Options

Self-Hosted Instance

Organizations can run their own Nexus instance with custom:

Content filtering and moderation policies
Event source selection (which Homeservers to index)
Caching strategies and database configuration
API rate limits and access controls

Public Instances

The Pubky team operates public instances:

Production: https://nexus.pubky.app
Staging: https://nexus.staging.pubky.app

Hybrid Approaches

Clients can use a combination of:

Public Nexus instances for general discovery
Private instances for specialized communities
Direct Homeserver queries for verification

Future Enhancements

The Nexus roadmap includes several planned improvements:

Light-weight mode: Return Homeserver URIs instead of full content
Federation protocols: Inter-Nexus communication for global discovery
Advanced ML models: Improved recommendation algorithms
Real-time WebSocket API: Push-based updates for clients
Content delivery optimization: Edge caching and CDN integration
Enhanced privacy controls: Encrypted graph operations

Contributing

Nexus is open source and welcomes contributions:

Fork the repository
Create a feature branch
Write tests for new functionality
Submit a pull request with clear description

All contributions should include tests and benchmarks where applicable.

Resources

Repository: https://github.com/pubky/pubky-nexus
Swagger UI (Staging): https://nexus.staging.pubky.app/swagger-ui/
Swagger UI (Production): https://nexus.pubky.app/swagger-ui/
Issue Tracker: https://github.com/pubky/pubky-nexus/issues

Building Clients for Nexus

If you’re building a social client application to consume the Nexus API:

Data Model Specification: Use pubky-app-specs as your authoritative reference for data structures and validation rules
Web App: pubky.app (github.com/pubky/pubky-app) — the production reference implementation
API Exploration: Use the Swagger UI to explore available endpoints and test queries