Appearance

Architecture Overview ​

Last Updated: March 2026

Scrapalot is an open-source, enterprise-grade RAG platform built with modern, scalable microservices architecture for intelligent document processing and AI-powered knowledge retrieval.

For Open Source Users

This architecture is 100% open source (MIT License). You can self-host everything on your own infrastructure - no vendor lock-in, no hidden costs.

System Overview ​

Scrapalot uses a modern microservices architecture with clear separation of concerns:

Request Flow ​

User asks a question:

  1. React UI sends request to Gateway (port 8080)
  2. Gateway routes to Kotlin Backend (port 8091)
  3. Kotlin authenticates, validates, retrieves user context
  4. Kotlin calls Python Chat via gRPC (port 9091) with user IDs
  5. Python performs AI operations (RAG, LLM, embeddings)
  6. Results stream back through the chain to user

Core Components ​

Web Interface ​

Modern, responsive React application

Features:

  • Real-time streaming answers
  • Document management
  • Multi-language support (English, Croatian)
  • Dark/light themes
  • Mobile-friendly

Technology:

  • React 18.3.1 with TypeScript 5.9.3
  • Tailwind CSS + Shadcn/ui (178 components)
  • Real-time STOMP WebSocket updates
  • Vite 5.4.1 for ultra-fast builds
  • i18next (en, hr translations)

API Gateway ​

Spring Cloud Gateway routing layer

Capabilities:

  • Single entry point for all UI requests
  • Routes to appropriate backend services
  • Load balancing and circuit breaking
  • Request/response transformation

Technology:

  • Spring Cloud Gateway
  • Port 8080
  • Routes: /api/v1/** → Kotlin Backend

Kotlin Backend ​

User data and business logic service

Capabilities:

  • User authentication (JWT, OAuth, Sessions)
  • All user data management (users, workspaces, collections)
  • Document metadata and chat sessions
  • Background jobs and API key management
  • Calls Python AI service via gRPC when needed

Technology:

  • Kotlin 2.1.0 + Spring Boot 3.4.1
  • PostgreSQL (local, port 5432)
  • gRPC client (calls Python on port 9091)
  • Liquibase migrations
  • MapStruct for DTOs

Python Chat Service ​

Pure AI/ML operations service

Capabilities:

  • Advanced RAG strategies (17 strategies + 11 orchestrators)
  • Multi-provider AI model support
  • Document processing and embeddings
  • Vector search with pgvector
  • Knowledge graph (Neo4j)
  • Deep Research (5-phase architecture)

Technology:

  • Python 3.12.8 + FastAPI
  • PostgreSQL with pgvector
  • LangChain 1.2.0 (RAG framework)
  • Pydantic AI 1.1.0 (agent framework)
  • Edge-TTS 7.2.7 (text-to-speech)
  • gRPC server (port 9091)

Data Storage ​

Flexible, powerful databases

PostgreSQL with pgvector (port 5432):

  • Two databases: scrapalot_backend (Kotlin), scrapalot (Python)
  • Vector search capabilities for RAG
  • Local Docker container deployment
  • Row Level Security (RLS) for multi-tenancy

Redis (port 6379):

  • DB 0: Python cache
  • DB 1: Kotlin cache
  • Redis Streams SAGA for cross-service data sync

Neo4j (port 7687, optional):

  • Knowledge graph storage
  • Entity and relationship extraction
  • Cross-document connection discovery

Benefits:

  • Scalable to millions of documents
  • Fast semantic search with pgvector
  • Secure multi-tenant isolation
  • Real-time event propagation

AI Models ​

Your choice of providers

Cloud Options:

  • OpenAI (GPT-4, GPT-3.5)
  • Anthropic (Claude)
  • Google (Gemini)

Local Options:

  • Ollama (easy local server)
  • LM Studio (GPU-accelerated)
  • Direct GGUF models
  • vLLM (production inference)

Flexibility:

  • Mix and match providers
  • Change models anytime
  • No vendor lock-in

RAG Engine: How Search Works ​

Scrapalot uses Tri-Modal Fusion - three search methods working together:

Why Three Search Methods? ​

Semantic Search (Vector embeddings):

  • Understands conceptual similarity
  • "renewable energy" finds "solar power"
  • Best for: Conceptual questions

Keyword Search (BM25):

  • Exact term matching
  • "error 221" finds exactly that
  • Best for: Technical terms, codes, specific phrases

Graph Search (Neo4j, optional):

  • Understands relationships
  • "How are X and Y connected?"
  • Best for: Relationship questions

Intelligent Routing: The system automatically chooses the best method(s) for each question. You don't need to think about it - it just works.

How Documents Become Knowledge ​

Processing steps:

  1. Upload - You upload document
  2. Extract - Text extracted from PDF/Word/etc
  3. Chunk - Intelligently split into searchable segments
  4. Embed - Generate vector embeddings
  5. Index - Store in database with metadata
  6. Ready - Available for search immediately

Processing time:

  • Small doc (10 pages): ~30 seconds
  • Medium doc (100 pages): ~2 minutes
  • Large doc (500 pages): ~10 minutes

Security & Privacy ​

Data Protection ​

Multi-tenant isolation:

  • Your data completely separate from others
  • Database-level security (Row Level Security)
  • Workspace-based access control
  • Cannot be bypassed

Encryption:

  • All connections encrypted (TLS/SSL)
  • API keys encrypted at rest
  • Secure credential storage

Access control:

  • JWT authentication
  • Role-based permissions
  • Workspace sharing controls
  • Audit logging

Privacy Options ​

Cloud deployment:

  • Use managed services or self-hosted Docker
  • Your data isolated in your account
  • Encrypted in transit and at rest

Self-hosted:

  • Complete data sovereignty
  • Never leaves your infrastructure
  • Full control over everything
  • Local AI models for zero external calls

Deployment Options ​

Perfect for getting started:

  • Single server deployment
  • Self-hosted PostgreSQL with pgvector (Docker)
  • Cloud AI models (OpenAI, Claude)
  • 10 minutes to running

Requirements:

  • 4GB RAM minimum
  • Docker installed
  • Internet connection

Production Deployment ​

For serious use:

  • Load-balanced API servers
  • Database with replicas
  • Redis caching layer
  • Background worker pool
  • Monitoring and logging

Scaling:

  • Horizontal API scaling
  • Worker pool sizing
  • Read replicas for database
  • CDN for frontend

Privacy-First Deployment ​

For maximum data control:

  • Self-hosted infrastructure
  • Local AI models only
  • Air-gapped if needed
  • Complete audit trail

Technology Stack ​

Frontend (scrapalot-ui) ​

  • React 18.3.1 + TypeScript 5.9.3
  • Vite 5.4.1 (ultra-fast builds)
  • Tailwind CSS + Shadcn/ui (178 components)
  • STOMP WebSocket for real-time updates
  • i18next (English, Croatian)
  • TipTap 2.27.1 (collaborative editor)
  • Framer Motion 12.23.24 (animations)

Gateway (scrapalot-gw) ​

  • Spring Cloud Gateway
  • Kotlin 2.1.0
  • Port 8080 (single entry point)

Kotlin Backend (scrapalot-backend) ​

  • Kotlin 2.1.0 + Spring Boot 3.4.1
  • PostgreSQL (local Docker)
  • Liquibase (migrations)
  • MapStruct (DTO mapping)
  • gRPC client
  • Redis (caching)

Python Chat (scrapalot-chat) ​

  • Python 3.12.8 + FastAPI
  • SQLAlchemy ORM
  • LangChain 1.2.0 (RAG framework)
  • Pydantic AI 1.1.0 (agent framework)
  • llama-cpp-python (local models)
  • Edge-TTS 7.2.7 (text-to-speech)
  • gRPC server (port 9091)

Databases ​

  • PostgreSQL 18 with pgvector (two databases)
  • Redis 7.x (caching + Redis Streams SAGA)
  • Neo4j (optional, for Graph RAG)

AI Providers (all optional) ​

  • OpenAI, Anthropic, Google Gemini
  • Ollama, vLLM, LM Studio, LlamaCPP
  • DeepSeek, OpenRouter, Qwen
  • Any OpenAI-compatible endpoint

What Makes Scrapalot Different ​

Most RAG systems use only vector search. Scrapalot combines three methods for superior accuracy.

Intelligent Routing ​

AI automatically selects the best search strategy for each question. No manual configuration needed.

Context Expansion ​

Understands document structure to provide complete context, not just isolated chunks.

Model Flexibility ​

Use any AI model - cloud, local, or mixed. Switch anytime without data migration.

True Open Source ​

MIT licensed, self-host everything, no vendor lock-in, active community.

Production Ready ​

Enterprise security, multi-tenancy, real-time streaming, comprehensive monitoring.

Next Steps ​

Explore More

Get Started:

Learn the Features:

Advanced Topics:

Scrapalot makes advanced RAG accessible. Complex technology, simple experience.

Released under the MIT License.

Copyright © 2026 Scrapalot