Real-Time Communication & Progress Tracking
Last Updated: September 2026
This document explains how Scrapalot keeps you informed with live updates during document processing, AI responses, collaboration and other background operations.
Overview
Scrapalot uses real-time communication technology to provide instant feedback as your documents are processed, AI generates responses, and background tasks complete. You'll never have to refresh the page or wonder about progress - everything updates automatically.
What You'll Experience
Live Progress Updates
Watch your operations progress in real-time:
- Document Upload & Processing: See each stage as your documents are processed
- AI Chat Responses: Watch AI responses stream in word-by-word
- Deep Research: Follow a research run's progress, even one running in the background
- Background Jobs: Get notified when long-running tasks complete
- Team collaboration: Workspace chat, direct messages, huddles and project rooms update live
- Notes: Several people can edit the same note at once
Automatic Reconnection
The system maintains reliable connections even when:
- Your internet connection drops briefly
- You switch between WiFi networks
- Your computer goes to sleep and wakes up
- The server briefly restarts for updates
The client reconnects automatically with a short back-off. If the server keeps rejecting the connection (for example because your session expired), it stops retrying and tries again once you sign in or return to the tab.
Document Processing Progress
When you upload documents, you'll see detailed progress through each stage. The percentages below are approximate — the exact points differ per file format (PDF, EPUB, Word, spreadsheets):
Progress Stages
- Upload (0-10%): Your file is being uploaded to Scrapalot
- Validation (10-20%): Checking file format and content
- Processing (20-50%): Extracting text and structure
- Chunking (50-70%): Breaking content into searchable segments
- Embedding (70-95%): Creating AI-powered search vectors
- Indexing (95-100%): Making your document searchable
Visual Indicators
- Progress Bar: Shows overall completion percentage
- Stage Name: Displays current operation
- Success/Error Messages: Clear status updates
Chat Streaming
AI responses stream to you in real-time over the chat request itself (an HTTP stream through the Gateway and backend), not over the WebSocket:
Streaming Features
- Word-by-Word Display: See the response as it's generated
- Stop Generation: Cancel anytime if you have enough information
- Progress Indicators: Visual feedback while AI is thinking
- Citations: Source references appear inline as they're discovered
Background Job Notifications
Long-running operations notify you when complete:
- Processing Status: See which documents are still processing
- Completion Alerts: Get notified when jobs finish
- Error Notifications: Immediate alerts if something goes wrong
Connection Status
The app keeps you informed about your connection:
Connection States
- 🟢 Connected: Everything working normally
- 🟡 Reconnecting: Attempting to restore connection
- 🔴 Disconnected: No connection - check your internet
What Happens During Disconnection
Automatic Recovery:
- System detects connection loss
- Attempts reconnection automatically
- Re-subscribes to your channels
- Restores all active progress tracking when reconnected
Your Data is Safe:
- Operations continue on the server
- Progress updates resume when reconnected
- No data is lost during brief disconnections
Real-Time Channels
Scrapalot uses three real-time connections, each for a different job. All of them go through the Gateway and require a valid login.
| Connection | Served by | Carries |
|---|---|---|
| STOMP (AI) | Python AI service | Document job progress, deep research progress, workspace update notices |
| STOMP (collaboration) | Kotlin backend | Workspace chat (messages, typing, presence, reactions, AI typing), direct messages, mentions, huddles, project rooms |
| Y.js WebSocket | Python AI service | Real-time collaborative editing of notes |
Chat answers themselves are not sent over any of these — they stream back on the HTTP request that asked the question.
Progress Tracking Reliability
Intelligent Updates
The system optimizes update frequency:
- Frequent Updates: During critical stages (upload, embedding)
- Throttled Updates: Prevents overwhelming with too many notifications
- Final Confirmation: Always confirms 100% completion
Persistent Tracking
Dialog Reopening: Close and reopen the upload dialog - your progress is preserved
Page Refresh: Refresh the page - active jobs continue tracking
Session Persistence: Progress survives across browser sessions
Privacy & Security
Connection Security
- Authenticated Connections: All WebSocket connections require valid login
- Token-Based Auth: Secure JWT tokens validate every connection
- Encrypted Transport: All data encrypted in transit
Data Privacy
- Job Isolation: You only see updates for your own operations
- User Verification: Server verifies ownership before sending updates
- No Cross-Contamination: Strict user separation in all notifications
Best Practices
For Optimal Experience
- Stable Connection: Use reliable WiFi or ethernet for large uploads
- Keep Tab Active: Browser may throttle background tabs
- Allow Notifications: Enable browser notifications for completion alerts
- Monitor Progress: Watch progress for large documents to catch errors early
Troubleshooting Tips
Slow Progress Updates:
- Check your internet speed
- Close unnecessary browser tabs
- Ensure server isn't overloaded (check status page)
Missing Progress Updates:
- Refresh the page to reconnect
- Check browser console for connection errors
- Verify you're still logged in
Disconnection Issues:
- Check firewall/proxy settings
- Try different network connection
- Contact support if persistent
Related Features
- Document Processing: Learn about what happens during document processing
- Background Workers: Where document and research jobs run
- FAQ: Common questions about real-time features
Technical Note: Job and research progress use a STOMP WebSocket served by the Python AI service; workspace chat, direct messages, huddles and project rooms use a separate STOMP WebSocket served by the Kotlin backend; notes collaboration uses Y.js. Chat answers stream over HTTP. Most REST traffic goes through the Gateway to the Kotlin backend, which calls the Python service over gRPC; only a small set of AI-only endpoints and the WebSockets are routed by the Gateway directly to Python.