Real-Time Communication & Progress Tracking
This document explains how Scrapalot keeps you informed with live updates during document processing, AI responses, and other background operations.
Overview
Scrapalot uses real-time communication technology to provide instant feedback as your documents are processed, AI generates responses, and background tasks complete. You'll never have to refresh the page or wonder about progress - everything updates automatically.
What You'll Experience
Live Progress Updates
Watch your operations progress in real-time:
- Document Upload & Processing: See each stage as your documents are processed
- AI Chat Responses: Watch AI responses stream in word-by-word
- Model Downloads: Track download progress when loading new AI models
- Background Jobs: Get notified when long-running tasks complete
Automatic Reconnection
The system maintains reliable connections even when:
- Your internet connection drops briefly
- You switch between WiFi networks
- Your computer goes to sleep and wakes up
- The server briefly restarts for updates
Fallback System: If one connection method fails, the system automatically tries alternatives to keep you connected.
Document Processing Progress
When you upload documents, you'll see detailed progress through each stage:
Progress Stages
- Upload (0-10%): Your file is being uploaded to Scrapalot
- Validation (10-20%): Checking file format and content
- Processing (20-50%): Extracting text and structure
- Chunking (50-70%): Breaking content into searchable segments
- Embedding (70-95%): Creating AI-powered search vectors
- Indexing (95-100%): Making your document searchable
Visual Indicators
- Progress Bar: Shows overall completion percentage
- Stage Name: Displays current operation
- Time Estimate: Approximate time remaining
- Success/Error Messages: Clear status updates
Chat Streaming
AI responses stream to you in real-time, creating a natural conversation flow:
Streaming Features
- Word-by-Word Display: See the response as it's generated
- Stop Generation: Cancel anytime if you have enough information
- Progress Indicators: Visual feedback while AI is thinking
- Citations: Source references appear inline as they're discovered
Background Job Notifications
Long-running operations notify you when complete:
- Processing Status: See which documents are still processing
- Completion Alerts: Get notified when jobs finish
- Error Notifications: Immediate alerts if something goes wrong
- Queue Position: Know how many operations are ahead of yours
Connection Status
The app keeps you informed about your connection:
Connection States
- 🟢 Connected: Everything working normally
- 🟡 Reconnecting: Attempting to restore connection
- 🔴 Disconnected: No connection - check your internet
What Happens During Disconnection
Automatic Recovery:
- System detects connection loss
- Attempts reconnection with increasing intervals
- Switches to fallback connection methods if needed
- Restores all active progress tracking when reconnected
Your Data is Safe:
- Operations continue on the server
- Progress updates resume when reconnected
- No data is lost during brief disconnections
Multi-Protocol Support
Scrapalot uses multiple connection technologies for reliability:
Why This Matters:
- Works in restrictive network environments
- Maintains connection through firewalls
- Adapts to your internet quality
- Provides consistent experience across devices
Progress Tracking Reliability
Intelligent Updates
The system optimizes update frequency:
- Frequent Updates: During critical stages (upload, embedding)
- Throttled Updates: Prevents overwhelming with too many notifications
- Final Confirmation: Always confirms 100% completion
Persistent Tracking
Dialog Reopening: Close and reopen the upload dialog - your progress is preserved
Page Refresh: Refresh the page - active jobs continue tracking
Session Persistence: Progress survives across browser sessions
Privacy & Security
Connection Security
- Authenticated Connections: All WebSocket connections require valid login
- Token-Based Auth: Secure JWT tokens validate every connection
- Encrypted Transport: All data encrypted in transit
Data Privacy
- Job Isolation: You only see updates for your own operations
- User Verification: Server verifies ownership before sending updates
- No Cross-Contamination: Strict user separation in all notifications
Best Practices
For Optimal Experience
- Stable Connection: Use reliable WiFi or ethernet for large uploads
- Keep Tab Active: Browser may throttle background tabs
- Allow Notifications: Enable browser notifications for completion alerts
- Monitor Progress: Watch progress for large documents to catch errors early
Troubleshooting Tips
Slow Progress Updates:
- Check your internet speed
- Close unnecessary browser tabs
- Ensure server isn't overloaded (check status page)
Missing Progress Updates:
- Refresh the page to reconnect
- Check browser console for connection errors
- Verify you're still logged in
Disconnection Issues:
- Check firewall/proxy settings
- Try different network connection
- Contact support if persistent
Related Features
- Document Processing: Learn about what happens during document processing
- Model Management: Understand AI model downloads with progress tracking
- FAQ: Common questions about real-time features
Technical Note: Scrapalot uses industry-standard WebSocket protocols (Socket.IO and STOMP) with automatic fallback to Server-Sent Events (SSE) and HTTP polling for maximum compatibility across all networks and devices.