API Reference

Last Updated: September 2026

REST API reference for Scrapalot, as reached through the API Gateway, with real request/response shapes.

One public entry point

All API calls go through the Gateway (https://api.scrapalot.app in the cloud, port 8080 when self-hosted). The gateway validates your credentials, applies rate limits and routes the request to the right internal service — you never call the internal services directly.

The gateway also serves an interactive Swagger UI at its root (http://localhost:8080/ when self-hosted) that combines the backend, AI and gateway OpenAPI specs with "Try it out" support.

Quick Start

All endpoints except login/registration require authentication.

Base URL

Where you run Scrapalot	Base URL
Hosted cloud	`https://api.scrapalot.app/api/v1`
Self-hosted	`http://localhost:8080/api/v1`

The path is /api/v1, not /v1 — the OpenAI-compatible chat endpoint is /api/v1/chat/completions. Examples below use the self-hosted URL; swap in the cloud base URL if you are on the hosted product.

Authentication headers (either one):

bash

# Option 1: Bearer JWT (from login)
Authorization: Bearer <access_token>

# Option 2: API key (for integrations) — either header works
X-API-Key: scp-1a2b3c4d-0123456789abcdef
Authorization: Bearer scp-1a2b3c4d-0123456789abcdef

JSON conventions: request and response bodies use snake_case field names. Query-string parameters are listed per endpoint exactly as the server expects them.

Authentication

Authentication Flow

Log In

Endpoint: POST /auth/login

bash

curl -X POST http://localhost:8080/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "username_or_email": "user@example.com",
    "password": "your-password"
  }'

Response:

json

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expires_in": 14400,
  "token_type": "bearer"
}

expires_in is in seconds (access tokens last 4 hours by default). The refresh token is also set as an HTTP-only cookie for browser clients. Invalid credentials return 401.

Refresh, Log Out, Register

Method	Path	Purpose
`POST`	`/users/token/refresh`	Exchange a refresh token (body `{"refresh_token": "…"}` or the cookie) for a new token pair. Refresh tokens rotate on every use.
`POST`	`/users/token/logout`	Revoke the current refresh token and clear the auth cookies
`POST`	`/users/token/logout-all`	Revoke every session of the current user
`POST`	`/users/register`	Create an account (`username`, `email`, `password` 8–128 chars, optional `first_name`, `last_name`, `license_agreement_consent`)
`GET`	`/users/me`	The current user's profile

API Keys

Long-lived keys for scripts, integrations and MCP clients. Creating a key requires the api_access plan feature (Pro plan and above); existing keys stay listable and revocable after a downgrade.

Method	Path	Purpose
`POST`	`/auth/api-keys`	Create a key
`GET`	`/auth/api-keys`	List your keys
`PATCH`	`/auth/api-keys/{key_id}/toggle`	Enable / disable a key
`DELETE`	`/auth/api-keys/{key_id}`	Delete a key (`204`)

Create request:

bash

curl -X POST http://localhost:8080/api/v1/auth/api-keys \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "production-integration",
    "expires_at": "2026-12-31T00:00:00Z"
  }'

expires_at (ISO-8601) is optional — omit it for a key that does not expire.

Response (201 Created):

json

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "name": "production-integration",
  "key_prefix": "scp-1a2b",
  "plain_text_key": "scp-1a2b3c4d-0123456789abcdef",
  "is_active": true,
  "expires_at": "2026-12-31T00:00:00Z",
  "created_at": "2026-09-18T10:30:00Z",
  "last_used_at": null
}

Store securely

plain_text_key is returned only in the create response. Scrapalot stores just a hash of the key and cannot show it again. List responses carry key_prefix only.

Workspaces

Method	Path	Purpose
`GET`	`/workspaces?page=1&page_size=20`	Workspaces you own or that were shared with you
`GET`	`/workspaces/default`	Your default workspace
`GET`	`/workspaces/{workspace_id}`	One workspace
`GET`	`/workspaces/{workspace_id}/my-role`	Your permission in it
`POST`	`/workspaces`	Create (`{"name": "…", "description": "…"}`)
`PUT`	`/workspaces/{workspace_id}`	Update (edit permission required)
`DELETE`	`/workspaces/{workspace_id}`	Delete (owner only)
`GET`	`/workspaces/{workspace_id}/users`	Members
`POST`	`/workspaces/{workspace_id}/share`	Share with a user (owner only): `{"user_id": "…", "permission": "read" \| "write" \| "admin"}`
`PUT`	`/workspaces/{workspace_id}/share/{user_id}`	Change a member's permission (owner only)
`DELETE`	`/workspaces/{workspace_id}/users/{user_id}`	Remove a member (owner only)

Sharing a workspace requires the shared_workspaces plan feature.

Collections

Collections organize documents into knowledge stacks and can be nested.

List Collections

Endpoint: GET /collections

Query parameters:

workspaceId (UUID, optional) — limit to one workspace; otherwise all accessible workspaces
page (integer, default 1)
limit (integer, default 20)
sort_by (default name), sort_order (asc | desc)

GET /collections/workspace/{workspace_id} is the same list scoped by path.

Response:

json

{
  "collections": [
    {
      "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "name": "Research Papers",
      "slug": "research-papers",
      "description": null,
      "workspace_id": "0f1e2d3c-4b5a-6978-8796-a5b4c3d2e1f0",
      "parent_collection_id": null,
      "chunking_strategy": "recursive",
      "chunk_size": 1000,
      "chunk_overlap": 200,
      "processing_status": "completed",
      "graph_tier": null,
      "created_at": "2026-09-01T10:00:00Z",
      "updated_at": "2026-09-18T14:30:00Z"
    }
  ],
  "pagination": { "page": 1, "limit": 20, "total": 42, "has_more": true }
}

The slug is what you use in the OpenAI-compatible model field (scrapalot:<workspace-slug>:<collection-slug>).

Create Collection

Endpoint: POST /collections (JSON body; edit permission on the workspace required)

bash

curl -X POST http://localhost:8080/api/v1/collections \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Research",
    "workspace_id": "0f1e2d3c-4b5a-6978-8796-a5b4c3d2e1f0",
    "chunking_strategy": "recursive",
    "chunk_size": 1000,
    "chunk_overlap": 200
  }'

Field	Default	Notes
`name`	required	1–100 characters
`workspace_id`	required
`description`	`null`	up to 2000 characters
`chunking_strategy`	`recursive`	`recursive`, `semantic`, `sentence`, `paragraph`, `fixed`
`chunk_size`	`1000`	100–10000
`chunk_overlap`	`200`	0–1000
`parent_collection_id`	`null`	nest under another collection
`graph_tier`	inherit	knowledge-graph build tier: `0` none, `1` light, `2` full

Returns 201 Created with the collection object.

Other Collection Endpoints

Method	Path	Purpose
`GET`	`/collections/{collection_id}`	One collection
`PUT`	`/collections/{collection_id}`	Update name, description, chunking, `custom_instructions`, `graph_tier`
`POST`	`/collections/{collection_id}/move`	Re-parent (`{"parent_collection_id": …}`) or move to another workspace (`{"workspace_id": …}`)
`DELETE`	`/collections/{collection_id}`	Delete (`204`)
`GET`	`/collections/{collection_id}/summary`	Collection summary

Documents

Upload, manage, and retrieve documents.

Document Upload Flow

Upload Document

Endpoint: POST /documents/upload (multipart; edit permission on the collection required)

bash

curl -X POST http://localhost:8080/api/v1/documents/upload \
  -H "Authorization: Bearer <access_token>" \
  -F "file=@document.pdf" \
  -F "collection_id=a1b2c3d4-e5f6-7890-abcd-ef1234567890"

Form field	Default	Notes
`file`	required
`collection_id`	required
`auto_process`	`true`	queue parsing/embedding immediately
`build_graph`	`false`	also build the knowledge graph
`generate_summary`	`false`	generate a document summary
`store_file`	`true`	keep the original file

Response:

json

{
  "success": true,
  "document_id": "d1234567-e89b-12d3-a456-426614174000",
  "job_id": "j1234567-e89b-12d3-a456-426614174000",
  "message": "…"
}

POST /documents/upload_stream accepts the same form fields (with auto_process defaulting to false) and answers with newline-delimited JSON status lines for the upload phase. Processing progress after that is reported by the job endpoint below.

Get Processing Status

Endpoint: GET /documents/processing_status/{job_id}

json

{
  "job_id": "j1234567-e89b-12d3-a456-426614174000",
  "document_id": "d1234567-e89b-12d3-a456-426614174000",
  "filename": "ai_research.pdf",
  "collection_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "collection_name": "Research Papers",
  "status": "processing",
  "progress": 65,
  "message": "…",
  "last_update_time": "…",
  "estimated_completion_time": "…"
}

progress is a percentage (0–100). A job that finished and aged out returns 404 — stop polling at that point.

List and Manage Documents

Method	Path	Purpose
`GET`	`/documents/collection/{collection_id}`	List documents (`page`, `page_size`, `search`, `sort_by` = `date`\|`title`\|`size`\|`status`, `sort_dir`). Returns `{"documents": [...], "hasMore": …, "total": …}`
`GET`	`/documents/{document_id}`	Document metadata
`GET`	`/documents/{document_id}/file`	Download the original file
`GET`	`/documents/{document_id}/summary`	Document summary
`POST`	`/documents/reprocess/{document_id}`	Re-run processing
`DELETE`	`/documents/{document_id}`	Delete (`204`)
`POST`	`/documents/{document_id}/restore`	Restore a deleted document from the trash

Chat & Queries

Ask questions with RAG, deep research, or direct chat.

RAG Query Flow

Chat Completions

Scrapalot exposes a single chat endpoint that is wire-compatible with the OpenAI POST /v1/chat/completions API — vanilla openai SDK clients work when pointed at the /api/v1 base. The full Scrapalot feature set (deep research, agentic, web search, tutor, attachments, mentions, …) is opt-in via a top-level scrapalot extras block on the request.

Endpoint: POST /api/v1/chat/completions

Request (basic RAG):

bash

curl -X POST http://localhost:8080/api/v1/chat/completions \
  -H "Authorization: Bearer <access_token_or_scp_api_key>" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "scrapalot:my-workspace:my-collection",
    "messages": [{"role": "user", "content": "What are the key findings?"}],
    "stream": true,
    "scrapalot": {
      "similarity_threshold": 0.5,
      "top_k": 15
    }
  }' \
  --no-buffer

Top-level fields (OpenAI envelope):

Parameter	Type	Default	Description
`model`	string	required	`scrapalot:<workspace-slug>[:<collection-slug>]` for routing, or any other string to fall back to your default workspace
`messages`	array	required	OpenAI chat-message array (plain string `content`); the last `user` message is the question
`stream`	boolean	`false`	When `true`, returns SSE `chat.completion.chunk` events

scrapalot extras block (all optional):

Parameter	Type	Default	Description
`mode`	string	`rag`	One of `rag`, `direct`, `deep_research`, `agentic`, `web_search`, `tutor`, `thought_partner`, `document_qa`
`web_search_enabled`	boolean	`false`	Add web search on top of any mode
`collection_ids`	UUID[]	from slug	Collections to search
`document_ids`	UUID[]	`[]`	Specific documents to scope retrieval (pass `collection_ids` too)
`workspace_id`	UUID	from slug	Workspace override
`similarity_threshold`	float	`0.5`	RAG relevance cutoff (0.0–1.0)
`top_k`	integer	`15`	Number of chunks to retrieve
`research_breadth`	integer	`4`	Deep research: sources per step
`research_depth`	integer	`2`	Deep research: depth
`approved_plan_id`	string	`null`	Deep research: run a plan you approved from a `plan_preview`
`clarification_answers`	array	`[]`	Deep research: answers to `clarification_questions`
`attachments`	array	`[]`	Inline files / images / YouTube
`mentions`	array	`[]`	@-mentioned documents / collections
`annotation_color_filter`	string[]	`[]`	Hex colors to filter retrieval to user-highlighted pages
`prompt_template_name`	string	`null`	Settings → Prompts → Custom Templates picker
`language`	string	`en`	Response language hint
`incognito`	boolean	`false`	Neither read nor update your personal memory for this turn
`session_id`	string	`null`	Alternative to the `Conversation-Id` header

The Python openai SDK passes the block via extra_body:

python

from openai import OpenAI

client = OpenAI(base_url="https://api.scrapalot.app/api/v1", api_key="scp-...")
client.chat.completions.create(
    model="scrapalot:my-workspace",
    messages=[{"role": "user", "content": "…"}],
    stream=True,
    extra_body={"scrapalot": {"mode": "deep_research", "research_breadth": 4}},
)

Authentication: Authorization: Bearer <token> accepts either a JWT issued by /auth/login or a Scrapalot API key (scp-…). The X-API-Key: scp-… header is also accepted.

Conversation continuity: send the same session UUID in the Conversation-Id HTTP header on every follow-up request. Without it, each request starts a new chat session.

Response — streaming SSE:

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1758700000,"model":"scrapalot:my-workspace:my-collection","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk",…,"choices":[{"index":0,"delta":{"content":"The key findings"}}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk",…,"choices":[{"index":0,"delta":{"scrapalot":{"type":"citation_info","citation_num":1,"document_id":"d1234567-…","document_title":"AI Research Paper","page":5,"score":0.92}}}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk",…,"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Answer tokens (message_delta packets) arrive in delta.content. Every other Scrapalot packet — status, citation_info, strategy_transparency, plan_preview, clarification_questions, research_report, chart_data, stream_end, … — is relayed verbatim under delta.scrapalot on its own chunk. Vanilla OpenAI clients ignore unknown fields and see only the token stream. The full packet catalogue is in Streaming Protocol.

Response — non-streaming (stream: false): a single chat.completion JSON with the accumulated answer in choices[0].message.content.

Stop a running answer: POST /chat/sessions/{session_id}/stop — the partial answer is kept.

Deep Research

Enable multi-source research with scrapalot.mode = "deep_research". Deep research is plan-gated (not every subscription tier includes it).

bash

curl -X POST http://localhost:8080/api/v1/chat/completions \
  -H "Authorization: Bearer <access_token_or_scp_api_key>" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "scrapalot:my-workspace",
    "messages": [{"role": "user", "content": "Comprehensive analysis of AI safety research"}],
    "stream": true,
    "scrapalot": { "mode": "deep_research", "research_depth": 2, "research_breadth": 4 }
  }' \
  --no-buffer

A research run is interactive. The first stream typically ends with clarification_questions (answer them via clarification_answers) or a plan_preview (send its plan_id back as approved_plan_id to execute it). The executing run emits research packets under delta.scrapalot, for example:

json

{"type":"planning_progress","stage":"structuring_plan","progress":0.5,"message":""}
{"type":"plan_preview","plan_id":"…","title":"…","sections":[{"title":"Background","question_count":3}],"total_questions":12}
{"type":"research_report","plan_id":"…","title":"…","executive_summary":"…","full_report_markdown":"…","total_sources":25}
{"type":"stream_end","reason":"completed"}

Research endpoints (runs can outlive the HTTP request):

Method	Path	Purpose
`GET`	`/research/active`	Your running research, if any
`GET`	`/research/{plan_id}/stream?from_id=`	Re-attach to a running research as SSE (each event carries a resumable `id:`)
`GET`	`/research/by-plan/{plan_id}`	A research result by plan
`GET`	`/research/session/{session_id}`	Research runs in a chat session
`POST`	`/research/cancel`	Cancel: `{"research_id": "<plan_id>"}`
`GET`	`/research/templates`	Available research templates
`POST`	`/research/export`	Export a report (DOCX / HTML)

OpenAI-Compatible Model List

Method	Path	Purpose
`GET`	`/models`	Your workspaces and collections as OpenAI "models" (`scrapalot:<workspace>[:<collection>]`)
`GET`	`/models/{id}`	One model entry

Sessions and Messages

Method	Path	Purpose
`GET`	`/sessions?page=0&pageSize=50`	Your chat sessions (`collectionId`, `archived` filters)
`GET`	`/sessions/{session_id}`	One session
`PUT`	`/sessions/{session_id}`	Rename / update
`DELETE`	`/sessions/{session_id}`	Delete
`GET`	`/sessions/search`	Search sessions
`GET`	`/messages?sessionId=…`	Messages of a session (`page`, `pageSize`, `order`)
`PUT`	`/messages/{message_id}/feedback`	Rate an answer

Notes

Method	Path	Purpose
`GET`	`/notes?workspaceId=…`	Your notes (optionally one workspace)
`GET`	`/notes/search?workspaceId=…&query=…`	Search notes
`GET`	`/notes/{note_id}`	One note
`POST`	`/notes`	Create: `{"title": "…", "content": "…", "workspace_id": "…", "note_type": "markdown", "tags": []}`
`PUT`	`/notes/{note_id}`	Update
`DELETE`	`/notes/{note_id}`	Delete
`GET`	`/notes/{note_id}/versions`	Version history
`POST`	`/notes/{note_id}/versions/{version_id}/restore`	Restore a version
`POST`	`/notes/{note_id}/share`	Share: `{"user_id": "…", "permission": "read" \| "write" \| "owner"}`
`GET` / `POST`	`/notes/{note_id}/comments`	Read / add comments

Models

List Models

Endpoint: GET /llm-inference/list-models

Query parameters:

providers — repeat to filter by provider type
modelType — e.g. NORMAL, EMBEDDING, VISION
search — name filter
page (default 1), limit (default 50)
refresh — true to re-fetch from the providers

The response groups the available models by configured provider.

List Embedding Models

Endpoint: GET /llm-inference/embedding-models

Error Responses

Backend errors use one JSON shape:

json

{
  "status": 404,
  "error": "Not Found",
  "message": "Collection not found: a1b2c3d4-…",
  "path": "/api/v1/collections/a1b2c3d4-…",
  "timestamp": "2026-09-18T10:30:00Z",
  "field_errors": []
}

Status	Typical cause
`400`	Validation failed (`field_errors` lists each field) or malformed input
`401`	Missing, expired or invalid token / API key
`403`	No permission on the workspace or collection, or a feature not included in your plan
`404`	Resource not found or not accessible
`409`	Operation not allowed in the resource's current state
`429`	Rate limit exceeded
`503`	An internal service is temporarily unavailable — retry

The OpenAI-compatible endpoints return OpenAI-shaped errors instead: {"error": {"message": "…", "type": "…"}}.

Rate Limits

AI routes (/chat/** — including chat completions and deep-research runs — /llm-inference/** and the other AI endpoints) are rate-limited per user and per subscription tier over a one-second window. Research status, streaming and cancel calls are not rate-limited. The limits are set by the operator; the reference deployment uses:

Tier	AI requests per second
Free	3
Researcher	10
Pro	50
Team, Enterprise	unlimited

A limited response is 429 Too Many Requests and carries:

X-RateLimit-Limit: 10
X-RateLimit-Remaining: 0

Storage, document counts and feature access are governed by your subscription plan rather than request rate limits.

Pagination

Pagination parameters differ slightly between endpoints — check each table above:

Endpoint	Parameters	Response
`/collections`	`page` (from 1), `limit`	`pagination: {page, limit, total, has_more}`
`/workspaces`	`page` (from 1), `page_size`	paginated wrapper
`/documents/collection/{id}`	`page` (from 1), `page_size`	`total`, `hasMore`
`/sessions`, `/messages`	`page` (from 0), `pageSize`	paginated wrapper

Best Practices

Authentication

Use API keys for server-to-server integrations and MCP clients; name one key per client so you can revoke it alone
Keep keys and tokens in environment variables or a secret store, never in version control
Refresh the access token before it expires instead of logging in again

Uploads

Poll /documents/processing_status/{job_id} (or watch the job in the app) until status is completed or failed
Check your plan's storage quota before bulk uploads

RAG queries

Start with the defaults (similarity_threshold: 0.5, top_k: 15)
Use mode: "agentic" when you want Scrapalot to pick the retrieval strategy
Use deep research for complex multi-source questions, not simple lookups

Streaming

Use --no-buffer with curl and parse the SSE stream line by line
Treat unknown delta.scrapalot.type values as optional — new packet types are added over time
Handle stream_end with a non-completed reason (error, cancelled, clarification_needed, plan_preview_ready, background_job_dispatched)

Streaming Protocol - Packet types and stream format
Security - Authentication details
MCP - Using Scrapalot from Claude Code and other MCP clients
Deep Research - Research features
RAG Strategy - Search strategies

API Reference ​

Quick Start ​

Authentication ​

Authentication Flow ​

Log In ​

Refresh, Log Out, Register ​

API Keys ​

Workspaces ​

Collections ​

List Collections ​

Create Collection ​

Other Collection Endpoints ​

Documents ​

Document Upload Flow ​

Upload Document ​

Get Processing Status ​

List and Manage Documents ​

Chat & Queries ​

RAG Query Flow ​

Chat Completions ​

Deep Research ​

OpenAI-Compatible Model List ​

Sessions and Messages ​

Notes ​

Models ​

List Models ​

List Embedding Models ​

Error Responses ​

Rate Limits ​

Pagination ​

Best Practices ​

Related Documentation ​

API Reference

Quick Start

Authentication

Authentication Flow

Log In

Refresh, Log Out, Register

API Keys

Workspaces

Collections

List Collections

Create Collection

Other Collection Endpoints

Documents

Document Upload Flow

Upload Document

Get Processing Status

List and Manage Documents

Chat & Queries

RAG Query Flow

Chat Completions

Deep Research

OpenAI-Compatible Model List

Sessions and Messages

Notes

Models

List Models

List Embedding Models

Error Responses

Rate Limits

Pagination

Best Practices

Related Documentation