Documents Endpoints
Upload and manage documents that are automatically chunked, indexed, and made searchable.
Upload Document
Upload and process a document.
Headers
| Header | Required | Description |
|---|---|---|
Authorization |
Yes | Bearer YOUR_JWT_TOKEN |
Content-Type |
Yes | multipart/form-data |
Request Body (multipart/form-data)
| Field | Type | Required | Description |
|---|---|---|---|
file |
file | Yes | Document file to upload |
metadata |
string | No | JSON string with additional metadata |
Supported File Types
| Type | Extensions | Max Size |
|---|---|---|
| Text | .txt |
20MB |
| Markdown | .md |
20MB |
.pdf |
20MB | |
| EPUB | .epub |
20MB |
| HTML | .html, .htm |
20MB |
| JSON | .json |
20MB |
| YAML | .yaml, .yml |
20MB |
| Code | .py, .js, .ts |
20MB |
Planned: Word (.doc, .docx). Word files are not processed yet; uploading one returns 501 SYS_NOT_IMPLEMENTED. Convert to PDF, Markdown or plain text first.
Chunking Strategy
Documents are automatically split into searchable chunks. Approximate behavior:
| Parameter | Value |
|---|---|
| Target chunk size | ~400 characters |
| Maximum chunk size | ~450 characters |
| Chunk overlap | ~50 characters |
Different file types use optimized chunking strategies:
| File Type | Strategy | Notes |
|---|---|---|
| Text | Sentence-based | Splits at sentence boundaries |
| Markdown | Structure-aware | Respects headers, code blocks, tables |
| Page/section-aware | Uses TOC when available | |
| EPUB | Chapter-aware | Preserves chapter boundaries |
| Code | Markdown-style | Keeps code blocks together |
Response
201 Created
{
"status": "success",
"data": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"filename": "python-guide.pdf",
"content_type": "application/pdf",
"size": 1048576,
"chunk_count": 45,
"processing_status": "active",
"embedding_progress": null,
"metadata": {
"project": "documentation",
"version": "2.0",
"content_type": "application/pdf",
"upload_timestamp": "2026-01-06T12:00:00.500274+00:00",
"original_filename": "python-guide.pdf",
"chunk_ids": ["uuid1", "uuid2", "..."],
"message": "Document processed into 45 chunks. Embeddings are being generated.",
"processing_time": 2.34
},
"created_at": "2026-01-06T12:00:00.502632Z",
"updated_at": "2026-01-06T12:00:01.114707Z"
},
"meta": {...}
}
The upload returns as soon as the document is chunked: processing_status is active and embedding_progress is null. Embeddings are generated in the background. Get Document then shows completed and the progress counts. Any metadata you sent with the upload is returned alongside the fields the server adds.
Processing Status
| Status | Description |
|---|---|
uploading |
File is being uploaded |
processing |
Document is being chunked |
active |
Ready for search |
completed |
All embeddings generated |
failed |
Processing failed |
Embedding Progress
Documents are chunked synchronously, but embeddings are generated asynchronously. The embedding_progress field shows:
total: Total chunks createdcompleted: Chunks with embeddingspending: Chunks awaiting embeddingfailed: Chunks that failed embeddingpercent_complete: Completion percentage
Errors
| Status | Code | Description |
|---|---|---|
| 400 | VAL_VALIDATION_FAILED |
Unsupported file type (Invalid document: Unsupported file type: image/png) |
| 403 | AUTH_INSUFFICIENT_PERMISSIONS |
The token does not have the write scope |
| 413 | - | File too large (max 20MB) |
| 501 | SYS_NOT_IMPLEMENTED |
File type not yet supported (Word) |
Example
curl -X POST https://api.penfield.app/api/v2/documents/upload \
-H "Authorization: Bearer $JWT_TOKEN" \
-F "file=@/path/to/document.pdf" \
-F 'metadata={"project": "documentation", "version": "2.0"}'
Get Document
Retrieve document metadata and optionally its chunks.
Path Parameters
| Parameter | Type | Description |
|---|---|---|
document_id |
UUID | Document identifier |
Query Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
include_chunks |
boolean | false |
Include document chunks in response |
Response (without chunks)
{
"status": "success",
"data": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"filename": "python-guide.pdf",
"content_type": "application/pdf",
"size": 1048576,
"chunk_count": 45,
"processing_status": "completed",
"embedding_progress": {
"total": 45,
"completed": 45,
"pending": 0,
"failed": 0,
"percent_complete": 100.0
},
"metadata": {},
"created_at": "2026-01-06T12:00:00.502632Z",
"updated_at": "2026-01-06T12:00:02.502112Z"
},
"meta": {...}
}
Response (with chunks)
{
"status": "success",
"data": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"filename": "python-guide.pdf",
"content_type": "application/pdf",
"size": 1048576,
"chunk_count": 45,
"processing_status": "completed",
"embedding_progress": {
"total": 45,
"completed": 45,
"pending": 0,
"failed": 0,
"percent_complete": 100.0
},
"metadata": {},
"created_at": "2026-01-06T12:00:00.502632Z",
"updated_at": "2026-01-06T12:00:02.502112Z",
"chunks": [
{
"id": "chunk-uuid-1",
"document_id": "550e8400-e29b-41d4-a716-446655440000",
"chunk_index": 0,
"content": "Chapter 1: Introduction to Python...",
"start_char": 0,
"end_char": 399,
"metadata": {
"start_char": 0,
"end_char": 399,
"chunk_size": 399,
"filename": "python-guide.pdf",
"document_id": "550e8400-e29b-41d4-a716-446655440000",
"page_count": 12,
"chunk_strategy": "PDFChunking"
},
"embedding_status": "completed",
"created_at": "2026-01-06T12:00:00.592621Z"
},
{
"id": "chunk-uuid-2",
"document_id": "550e8400-e29b-41d4-a716-446655440000",
"chunk_index": 1,
"content": "Python is a versatile programming language...",
"start_char": 400,
"end_char": 780,
"metadata": {...},
"embedding_status": "completed",
"created_at": "2026-01-06T12:00:00.592621Z"
}
]
},
"meta": {...}
}
Errors
| Status | Code | Description |
|---|---|---|
| 404 | RES_NOT_FOUND |
Document not found |
List Documents
List all documents for the tenant.
Query Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
page |
integer | 1 |
Page number |
per_page |
integer | 20 |
Items per page (1-100) |
Response
{
"status": "success",
"data": {
"items": [
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"filename": "python-guide.pdf",
"content_type": "application/pdf",
"size": 1048576,
"chunk_count": 45,
"processing_status": "completed",
"embedding_progress": {
"total": 45,
"completed": 45,
"pending": 0,
"failed": 0,
"percent_complete": 100.0
},
"metadata": {},
"created_at": "2026-01-06T12:00:00.502632Z",
"updated_at": "2026-01-06T12:00:02.502112Z"
}
],
"pagination": {
"page": 1,
"per_page": 20,
"total": 5,
"pages": 1,
"has_next": false,
"has_prev": false
}
},
"meta": {...}
}
Download Document
Download the original uploaded document file.
Path Parameters
| Parameter | Type | Description |
|---|---|---|
document_id |
UUID | Document identifier |
Response
Returns the raw file content with appropriate headers:
Errors
| Status | Code | Description |
|---|---|---|
| 404 | RES_NOT_FOUND |
Document not found or file missing from storage |
Example
curl -X GET https://api.penfield.app/api/v2/documents/550e8400-e29b-41d4-a716-446655440000/download \
-H "Authorization: Bearer $JWT_TOKEN" \
-o downloaded-file.pdf
Delete Document
Permanently delete a document and all its chunks.
Path Parameters
| Parameter | Type | Description |
|---|---|---|
document_id |
UUID | Document identifier |
Response
Notes
- This operation is permanent and cannot be undone
- All associated chunks are deleted
- The original file is removed from storage
- Embeddings are also removed
- Requires the
deletescope
Errors
| Status | Code | Description |
|---|---|---|
| 403 | AUTH_INSUFFICIENT_PERMISSIONS |
The token does not have the delete scope |
| 404 | RES_NOT_FOUND |
Document not found |
Reprocess Document
Re-chunk and re-embed a document. Useful when chunking algorithms are updated.
Status
Not Yet Implemented - Returns 501 SYS_NOT_IMPLEMENTED
Future Behavior
When implemented, this will: 1. Retrieve the original document from storage 2. Re-run chunking with current algorithms 3. Replace existing chunks 4. Generate new embeddings
Document Search
Document chunks are automatically searchable via the hybrid search endpoint:
curl -X POST https://api.penfield.app/api/v2/search/hybrid \
-H "Authorization: Bearer $JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"query": "python async programming",
"source_types": ["document_upload"],
"include_neighbors": 2
}'
Context Expansion
When searching documents, use include_neighbors to get surrounding context:
This includes 2 chunks before and after each matching chunk, providing better context.