# Epic AI-18 - Semantic search for work packages
Project: Stream AI
Status: in development
URL: https://community.openproject.org/work_packages/AI-18
## Detailed specification
### Motivation and background information
### Acceptance criteria
### Out of scope
| Attribute | Value |
| --- | --- |
| Mockups | |
## Planning
| Attribute | Value |
| --- | --- |
| Roadmap | 2027 Q1 |
| Date | |
| List | RM AI |
| Target versions | 18.1.0 |
| Relevance | |
## Subitems
| ID | Type | Subject | Status | Assignee | Target versions | Project | Priority | Spent time |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AI-163 | Feature | Update the semantic index when work packages are created or edited or deleted | needs review | Joel Koch | 18.1.0 | Stream AI | Medium | |
| AI-168 | Feature | Semantic search admin settings | review approved | | 18.1.0 | Stream AI | Medium | |
| AI-169 | Feature | Incorporate semantic search UI into the global search | needs review | Joel Koch | 18.1.0 | Stream AI | Medium | |
| AI-181 | Feature | Create semantic index with embeddings for work packages | review approved | Joel Koch | 18.1.0 | Stream AI | Medium | |
| AI-165 | Implementation | Extend the feature flag "semantic_search" to all semantic search functionality | new | Joel Koch | | Stream AI | Medium | |
| AI-170 | Feature | Semantic search ingestion management for admins | new | Joel Koch | | Stream AI | Medium | |
| AI-179 | Open Point | Add HNSW index to embeddings column? | decided | Joel Koch | | Stream AI | Medium | |
| AI-180 | Implementation | Enable pgvector extension if available | in review | Joel Koch | | Stream AI | Medium | |
| AI-182 | Feature | Report indexing progress | closed | Joel Koch | | Stream AI | Medium | |
| AI-184 | Feature | Configure prefixes for embedding models with support | new | | | Stream AI | Medium | |
| AI-188 | Open Point | Store embeddings of different entities in the same or separate tables? | new | | | Stream AI | Medium | |
| AI-189 | Open Point | Do we make pgvector a requirement for running OpenProject? | decided | Joel Koch | | Stream AI | Medium | |
| AI-191 | Open Point | Should we cache query vectors? | new | | | Stream AI | Medium | |
| AI-193 | Feature | Change batch size for embedding API calls | new | | | Stream AI | Medium | |
| AI-195 | Open Point | Register pgvector types or use neighbor gem | decided | Joel Koch | | Stream AI | Medium | |
| AI-200 | Implementation | Build and use custom postgres + pgvector image | new | | | Stream AI | Medium | |
| AI-201 | Open Point | What embedding model do we deploy on our managed AI? | new | | | Stream AI | Medium | |
## Bugs and other work packages
| ID | Type | Subject | Status | Assignee | Target versions | Module |
| --- | --- | --- | --- | --- | --- | --- |
| AI-195 | Open Point | Register pgvector types or use neighbor gem | decided | Joel Koch | | |
| AI-189 | Open Point | Do we make pgvector a requirement for running OpenProject? | decided | Joel Koch | | |
| AI-179 | Open Point | Add HNSW index to embeddings column? | decided | Joel Koch | | |
| AI-202 | Task | Configure a locally running embeddings model on managed AI | in progress | Thomas Werkmeister | | |
| AI-219 | Task | Prefill model dimension and context length when creating semantic index | new | | | |
| AI-218 | Task | Perform index updates in batches | new | | | |
| AI-201 | Open Point | What embedding model do we deploy on our managed AI? | new | | | |
| AI-191 | Open Point | Should we cache query vectors? | new | | | |
| AI-188 | Open Point | Store embeddings of different entities in the same or separate tables? | new | | | |
| AI-187 | Bug | Changing the provider in the admin settings should not drop the current model | new | | | |
| AI-185 | Task | Handle embedding failures of individual work packages | new | | | |
| AI-183 | Task | Measure success of semantic search | new | | | |
## Details
| Attribute | Value |
| --- | --- |
| Module | Artificial Intelligence / Semantic Search (SemS) |
| Enterprise plan | Enterprise / Professional (PRO) |
| Labels | |
## People
| Attribute | Value |
| --- | --- |
| Assignee | Joel Koch |
| Accountable | Dominic Bräunlein |
| Product manager | Dominic Bräunlein |
| Designer | |
| Developers | bitcrowd |
| Interest by | |
## Estimates and progress
| Attribute | Value |
| --- | --- |
| Work | 50d 0h · Σ 50d 0h |
## Description
# User Problem
## Problem
* Keyword search only returns results when the user already knows how something was described.
* Users can narrow results with filters, but that requires knowing the right project, version, or other criteria upfront.
## Pain
* **Duplicate work:** Feature requests and bug reports get created multiple times because contributors cannot verify whether the same issue already exists.
* **Lost decisions:** When implementing a feature or investigating a bug, developers and product managers cannot easily find the work packages that explain why a functionality works the way it does.
* **Incomplete discovery:** Users researching a problem area cannot surface all related work packages unless they know every keyword and synonym used across all contributors and projects.
# Solution
* Notes:
* Use pgvector
* Use half-precision then up to 4000 dimensions are supported
* Dimension are configurable in the admin config
* The embeddings are saved in its own table
* Dimension size is optionally used for API calls to define dimension size
* On admin configuration the column is dropped and newly generated with index based on the configured size
* Admin configuration also needs a configureable prefix for search and document query
* Configurable batch count
* Should display progress on re-calculation of index
* Search UI should show when re-calculating
* Search could be limited to work packages first and be as small as a checkbox "Use semantic search"
*
## Out of Scope for the MVC
* Meetings, documents, projects, wiki
# Launch and Growth
## Measures
* Measure the position of the work package the user was looking for within the search results
* Measure how often it is used
## Messaging
Headline
Search smarter, not harder.
First Paragraph
Stop scanning filtered lists and guessing keywords. Semantic search understands the meaning behind your query and surfaces relevant work packages