Prompt API: consider local use cases for RAG and Agentic RAG

[zolkis]

[Refers to / transferred from the Prompt API ]

As a possible complex local application to prompting, Retrieval-Augmented Generation (RAG) enhances AI systems by dynamically retrieving external (to the model) information to improve response accuracy and relevance. Agentic RAG extends these capabilities with autonomous decision-making and iterative refinement, particularly valuable in local deployments for enhanced privacy and customization.

Some of the main RAG use cases include:

• Enhanced search & question answering: RAG improves search engines with up-to-date featured snippets and powers domain-specific QA systems that combine proprietary data (e.g., medical literature or legal documents) with foundational LLM knowledge.
• Healthcare: Provides clinicians with real-time access to medical guidelines and research.
• Legal: Accelerates case law research and compliance checks.
• E-commerce: Delivers personalized product recommendations using customer behavior data.
• Content generation: generates context-aware summaries, reports, and marketing copy by retrieving relevant source materials.
• Enterprise knowledge management: enables chatbots to answer internal queries about company policies, manufacturing protocols, or regulatory updates.

Agentic RAG introduces autonomous decision-making layers to traditional RAG workflows (by using the LLMs not only for generating answers, but also preparing the prompts), enabling:

• Adaptive problem solving: customer support agents offering discounts proactively.
• Continuous learning: medical diagnosis systems updating with new research.
• Multi-step reasoning: legal tools cross-referencing precedents and statutes.
• Local deployment advantages: privacy-focused healthcare data analysis.

Traditional RAG relies on cloud-based models/vector DBs, where limited customization due to API constraints.

Agentic RAG in a local deployment have advantages like:

• Data privacy: runs fully offline via tools like Langchain/Qdrant.
• Customization: supports domain-specific models (e.g., Gemma-3 for multilingual tasks).
• Cost Efficiency: eliminates API fees using local LLMs.
• Transparency: provides verifiable source citations from internal databases.

A local Agentic RAG pipeline (e.g. RAGapp deployed via Docker) typically includes:

# Simplified architecture
local_llm = Ollama(model="llama3")  # Local model
vector_db = Qdrant(embeddings=FastEmbed())  # On-premise storage
agent = LangchainAgent(
    tools=[KnowledgeBase(retriever=vector_db)],
    system_prompt="Generate markdown responses with citations"
)

This approach is particularly impactful in regulated industries like healthcare and finance, where data sovereignty and low-latency responses are critical.

In the CG, please consider/discuss/decide if any of these use cases would make it worth developing support as a new type of standardized assistance API, or as example application of prompting (if apps are a better place to manage RAG).

[zolkis]

Continuing, there are several tools and frameworks available that support RAG and Agentic RAG development in JavaScript, e.g. using LlamaIndex or EmbedJS:

const ragApplication = await new RAGApplicationBuilder()
  .setModel(SIMPLE_MODELS["OPENAI_GPT3.5_TURBO"])
  .setEmbeddingModel(new OpenAi3SmallEmbeddings())
  .setVectorDb(new LanceDb({ path: path.resolve("./db") }))
  .setCache(new LmdbCache({ path: path.resolve("./cache") }))
  .addLoader(new WebLoader({ urlOrContent: "https://www.example.com" }))
  .build();

const response = await ragApplication.query('Your question here');

Exposing [access to] something similar using a built-in API might be widely useful.

[domenic]

I can't really understand this issue, sorry. Can you summarize how it's related to the prompt API being developed in this repository?

[anssiko]

In the CG, please consider/discuss/decide if any of these use cases would make it worth developing support as a new type of standardized assistance API, or as example application of prompting (if apps are a better place to manage RAG).

Thanks for the proposal. RAG paradigms are currently considered out of scope for the Prompt API.

To that end, I'll transfer this issue to the proposals repo for further discussion.

[anssiko]

Apparently transferring issues using the built-in GH feature is not available right now.

Let's do this manually. @zolkis please open a new issue in the proposals repo, and close this one once done.

[zolkis]

Can you summarize how it's related to the prompt API being developed in this repository?

Prompt API workflow:
User Query → LLM → Response

RAG workflow:
User query → Retrieve Context (external data, internal databases, vector databases) → Augmented Prompt (query + context) → LLM → Response

So there are some extra steps in prompt preparation, which is why I mentioned this could be delegated to applications.

However, the Prompt API could be made more friendly to RAG implementations, supporting e.g. constraining responses to retrieved context, in order to mitigate factual errors. Other RAG functionality could be done in the apps, e.g. prompt engineering using QueryTransformer, ContentRetriever, ContentAggregator, etc. The benefits are wide and large, as RAG enables customization for specialized use cases (e.g., enterprise docs) without costly fine-tuning, requiring 10x less data than fine-tuning while remaining model-agnostic. So it would be a way to multiply the impact of the Prompt API.

RAG paradigms are currently considered out of scope for the Prompt API.

RAG and other prompt engineering were not excluded in the non-goals, hence this issue came here. If the CG should not work on these, they could be explicitly added to the non-goals, for clarity.

Apart from that, of course this can be moved to the proposals repo -- but that is a bit too generic, while this issue is tightly linked to the Prompt API. If we'd have a "future work" label it could be better, or I could turn/extend this into a .md file in a
"proposals" folder in this repo -- or indeed bury it to the generic proposals repo. But the simplest is to dismiss this issue and just close it.

[AdamSobieski]

@zolkis, hello. I'm also interested in RAG paradigms involving larger-sized documents in Web browsers (e.g., textbooks and instructional manuals) such that documents' scripts would be capable of interoperating with local or remote vector databases as users navigated through the documents.

With respect to RAG-related chunking algorithms, also interesting are:

1. markup-based solutions, e.g., marking <section>, <p>, or <span> elements with metadata or markup attributes having values meaningful to chunking algorithms.
2. style-based solutions, e.g., using cascading style selectors and (custom) style properties having values meaningful to chunking algorithms.
3. combinations of 1 and 2.

As considered, AI systems could provide responses referencing positions in documents such that users could click on citation-marks to navigate to, scroll to, and highlight cited content in those interoperating documents opened in Web browsers.

There are also exciting possibilities involving content selection. That is, as envisioned, users could make selections of content to subsequently engage in selection-focused Q&A and dialogue.

In any event, I wanted to share my two cents / brainstorming / feature requests. I look forward to any possible discussion, wherever these topics best fit, towards enhancing educational-technology scenarios.

[anssiko]

I look forward to any possible discussion, wherever these topics best fit, towards enhancing educational-technology scenarios.

@AdamSobieski, you are welcome to join the Community Group meetings to discuss new proposals.

Your another markup-based proposal webmachinelearning/prompt-api#70 is on the 2025-03-31 agenda.

[anssiko]

Issue transferred. It would be helpful to document this proposal in terms of an explainer
https://tag.w3.org/explainers/

Can some of this be userland scripts? What is the status of native SDKs and platform APIs required to implement this across major platforms? Can we reuse some of the existing Storage APIs?

[AdamSobieski]

With respect to local RAG and local graph RAG, I'm interested in markup-related and style-related solutions for interoperating with chunking, context management, and dynamic content storage algorithms. Which items should be stored or prioritized as end-users read through larger-sized documents and large multi-page documents? Might documents' authors (or their software algorithms) be able to make suggestions, in these regards?

Brainstorming, some content for local RAG and local graph RAG could be scopal, suggested to be relevant while a containing scope was active, visible, or displayed on screen. Perhaps data-* attributes could be used to signal to an interoperating JavaScript algorithm those elements which defined such scopes.

Single-page Documents

CSS selectors could be used to match (see: Element.matches()) element's selves and ancestors, algorithmically ascending document trees to find a first match (see: Element.closest()).

<section id="chapter3" epub:type="chapter">
  ...
  <section id="section10">
    ...
    <p class="example">
      ...
      <span id="s1" data-rag-scope="html">Relevant to local RAG while the document is displayed.</span>
      <span id="s2" data-rag-scope="[epub|type='chapter']">Relevant to local RAG while chapter 3 is displayed.</span>
      <span id="s3" data-rag-scope="#section10">Relevant to local RAG while section 10 is displayed.</span>
      <span id="s4" data-rag-scope="p">Relevant to local RAG while containing paragraph is displayed.</span>
      <span id="s5" data-rag-scope=".example">Relevant to local RAG while containing paragraph is displayed.</span>
      <span id="s6" data-rag-scope="*">Relevant to local RAG while it is displayed.</span>
      ...
    </p>
    ...
  </section>
  ...
</section>

Another useful feature would be a means for document authors to be able to provide content and hints intended for AI assistants, e.g., for storage with local RAG, which was outside of the flow of a readable document for end-users. I'm not yet sure how this would best be expressed, but it might resemble something like:

<section id="chapter3">
  ...
  <aside class="for-use-by-ai" style="display: none">
    <p data-rag-scope="section">This isn't visible to end-users but could be informative to AI assistants.</p>
  </aside>
  ...
</section>

These or similar techniques could work with local graph RAG. For local graph RAG, one could express something like:

<html>
  <head>
    ...
    <script id="g1" type="text/n3" src="graph1.n3" />
    ...
  </head>
  <body>
    ...
    <section>
      ...
      <p data-rag-xref="#g1" data-rag-scope="section">
        The contents of 'graph1.n3' are relevant to local graph RAG while the containing section is displayed.
      <p>
      ...
    </section>
    ...
  </body>
</html>

or, more succinctly, something resembling:

<html>
  <body>
    ...
    <section>
      ...
      <p data-rag-src="graph1.n3" data-rag-scope="section">
        The contents of 'graph1.n3' are relevant to local graph RAG while the containing section is displayed.
      <p>
      ...
    </section>
    ...
  </body>
</html>

Multi-page Documents

For multi-page documents, how might one express in a data-rag-scope attribute that an element's indicated content would be suggested as being relevant to end-users for scopes outside those of the current page?

Brainstorming, an interoperating JavaScript library could detect a custom keyword or function, e.g., outline(), envisioned as having meaning in the context of the data-rag-scope attribute.

<html>
  <head>
    <link rel="outline" href="outline.xml" />
    <meta name="outline-id" content="page164" />
    <link rel="prev" href="163.xhtml" />
    <link rel="next" href="165.xhtml" />
    <script type="text/javascript" src="local-rag.js" />
  </head>
  <body>
    <p data-rag-scope="outline([*|role='chapter'])">Relevant to local RAG in a scope outside the current page.</p>
  </body>
</html>

For this to work, the page would need to part of some sort of outline structure. What might the outline.xml resource resemble? Perhaps something like:

<seq role="book" xmlns="...">
  ...
  <seq role="chapter" id="chapter3">
    <item id="page103" src="103.xhtml" />
    ...
    <seq role="section" id="chapter3/section10">
      <item id="page149" src="149.xhtml" />
      ...
      <seq role="section" id="chapter3/section10/section4">
        <item id="page163" src="163.xhtml" />
        <item id="page164" src="164.xhtml" /> <!-- This is the page for the example above. -->
        <item id="page165" src="165.xhtml">
          <iframe id="page165-widget" src="widget.xhtml" />
        </item>
      </seq>
      ...
    </seq>
    ...
  </seq>
  ...
</seq>

Towards optimizing multi-page local RAG scenarios, one could use metadata on outlines' elements to alleviate the need to retrieve each and every page to determine which contents were scoped to the current book, chapter, section, and so forth.

One could enhance outlines to reference JSON-based resources to bulk-insert contents into local RAG databases. In the following example, all of the content for an extra-page local RAG scope for Chapter 3 is stored in chapter3-rag-scope.json which is referenced from that chapter's metadata in the outline.

<seq role="book" xmlns="...">
  <link rel="rag-content" href="book-rag-scope.json" />
  ...
  <seq role="chapter" id="chapter3">
    <link rel="rag-content" href="chapter3-rag-scope.json" />
    <item id="page103" src="103.xhtml" />
    ...
    <seq role="section" id="chapter3/section10">
      <link rel="rag-content" href="chapter3-section10-rag-scope.json" />
      <item id="page149" src="149.xhtml" />
      ...
      <seq role="section" id="chapter3/section10/section4">
        <link rel="rag-content" href="chapter3-section10-section4-rag-scope.json" />
        <item id="page163" src="163.xhtml" />
        <item id="page164" src="164.xhtml" />
        <item id="page165" src="165.xhtml">
          <iframe id="page165-widget" src="widget.xhtml" />
        </item>
      </seq>
      ...
    </seq>
    ...
  </seq>
  ...
</seq>

Conclusion

Broadly, enhancing digital books, textbooks, and their features are interesting to me. Perhaps outlines, tables of contents, glossaries, and indexes could be enhanced for AI interoperation. Perhaps featureful markup, JSON, and/or JS scripts in digital books could be used to enhance Q&A and dialogue about the structures of books and about the locations of sought contents within them, in addition to supporting Q&A and dialogue about their contents.

I'm also interested in enabling end-users to be able to receive responses from AI models with citations referencing into their local documents. So, if used in a response, the citation mark, e.g., [1], for some content would, if clicked upon, navigate to, scroll to, and highlight that content for the end-user.

[AdamSobieski]

With respect to implementing local vector databases in Web browsers, I found some reference materials [1][2][3] which discuss how vector databases are much more, under the hood, than databases with columns for embedding vectors. Multiple families of algorithms exist to deliver higher-performance results.

Along these lines, one of the materials [3] states:

We introduced the families of algorithms available for vector databases, namely tree-based, quantization, hashing, clustering, and graph methods. Each family of algorithms comes with its own set of performance trade-offs, which can significantly impact the performance of a vector database. Then we highlighted that many modern vector databases use hybrid methods, combining the strengths of different methods to achieve high speed and accuracy. Finally we took an in-depth look at several specific algorithms including: Annoy, Product Quantization, Local Sensitivity Hashing, DBSCAN, and HNSW, comparing how their tradeoffs will affect their use on production datasets. While each algorithm has its strengths and weaknesses, the choice of algorithm should depend on the specific requirements of the task, including the size and dimensionality of the dataset, the available computational resources and budget, and the acceptable trade-off between accuracy and efficiency.

As @zolkis shared, there exist a number of JavaScript (and some WASM) solutions. As he sketched, Web developers could select from interchangeable local and remote vector databases using dependency injection when building and creating components.

In my opinion, something like IndexedDB would be very useful to provide local vector databases. Per the above quote, local vector database implementations could, through an API, provide Web developers with their capabilities regarding available choices of algorithms and could be configurable during the creation and initialization of instances, in these regards.

[1] https://medium.com/@david.gutsch0/vector-databases-the-unseen-powerhouse-of-the-ai-revolution-part-1-6685653abd92
[2] https://medium.com/@david.gutsch0/the-art-of-embeddings-transforming-text-for-vector-databases-926738443e70
[3] https://medium.com/@david.gutsch0/vector-databases-understanding-the-algorithm-part-3-bc7a8926f27c