ui.Chat: Adding Shiny inputs via `chat.append_message()`

[kovla]

I’m using Python Shiny’s built-in ui.chat_ui() and chat.append_message() to build a chat interface. I’d like to insert a Shiny input (for example, an action link or a button) directly into a new chat message so that the user can interact with it. However, when I embed an input_action_link in the text passed to chat.append_message(...), Shiny displays the link visually but does not treat it as a real input (i.e., input.my_special_link() never increments).

The rationale for this approach is linked to best practices when building AI agents with a human in the loop. A chat-based input (which could even be constructed by an LLM) is a popular approach. It can be observed in ChatGPT, when user is asked to provide feedback. It can be found in the assistant-ui library (labeled "Generative UI"), where it is used to confirm an AI action, such transaction (https://blog.langchain.dev/assistant-ui/). Of course, one could just add such an element outside the chat, but it makes the GUI more complex and can be cumbersome when you have multiple tools and/or actions. Aligning all temporary one-off inputs with their chat context seems more proper.

Below is my minimal reproducible example. When you run it and enter a message in the chat, it appends a new chat message containing a “special link,” but that link isn’t recognized by Shiny. Uncommenting the workaround (inserting an empty <script> or using ui.insert_ui) does force a re-scan, making it work. But I’d like to avoid that if possible.

from shiny import App, ui, reactive

app_ui = ui.page_fluid(
    ui.chat_ui("chat")
)

def server(input, output, session):
    # Initialize the chat
    chat = ui.Chat("chat")

    @chat.on_user_submit
    async def on_user_submit():
        user_messages = chat.messages()
        user_text = user_messages[-1]["content"]
        
        shiny_link_html = ui.input_action_link("my_special_link", "Click me!")
        # This appears as a link visually but doesn't increment input.my_special_link()
        await chat.append_message(
            f"Here is a special link in the chat: {shiny_link_html}"
        )

        # Uncommenting the code below triggers a re-scan, which "fixes" it:
        # ui.insert_ui(
        #     ui.HTML("<script></script>"),
        #     selector="body",
        #     where="beforeEnd"
        # )

    @reactive.effect
    def watch_special_click():
        clicks = input.my_special_link()
        if clicks > 0:
            print(f"[Server] Special link clicked {clicks} time(s).")

app = App(app_ui, server)

1. Is there an official or recommended way to add Shiny inputs inside a chat message so that they’re automatically recognized by input[...] without needing a manual DOM insertion workaround?

2. Is there a stable, built-in function to trigger a “DOM re-bind” (like Shiny.bindAll() in R Shiny) from Python, rather than using ui.insert_ui(ui.HTML("<script></script>")) as a hack?

3. Is embedding Shiny inputs in chat messages expected to work in the future, or should I rely on separate UI insertion?

[gadenbuie]

2. Is there a stable, built-in function to trigger a “DOM re-bind” (like Shiny.bindAll() in R Shiny) from Python, rather than using ui.insert_ui(ui.HTML("<script></script>")) as a hack?

Shiny.bindAll() is part of the common client-side library used for both Shiny for Python and for R, so you can use it in this case.

@cpsievert will be able to give more context on the road map and his vision for embedding Shiny inputs in chat messages when he returns from vacation later this month. In the mean time, here's a small amount of JavaScript that you can use to bind Shiny inputs after they are added to the chat:

$('#chat').on('shiny-chat-append-message', () => $(document).one('shiny:idle', () => Shiny.bindAll()))

Throw that line in a ui.tags.script() in your UI and you're good to go. (I should warn, I'm not recommending this as The Way to Do It. We'll very likely do this automatically for users in the future.)

[gadenbuie]

@cpsievert While here I noticed that we use a shiny-chat-append-message custom event to add messages to the chat; it might be useful if shiny chat also emitted a shiny-chat-appended-message event, maybe in #finalizeMessage, that could serve as a follow-up hook for custom interactions.

[kovla]

Thanks for the answer!

It would seem there are some additional considerations when using chat.append_message_stream(). For example, the following code does not work:

from shiny import App, ui, reactive

app_ui = ui.page_fluid(
    ui.chat_ui("chat"),
    ui.tags.script(
        """
        $(function() {
          // We'll call Shiny.bindAll() after each appended message/chunk.
          function rebind() {
            console.log("[Debug] => calling Shiny.bindAll()");
            Shiny.bindAll();
          }

          // Fired by chat.append_message(...)
          $('#chat').on('shiny-chat-append-message', () => {
            console.log("[Debug] chat-append-message event fired!");
            $(document).one('shiny:idle', rebind);
          });

          // Fired by chat.append_message_stream(...)
          $('#chat').on('shiny-chat-append-message-chunk', () => {
            console.log("[Debug] chat-append-message-chunk event fired!");
            $(document).one('shiny:idle', rebind);
          });
        });
        """
    ),
)

def server(input, output, session):
    chat = ui.Chat("chat")

    @chat.on_user_submit
    async def on_user_submit():
        user_messages = chat.messages()
        user_text = user_messages[-1]["content"]

        # Attempt to stream a message that includes an action link
        shiny_link_html = ui.input_action_link("my_special_link", "Click me!")
        await chat.append_message_stream(
            f"Here is a special link in the chat: {shiny_link_html}"
        )

    @reactive.effect
    def watch_link():
        clicks = input.my_special_link()
        if clicks > 0:
            print(f"[Server] Link clicked {clicks} time(s).")

app = App(app_ui, server)

While the streaming variant of appending a message might be a less frequent use case, I think it cannot be fully excluded. For example, if UI is generated by an LLM, as a part of its structured output, one might want to stream it (this would be actual Generative UI, unlike my initial example, where the UI element is pre-defined).

Any tips on how to navigate this situation? Thanks in advance!

[gadenbuie]

The streaming case will definitely benefit from us solving this problem directly in the chat component, or at the very least in providing an "appended" event you can hook into.

For the time being, you can replace the $(document).one('shiny:idle', rebind) with a simple setTimeout(rebind, 0) (in the shiny-chat-append-message-chunk event handler), which will schedule the rebind to happen essentially right after the new chunk is appended.

[kramerrs]

This is an interesting feature. I am also trying to add inputs into the chat. Ideally, I would like to get user feedback on the quality of prompts and or allow for striking/rolling back a chat on either erroneous responses or prompts. I have successfully added an input into the chat in a test session. I haven't got to collecting input yet.

[jeffbryner]

For the time being, you can replace the $(document).one('shiny:idle', rebind) with a simple setTimeout(rebind, 0) (in the shiny-chat-append-message-chunk event handler), which will schedule the rebind to happen essentially right after the new chunk is appended.
@gadenbuie can you offer a sample code that does this? Would this also be the preferred way to append multimedia content produced by the model?

[gadenbuie]

@jeffbryner We're actively working on providing a better interface for this directly. You can follow progress for this feature in #1868. That PR likely won't be in the next release of Shiny but will land in main shortly after.

[kovla]

@jeffbryner We're actively working on providing a better interface for this directly. You can follow progress for this feature in #1868. That PR likely won't be in the next release of Shiny but will land in main shortly after.

Very glad to see this incoming feature, I cannot wait to replace my custom code with this out-of-the-box functionality!

One of the most typical interactive inputs to add to chat is a feedback form. Looking at ChatGPT,

one has thumbs-up and thumbs-down, a copy button - seems like a useful addition to most chats. The rest is not necessarily needed in each application. I imagine this widget as an input element with a custom function to persist the feedback (maybe with a default simply writing to std.out). An actual user implementation would typically entail writing to logs or a database.

Do you have something like this in mind upon release of the feature? Or do you see any other straightforward way to add such an interactive feedback widget? Thanks.

[linogaliana]

Would be a great feature ! This would make chatbot front in Shiny quite competitive with streamlit developed front that proposes such features (https://docs.streamlit.io/develop/tutorials/chat-and-llm-apps/chat-response-feedback)