Skip to content

Add Markdown rendering to AI Chat #13194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

Conversation

Yubo-Cao
Copy link
Contributor

@Yubo-Cao Yubo-Cao commented May 30, 2025

Closes #12234 (partially)

See this: https://drive.google.com/file/d/1VOvWBLP5E9I_sLXd7RpShfXoP0ttg0JZ/view?usp=sharing for a quick demo. Ctrl+C is supported as a part of this PR as well.

Before:
image

After:
image
image
image

Mandatory checks

  • I own the copyright of the code submitted and I license it under the MIT license
  • Change in CHANGELOG.md described in a way that is understandable for the average user (if change is visible to the user)
  • Tests created for changes (if applicable) (not applicable)
  • Manually tested changed features in running JabRef (always required)
  • Screenshots added in PR description (if change is visible to the user)
  • Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
  • Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

subhramit
subhramit previously approved these changes May 30, 2025
Copy link
Member

@subhramit subhramit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code reads really good.
Did not try out, but demo is attached to the description, so green from me.
Thanks @Yubo-Cao!

@koppor
Copy link
Member

koppor commented May 31, 2025

See this: drive.google.com/file/d/1VOvWBLP5E9I_sLXd7RpShfXoP0ttg0JZ/view?usp=sharing for a quick demo.

Looks very nice.

Side track: "Summary" is a very different AI function, therefore @InAnYan put another tab for "Summary". -- For instance, for large PDFs, JabRef splits into chuncks and summarizes chunks.

Copy link
Member

@koppor koppor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a small comment.

Are spaces correctly handled for the getTextFlowContent()?

I assume, adding test cases is too hard here?

@ThiloteE

This comment was marked as resolved.

@InAnYan
Copy link
Member

InAnYan commented May 31, 2025

This is wonderful, Yubo!

image

I can copy text! And you also solved a very very naughty problem, when you could click on past messages and chat scrolled down! That problem make me feel desperate about JavaFX.

I could approve this PR without reviewing code 😄

Though, I remembered I have one small addition

Copy link
Member

@InAnYan InAnYan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, I had to request one small change

/**
* A TextFlow that allows text selection and copying.
*/
public class SelectableTextFlow extends TextFlow {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OMG, Yubo, is it possible to submit this code to some library? GemsFX maybe? This is a very useful piece of code

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for GemsFX. If the maintainer Dirk doesn't want it, then ControlsFX

@Yubo-Cao
Copy link
Contributor Author

Yubo-Cao commented Jun 1, 2025

Only a small comment.

Are spaces correctly handled for the getTextFlowContent()?

I assume, adding test cases is too hard here?

So after actually switch to Arch Linux and run testcases over there (of course that was not needed at all to come up with this conclusion looking at how I butchered the Markdown AST), is that we currently don't handle whitespace perfectly, if at all.

  1. White space inside a single markdown node is always preserved, so *italic with space* that kind of thing is OK
  2. White space between nodes is always not preserved.
  3. Bulleted indentation always get destroyed (if you nest).
  4. Original markdown style marker would always disappear (think * and all those special characters)

I think my new attempt would be trying to keep track a bit more metadata in MarkdownTextflow and basically override getTextFlowContent.

@subhramit
Copy link
Member

@Yubo-Cao Please resolve conversations after you push the commits resolving them, not before

@Yubo-Cao Yubo-Cao requested a review from InAnYan June 2, 2025 22:10
@subhramit
Copy link
Member

If things get too convoluted, git merge upstream/main, git reset upstream/main and then make a fresh commit. you may have to force push.

Note: gitk --all to have the commit ids ready in gitk in case something goes wrong. Your local changes should not be lost, but it is always a good idea to have a trace/backup.

@subhramit
Copy link
Member

If things get too convoluted, git merge upstream/main, git reset upstream/main and then make a fresh commit. you may have to force push.

Note: gitk --all to have the commit ids ready in gitk in case something goes wrong. Your local changes should not be lost, but it is always a good idea to have a trace/backup.

Another handy resource: https://ohshitgit.com/

@koppor
Copy link
Member

koppor commented Jun 2, 2025

If things get too convoluted, git merge upstream/main, git reset upstream/main and then make a fresh commit. you may have to force push.

However, JabRef mainatinaers don't like force pushes - only in rare emergency cases ^^

I always direct to https://lostechies.com/joshuaflanagan/2010/09/03/use-gitk-to-understand-git/ -- then, the git commands should be more understandable.

@ThiloteE
Copy link
Member

Selecting text currently only works, if the mouse button still covers text, when letting go. If the mouse button is in a blank field when letting go at the end of the click, it will immediately unselect the text.

@subhramit
Copy link
Member

subhramit commented Jun 28, 2025

Currently does not handle tables, but that's ok. image

this is a good follow-up case to cover. once this is merged you or @Yubo-Cao can create an issue (reuse this screenshot)

@Yubo-Cao
Copy link
Contributor Author

I would like to briefly note that our current architecture does not support tables effectively. The best we will be able to achieve under the limitation of a TextFlow is something similar to rendering a table as a code block (with some bold/italic styling), which is part of the trade-off of not using an external markdown renderer and choosing to handroll a partial markdown support internally to save memory.

@InAnYan
Copy link
Member

InAnYan commented Jun 29, 2025

Here are my findings:

Important

image

When any kind of list is used, it adds a vertical space before previous text. I find it a bit ugly. I see that there are 2 ways to overcome this:

  • Do not add any spacing (I like this one more)
  • Add spacing before and after
  1. Is there a way to set maximum width of the message? It can span from left to right side of the screen. Maybe set maximum to 50.

  2. I had an error when I pasted a part of Thilo's message:

java.lang.StringIndexOutOfBoundsException: Range [1, -2) out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:55)
	at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:52)
	at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
	at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
	at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:4950)
	at java.base/java.lang.String.substring(String.java:2912)
	at org.jabref/org.jabref.gui.util.MarkdownTextFlow$MarkdownRenderer.visit(MarkdownTextFlow.java:323)
	at flexmark.util.ast@0.64.8-module/com.vladsch.flexmark.util.ast.NodeVisitor.visit(NodeVisitor.java:130)
	at flexmark.util.visitor@0.64.8-module/com.vladsch.flexmark.util.visitor.AstActionHandler.processNode(AstActionHandler.java:81)
	at flexmark.util.ast@0.64.8-module/com.vladsch.flexmark.util.ast.NodeVisitor.visit(NodeVisitor.java:116)
	at org.jabref/org.jabref.gui.util.MarkdownTextFlow$MarkdownRenderer.visit(MarkdownTextFlow.java:262)
	at flexmark.util.ast@0.64.8-module/com.vladsch.flexmark.util.ast.NodeVisitor.visit(NodeVisitor.java:130)
	at flexmark.util.visitor@0.64.8-module/com.vladsch.flexmark.util.visitor.AstActionHandler.processNode(AstActionHandler.java:81)
	at flexmark.util.ast@0.64.8-module/com.vladsch.flexmark.util.ast.NodeVisitor.visit(NodeVisitor.java:116)
	at org.jabref/org.jabref.gui.util.MarkdownTextFlow$MarkdownRenderer.render(MarkdownTextFlow.java:257)
	at org.jabref/org.jabref.gui.util.MarkdownTextFlow.setMarkdown(MarkdownTextFlow.java:73)
	at org.jabref/org.jabref.gui.ai.components.aichat.chatmessage.ChatMessageComponent.loadChatMessage(ChatMessageComponent.java:83)
	at org.jabref/org.jabref.gui.ai.components.aichat.chatmessage.ChatMessageComponent.lambda$new$0(ChatMessageComponent.java:48)
	at javafx.base@24.0.1/com.sun.javafx.binding.ExpressionHelper$SingleChange.fireValueChangedEvent(ExpressionHelper.java:192)
	at javafx.base@24.0.1/com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:91)
	at javafx.base@24.0.1/javafx.beans.property.ObjectPropertyBase.fireValueChangedEvent(ObjectPropertyBase.java:106)
	at javafx.base@24.0.1/javafx.beans.property.ObjectPropertyBase.markInvalid(ObjectPropertyBase.java:113)
	at javafx.base@24.0.1/javafx.beans.property.ObjectPropertyBase.set(ObjectPropertyBase.java:147)
	at org.jabref/org.jabref.gui.ai.components.aichat.chatmessage.ChatMessageComponent.setChatMessage(ChatMessageComponent.java:65)
	at org.jabref/org.jabref.gui.ai.components.aichat.chatmessage.ChatMessageComponent.<init>(ChatMessageComponent.java:60)
	at org.jabref/org.jabref.gui.ai.components.aichat.chathistory.ChatHistoryComponent.lambda$fill$1(ChatHistoryComponent.java:42)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.jabref/org.jabref.gui.ai.components.aichat.chathistory.ChatHistoryComponent.lambda$fill$0(ChatHistoryComponent.java:41)
	at javafx.graphics@24.0.1/com.sun.javafx.application.PlatformImpl.lambda$runLater$4(PlatformImpl.java:419)
	at javafx.graphics@24.0.1/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
	at javafx.graphics@24.0.1/com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
	at javafx.graphics@24.0.1/com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$1(GtkApplication.java:240)
	at java.base/java.lang.Thread.run(Thread.java:1447)

Message I pasted was:

can you use all mardkown syntax in your mesage? fill this template: # Heading 1

## Heading 2

### Heading 3

**Bold text**

*Italic text*

- Item 1
- Item 2
- Item 3

1. First item
2. Second item
3. Third item
[[Link](https://www.example.com/)](https://www.example.com)

Image

> Blockquote
`Inline code`

Note that code blocks within code-blocks are probably not trivial.

But then AI responded, everything was fine, but it also had error:

Uncaught exception occurred in Thread[#47,JavaFX Application Thread,5,main]
java.lang.StringIndexOutOfBoundsException: Range [1, -2) out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:55)
	at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:52)
	at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
	at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
	at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:4950)
	at java.base/java.lang.String.substring(String.java:2912)
	at org.jabref/org.jabref.gui.util.MarkdownTextFlow$MarkdownRenderer.visit(MarkdownTextFlow.java:323)
	at flexmark.util.ast@0.64.8-module/com.vladsch.flexmark.util.ast.NodeVisitor.visit(NodeVisitor.java:130)
	at flexmark.util.visitor@0.64.8-module/com.vladsch.flexmark.util.visitor.AstActionHandler.processNode(AstActionHandler.java:81)
	at flexmark.util.ast@0.64.8-module/com.vladsch.flexmark.util.ast.NodeVisitor.visit(NodeVisitor.java:116)
	at org.jabref/org.jabref.gui.util.MarkdownTextFlow$MarkdownRenderer.visit(MarkdownTextFlow.java:262)
	at flexmark.util.ast@0.64.8-module/com.vladsch.flexmark.util.ast.NodeVisitor.visit(NodeVisitor.java:130)
	at flexmark.util.visitor@0.64.8-module/com.vladsch.flexmark.util.visitor.AstActionHandler.processNode(AstActionHandler.java:81)
	at flexmark.util.ast@0.64.8-module/com.vladsch.flexmark.util.ast.NodeVisitor.visit(NodeVisitor.java:116)
	at org.jabref/org.jabref.gui.util.MarkdownTextFlow$MarkdownRenderer.render(MarkdownTextFlow.java:257)
	at org.jabref/org.jabref.gui.util.MarkdownTextFlow.setMarkdown(MarkdownTextFlow.java:73)
	at org.jabref/org.jabref.gui.ai.components.aichat.chatmessage.ChatMessageComponent.loadChatMessage(ChatMessageComponent.java:83)
	at org.jabref/org.jabref.gui.ai.components.aichat.chatmessage.ChatMessageComponent.lambda$new$0(ChatMessageComponent.java:48)
	at javafx.base@24.0.1/com.sun.javafx.binding.ExpressionHelper$SingleChange.fireValueChangedEvent(ExpressionHelper.java:192)
	at javafx.base@24.0.1/com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:91)
	at javafx.base@24.0.1/javafx.beans.property.ObjectPropertyBase.fireValueChangedEvent(ObjectPropertyBase.java:106)
	at javafx.base@24.0.1/javafx.beans.property.ObjectPropertyBase.markInvalid(ObjectPropertyBase.java:113)
	at javafx.base@24.0.1/javafx.beans.property.ObjectPropertyBase.set(ObjectPropertyBase.java:147)
	at org.jabref/org.jabref.gui.ai.components.aichat.chatmessage.ChatMessageComponent.setChatMessage(ChatMessageComponent.java:65)
	at org.jabref/org.jabref.gui.ai.components.aichat.chatmessage.ChatMessageComponent.<init>(ChatMessageComponent.java:60)
	at org.jabref/org.jabref.gui.ai.components.aichat.chathistory.ChatHistoryComponent.lambda$fill$1(ChatHistoryComponent.java:42)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.jabref/org.jabref.gui.ai.components.aichat.chathistory.ChatHistoryComponent.lambda$fill$0(ChatHistoryComponent.java:41)
	at javafx.graphics@24.0.1/com.sun.javafx.application.PlatformImpl.lambda$runLater$4(PlatformImpl.java:419)
	at javafx.graphics@24.0.1/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
	at javafx.graphics@24.0.1/com.sun.glass.ui.gtk.GtkApplication.enterNestedEventLoopImpl(Native Method)
	at javafx.graphics@24.0.1/com.sun.glass.ui.gtk.GtkApplication._enterNestedEventLoop(GtkApplication.java:310)
	at javafx.graphics@24.0.1/com.sun.glass.ui.Application.enterNestedEventLoop(Application.java:510)
	at javafx.graphics@24.0.1/com.sun.glass.ui.EventLoop.enter(EventLoop.java:107)
	at javafx.graphics@24.0.1/com.sun.javafx.tk.quantum.QuantumToolkit.enterNestedEventLoop(QuantumToolkit.java:649)
	at javafx.graphics@24.0.1/javafx.stage.Stage.showAndWait(Stage.java:429)
	at javafx.controls@24.0.1/javafx.scene.control.HeavyweightDialog.showAndWait(HeavyweightDialog.java:162)
	at javafx.controls@24.0.1/javafx.scene.control.Dialog.showAndWait(Dialog.java:346)
	at org.jabref/org.jabref.gui.JabRefDialogService.showErrorDialogAndWait(JabRefDialogService.java:210)
	at org.jabref/org.jabref.gui.JabRefGUI.lambda$start$1(JabRefGUI.java:99)
	at javafx.graphics@24.0.1/com.sun.javafx.application.PlatformImpl.lambda$runLater$4(PlatformImpl.java:419)
	at javafx.graphics@24.0.1/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
	at javafx.graphics@24.0.1/com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
	at javafx.graphics@24.0.1/com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$1(GtkApplication.java:240)
	at java.base/java.lang.Thread.run(Thread.java:1447)

Non-important

We can make a follow up issue with those moments + tables.

  1. I find that black text and blueish color is not super color-contrast. Previously, we had a text field that had a white background. What if we could do something like this (sorry for drawing, but in a hurry):

image

So, two rectangles with rounded corners.

  1. Different colors for AI and user message? I remember we had different colors, but maybe at some time we decided to use only one. In CSS there should be two classes: one for user message color, other for AI message.

  2. Would it be easy to add a "copy" button alongside the "delete" button when you hover any message? I know, the logic behind chat message component is very convoluted... I programmed in React ((Java/Type)Script), and tried to use their idioms in Java+JavaFX. Turned out to be ugly, but I don't know alternatives.

Offtopic

I think Markdown syntax is supported good-enough:
image

InAnYan
InAnYan previously approved these changes Jun 29, 2025
@Siedlerchr
Copy link
Member

The IOB looks like it's coming from Flexmark.
Debug and see what token(s) produce this and then file an issue at their repo

@koppor
Copy link
Member

koppor commented Jun 29, 2025

The IOB looks like it's coming from Flexmark.
Debug and see what token(s) produce this and then file an issue at their repo

Maybe MWE using a JUnit test...

subhramit
subhramit previously approved these changes Jun 29, 2025
Copy link
Member

@subhramit subhramit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Green from me.
To summarize, TODOs:

  1. Submission of useful jfx snippet to GemsFX/ControlsFX
  2. MWE reproduction and reporting to Flexmark
  3. File known issues in JabRef

Current:
Address Ruslan's comment on the changelog entry.

Rest seems good to go, thank you for such wonderful work, Yubo.

@Yubo-Cao Yubo-Cao dismissed stale reviews from subhramit and InAnYan via e3fa14f June 29, 2025 14:22
@Yubo-Cao
Copy link
Contributor Author

I cannot reproduce the errors that Ruslan encountered. However, I added a check for string boundaries to ensure that this won't happen. Regarding the color contrast, I simply derived -jr-theme up a bit, and I think that would be sufficient.

screenshot

W3C Contrast checker

@InAnYan
Copy link
Member

InAnYan commented Jun 29, 2025

OMG, maybe it is something wrong with my machine, if Yubo cannot reproduce my issues 🤣

Nice color change

@calixtus
Copy link
Member

Can you do a screenshot with dark theme too please?

@subhramit subhramit added status: awaiting-second-review For non-trivial changes and removed status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers labels Jun 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use Markdown in AI chat messages
7 participants