Fix escaping markdown by rendering plaintext #622

kyrias · 2017-01-17T15:23:41Z

We still need to parse "plaintext" messages through the markdown
renderer so that escappes are rendered properly.

matrixbot · 2017-01-17T15:23:42Z

Can one of the admins verify this patch?

We still need to parse "plaintext" messages through the markdown renderer so that escappes are rendered properly. Fixes element-hq/element-web#2870. Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kyrias · 2017-01-17T15:28:13Z

(This is a WIP at the moment, will take care of the errors)

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kyrias · 2017-01-17T20:08:54Z

Should be ready fro reviewing now!

kyrias · 2017-01-17T20:10:42Z

Oh, and I just realized that I accidentally got c819b43 in there, but it's not directly related to this PR, it just fixes the old messageinput component to use the new markdown render function. I can split that out into its own PR if you want.

kyrias · 2017-01-17T21:13:06Z

Okay, this isn't quite ready for merging yet, there are some more functions that need to be nulled-out, eg " will be HTML entity encoded currently.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kyrias · 2017-01-17T21:27:00Z

And now that is fixed! This should probably be tested out for a little while by more than just me before this is merged, to try to catch any other possible edge cases.

kegsay · 2017-01-18T13:05:08Z

src/Markdown.js

@@ -56,23 +57,47 @@ export default class Markdown {
        return is_plain;
    }

-    toHTML() {
+    render(html) {


Please don't call this render, as it is confusing and can make readers mistake the Markdown class as a React class.

Also, it is not at all clear that html is a boolean. Please document what it is supposed to do.

What do you want me to rename it to? toHTML isn't really an apt name anymore with this.

EDIT: I guess I can revert this to be separate functions actually.

Given most of this function is branching based on this conditional, separate functions sounds excellent, as it will also make it clearer at the call site what you're actually doing toHTML or toPlaintext or whatever.

kegsay · 2017-01-18T13:08:41Z

src/Markdown.js

-                real_paragraph.call(this, node, entering);
+        } else {
+            renderer.out = function(s) {
+                this.lit(s);


Uhh, what is this doing? No comments.

Ah, seems I only added the comment in the commit message. Will fix that. The normal out function runs the output through a sanitizer that will entity encode things like quotation marks, which is problematic when it happens multiple times.

Thanks for the explanation. What's lit(s) doing though?

kegsay

This should probably be tested out for a little while by more than just me before this is merged, to try to catch any other possible edge cases.

Please add some unit tests for this. This should be pretty easy to do given how modular the Markdown class is.

kyrias · 2017-01-18T13:17:51Z

Unit tests for what exactly? Just the issue I found (" being entity encoded), or in general? Because for the latter I could really use some suggestions for what to test.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kegsay · 2017-01-18T14:06:39Z

Unit tests for what exactly? Just the issue I found (" being entity encoded), or in general? Because for the latter I could really use some suggestions for what to test.

The point is to test that the Markdown class actually works. We've already got a good list of stuff we've already found as bugs:

" being entity encoded.
Parsing escapes correctly if not markdown (e.g the test data in Riot fails to remove backslashes if the resulting message is entirely plain text element-hq/element-web#2870 )
-_- not doing <hr> : Replace marked with commonmark #575 (comment)
Check <s> works as intended : Replace marked with commonmark #575 (comment)
Check ~~~ does not strikethrough.
Check that "plain" markdown doesn't incorrectly fail the isPlainText() check (e.g single paragraph test, multiple paragraph test)

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kyrias · 2017-01-18T19:56:12Z

Split the functions and added some test cases now. It seems like while commonmark doesn't strip <s> tags, riot-web seems to not render them. <del> tags work fine though.

kegsay

Other than 2 things, this LGTM!

kegsay · 2017-01-19T08:56:09Z

src/Markdown.js

@@ -56,12 +59,10 @@ export default class Markdown {
        return is_plain;
    }

-    toHTML() {
-        const parser = new commonmark.Parser();
+    toHTML(html) {


You don't appear to use html.

Aah, relics that I missed in my git add -p!

kegsay · 2017-01-19T08:57:32Z

src/components/views/rooms/MessageComposerInput.js

+            if (md.isPlainText()) {
+                contentText = md.toPlaintext();
+            } else {
+                contentHTML = md.toHTML(true);


Likewise I don't think you need the true anymore.

kegsay · 2017-01-19T08:57:58Z

src/components/views/rooms/MessageComposerInputOld.js

@@ -331,6 +331,7 @@ module.exports = React.createClass({
                MatrixClientPeg.get().sendHtmlMessage(this.props.room.roomId, contentText, htmlText);
        }
        else {
+            const contentText = mdown.toPlaintext(false);


kegsay · 2017-01-19T09:02:01Z

test/components/views/rooms/MessageComposerInput-test.js

+        mci.handleReturn(sinon.stub());
+
+        expect(spy.calledOnce).toEqual(true);
+        expect(spy.args[0][1]).toEqual('Lorem ipsum dolor sit amet, consectetur adipiscing elit.');


Don't you need to assert that spy.args[0][2] is actually blank to be sure it didn't try to send this as HTML?

If it tried to send it as HTML the calledOnce assertion will fail, because then sendHtmlMessage will have been used rather than sendTextMessage

Aha! Gotcha.

kegsay · 2017-01-19T09:03:51Z

test/components/views/rooms/MessageComposerInput-test.js

+        mci.handleReturn(sinon.stub());
+
+        expect(spy.calledOnce).toEqual(true);
+        expect(spy.args[0][1]).toEqual('Lorem ipsum dolor sit amet, consectetur adipiscing elit.\n\nFusce congue sapien sed neque molestie volutpat.');


Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kegsay · 2017-01-19T13:38:40Z

LGTM!

Fix escaping markdown by rendering plaintext

893a5c9

We still need to parse "plaintext" messages through the markdown renderer so that escappes are rendered properly. Fixes element-hq/element-web#2870. Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kyrias force-pushed the commonmark-fix-escaping branch from ddd7467 to 893a5c9 Compare January 17, 2017 15:24

kyrias added 3 commits January 17, 2017 20:32

markdown: Only add \n\n on multiple paragraphs

35d70f0

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

Make old message composer use new markdown

c819b43

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

Markdown: softbreak is not HTML

49d60ff

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

Markdown: Don't XML escape the output when not HTML

2e3bdcf

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kegsay reviewed Jan 18, 2017

View reviewed changes

kegsay suggested changes Jan 18, 2017

View reviewed changes

Markdown: Add comment about out function override

6d2e521

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kyrias added 2 commits January 18, 2017 19:29

Markdown: Split up render function into toHTML/toPlaintext

30bd01c

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

Add markdown test-cases

14ead37

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kegsay suggested changes Jan 19, 2017

View reviewed changes

Markdown: delete remaining pre-split relics

9c1c657

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>

kegsay approved these changes Jan 19, 2017

View reviewed changes

kegsay merged commit 89fa47d into matrix-org:develop Jan 19, 2017

kyrias deleted the commonmark-fix-escaping branch January 19, 2017 14:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix escaping markdown by rendering plaintext #622

Fix escaping markdown by rendering plaintext #622

kyrias commented Jan 17, 2017

matrixbot commented Jan 17, 2017

kyrias commented Jan 17, 2017

kyrias commented Jan 17, 2017

kyrias commented Jan 17, 2017

kyrias commented Jan 17, 2017

kyrias commented Jan 17, 2017

kegsay Jan 18, 2017

kegsay Jan 18, 2017

kyrias Jan 18, 2017 •

edited

Loading

kegsay Jan 18, 2017

kegsay Jan 18, 2017

kyrias Jan 18, 2017

kegsay Jan 18, 2017

kegsay left a comment

kyrias commented Jan 18, 2017

kegsay commented Jan 18, 2017

kyrias commented Jan 18, 2017

kegsay left a comment

kegsay Jan 19, 2017

kyrias Jan 19, 2017

kegsay Jan 19, 2017

kegsay Jan 19, 2017

kegsay Jan 19, 2017

kyrias Jan 19, 2017

kegsay Jan 19, 2017

kegsay Jan 19, 2017

kegsay commented Jan 19, 2017

Fix escaping markdown by rendering plaintext #622

Fix escaping markdown by rendering plaintext #622

Conversation

kyrias commented Jan 17, 2017

matrixbot commented Jan 17, 2017

kyrias commented Jan 17, 2017

kyrias commented Jan 17, 2017

kyrias commented Jan 17, 2017

kyrias commented Jan 17, 2017

kyrias commented Jan 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyrias Jan 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kegsay left a comment

Choose a reason for hiding this comment

kyrias commented Jan 18, 2017

kegsay commented Jan 18, 2017

kyrias commented Jan 18, 2017

kegsay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kegsay commented Jan 19, 2017

kyrias Jan 18, 2017 •

edited

Loading