`<br>` between `<span>` tags are not interpreted as whitespace

from <p class="p-adr"><span class="p-street-address">Street Name 9</span><br/><span class="p-locality">12345 NY, USA</span></p>

"adr": [
          "Street Name 912345 NY, USA"
        ]

[gRegorLove]

This appears to be due to the use of DOMNode::textContent, which will just remove the HTML. I found a suggested solution on Stackoverflow http://stackoverflow.com/a/4740264 to replace the <br> with line breaks (or spaces?) before calling textContent. Not sure how ideal that is; would we need to start adding other HTML elements to be replaced? What do you think, @barnabywalters?

closing as duplicate

[voxpelli]

@dym-cx A duplicate of what? 😛 The issue referencing this one is for a different repository

[voxpelli]

This is actually a pretty complex issue: http://perfectionkills.com/the-poor-misunderstood-innerText/

DOMNode::textContent only iterates over all DOM nodes and return their text representation.

The other property that's discussed in the above article is DOMNode::innerText, which might be the more accurate value to use. But – the behavior of that property hasn't been standardized and specified and it's also somewhat problematic in that it actually picks up changes added by eg. CSS as well – so if you eg. have a text-transform: lowercase then any parser that has applied that style to the HTML-document prior to checking the DOMNode::innerText will get all text lowercased – which isn't really what one wants either.

A similar edge-case here would probably be the <ul><li>One</li><li>Two</li></ul> – a list with no whitespace between the list items. This is popularized by the use of display: inline-block as the easiest way to get good spacing between each list-item when they are displayed as inline-blocks is to ensure that there are no whitespace characters between them (an alternative is to ensure that all whitespace characters have the font-size 0). Checking the DOMNode::textContent of such a list will return OneTwo when it's actually supposed to be One Two.

So it's probably not just enough to do a special case for <br /> – one probably has to do a special case for all block-level tags as well and ensure they are followed by a whitespace?

It would be good if this was specified out somewhere. Maybe the HTML5 parser specification has something?

Basically what one would want is something inbetween DOMNode::textContent and DOMNode::innerText – something that just returns the simplest possible parsed text representation of some HTML5 content, rather than the raw concatenated data of DOMNode::textContent or the over-processed data of DOMNode::innerText.

(Ping @kylewm in case he isn't subscribed)

[voxpelli]

Here's an earlier abandoned specification for innerText that was mentioned in the comments of the above article: https://rawgit.com/timdown/rangy/master/fiddlings/spec/innerText.htm I'm thinking that maybe that with all CSS-related stuff removed could be a good starting point for defining a method that's inbetween DOMNode::textContent and DOMNode::innerText?

[voxpelli]

@glennjones mentioned on IRC that he has an opt-in functionality on the node.js library for better extraction of text from DOM nodes: https://github.com/glennjones/microformat-shiv/blob/dev/lib/text.js More or less a CSS-less DOMNode::innerText.

Also a good basis to start from, but there needs to be an independent write up of it if it is to be the default of microformat parsers going forward I think. (And it's also useful for parsers outside of the microformat/indieweb community)

[glennjones]

I refer to the options.textFormat = 'normalised' switch in my parser as "a simple none-CSS aware version of innerText". The idea was not to create a direct copy of innerText, but a more pragmatic approach which gave me more human readable text output.

The approach as it stands is needs a lot more work, but its good enough in most case to improve the usability of the text output of a microformats parser. It could be expanded, formalised and documented to something more robust. Handling issue like what should replace a <br/> tag a space or a return char. If a return is that 1 or 2 return chars.

Happy to work on this with people if there is a enough interest

[aaronpk]

I believe another instance of this is when two <p> tags are adjacent. For example, the editor in Quill produces HTML such as:

<p>Hello</p><p>World</p>

Which is then parsed as HelloWorld instead of Hello World.

[gRegorLove]

I'm working on implementing @glennjones' solution currently.