Add support for C-style comments

[altmannmarcelo]

This commit adds support for C-style comments supported by MySQL. It parses and consumes the optional version number after the ! character.

[iffyio]

Looking at their docs, I'm wondering if/how we support these examples?

SELECT /*! STRAIGHT_JOIN */ col1 FROM table1,table2
/*!50110 KEY_BLOCK_SIZE=1024 */
SELECT /*! BKA(t1) */ FROM T

[altmannmarcelo]

@iffyio - Parsing the c_style comment unblocks sqlparser to not discard those as if they were a normal comment. Support for each hint will have to be added in a case by case bases. For example #2033 - MySQL adds a c-style comment if you run SHOW CREATE TABLE:

mysql> SHOW CREATE TABLE b;
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                    |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| b     | CREATE TABLE `b` (
  `ID` int DEFAULT NULL,
  `b` char(1) DEFAULT NULL /*!80023 INVISIBLE */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0,008 sec)

Without the current patch, the invisible keyword will be discarded.

[iffyio]

Ah so to clarify I'm rather wondering regarding the parser behavior for hints that aren't singe words e.g. /*!50110 KEY_BLOCK_SIZE=1024 */ - can we demonstrate the behavior with test cases for such scenarios?

[altmannmarcelo]

@iffyio thanks flagging this. I have fixed the issue and now we properly return individual tokens inside a C-style hint comment.

[iffyio]

Ah so to clarify I'm rather wondering regarding the parser behavior for hints that aren't singe words e.g. /*!50110 KEY_BLOCK_SIZE=1024 */ - can we demonstrate the behavior with test cases for such scenarios?

[iffyio]

@altmannmarcelo sorry for the delay in getting back to this, left some comments!

[iffyio]

Suggested change

	/// Returns true if the dialect supports hint and C-style comments
	/// e.g. `/*! hint */`
	fn supports_c_style_hints(&self) -> bool {
	/// Returns true if the dialect supports optimizer hints in multiline comments
	/// e.g. `/*!50110 KEY_BLOCK_SIZE = 1024*/`
	fn supports_multiline_comment_hints(&self) -> bool {

[iffyio]

Ah to clarify the suggested change, c-style comment I think is rather understood as /* */ - i.e. multiline comment in general, the feature here I think is the optimizer hints within comments as mysql calls them

[iffyio]

impl wise I'm thinking we can do something like the following that would be less invasive to the tokenizer:

If we take as an example /*!50110 KEY_BLOCK_SIZE = 1024*/ - today that will get parse as a Token::MultilineComment("!50110 KEY_BLOCK_SIZE = 1024") - which we receive here in the tokenizer loop.

Then the idea would be to essentially re-tokenize that string and add to the tokens buffer.

So something like this:

// we check if this is a token containing optimizer hints
match token {
	Token::Whitespace(Whitespace::MultiLineComment(comment)) if self.dialect.supports_multiline_comment_hints() && s.starts_with("!") => {
    // re-tokenize the hints
	buf.extend(tokenize_comment_hints(comment, span)?);
	token => {
		buf.push(TokenWithSpan{ token, span })
	}
}

// here we reuse existing tokenizer machinery to re-tokenize the hints.
fn tokenize_comment_hints(&self, hints: String, span: Span, tokens: &mut Vec<TokenWithSpan>) -> Result<()> {
	let mut state = State {
		peekable: hints.chars().peekable(),
		line: span.start.line,
		col: span.start.column,
	};

	while let Some(token) = self.next_token(&mut state, None)? {
		tokens.push(token);
	}

	Ok(())
}