[SwiftParser] Improve statement level recovery #3156

rintaro · 2025-09-26T23:13:15Z

Consider the following code:

}
@attr1 func foo() {}

Previously, this was parsed as a FunctionDeclSyntax with the unexpected code } @attr attached as unexpectedBeforeFuncKeyword. This was problematic because @attr was effectively ignored. The TokenPrecedence-based recovery strategy couldn't handle this well, since @ is not a declaration keyword and doesn't by itself signal the start of a declaration.

One option would be to check atStartOfDeclaration() after each skipSingle() and recognize @ as the start of a declaration. However, then the preceding } would need to be attached to the first attribute in the attribute list, which would require threading recovery logic into parseAttributeList() and only applying it to the first parseAttribute(). This would make the code more complex and harder to maintain.

This PR introduces a new syntax node UnexpectedCodeDeclSyntax

It represents unexpected code at the statement/declaration level.
All statement-level recovery is now handled by parseUnexpectedCodeDeclaration().
This simplifies recovery handling significantly.

Also:

Improve #if related recovery:
- orphan #endif et al at top-level is now UnexpectedCodeDecl instead of making everything after it "unexpected"
```
#endif
func foo() {}
```
- Unbalanced #endif et al closes code/member block with missing }.
```
struct S {
  #if FOO
  func foo() {
  #endif
}
```
- Simplified parsePoundDirective(). It now receives only one closure instead of 3.
Don't attach ';' to compiler directives. For example:
```
#sourceLocation(file: "<file>", line: 100)
;
```
; is now diagnosed as ;-only statement, or unexpected ; separator depending on the position.
Adjust TokenPrecedence for decl/statement keyword so. Decl keyword recovery won't go beyond } or statements. E.g.
```
func foo() {
  @attr
}
struct S {}
```
```
func foo() {
  @attr
  if true {}
  struct S {}
}
```
cont...

rintaro · 2025-09-27T01:58:17Z

Sources/SwiftParser/Declarations.swift

        _ = subparser.consumeAttributeList()
        hasAttribute = true
      } else if subparser.at(.poundIf) && subparser.consumeIfConfigOfAttributes() {
-        subparser.skipSingle()


subparser.consumeIfConfigOfAttributes() already skipped #if .... #endif

rintaro · 2025-09-27T03:54:09Z

CodeGeneration/Sources/SyntaxSupport/CommonNodes.swift

    base: .syntax,
    nameForDiagnostics: "declaration",
-    parserFunction: "parseDeclaration"
+    parserFunction: "parseDeclarationOrIfConfig"


parseDeclaration() don't parse #if ... #endif. Newly introduced parseDeclarationOrIfConfig() just for DeclSyntax.parse(from:)

rintaro · 2025-09-27T04:01:32Z

Sources/SwiftParser/Declarations.swift

  mutating func atStartOfDeclaration(
-    isAtTopLevel: Bool = false,
    allowInitDecl: Bool = true,
-    allowRecovery: Bool = false


Removed allowRecovery from atStartOfDeclaration()/atStartOfStatement()/atStartOfSwitchCase() because all statement-level recoveries are now done by parseUnexpectedCodeDecl()

rintaro · 2025-09-27T04:07:14Z

Sources/SwiftParser/Declarations.swift

  mutating func parseDeclaration(in context: DeclarationParseContext = .topLevelOrCodeBlock) -> RawDeclSyntax {
-    // If we are at a `#if` of attributes, the `#if` directive should be
-    // parsed when we're parsing the attributes.
-    if self.at(.poundIf) && !self.withLookahead({ $0.consumeIfConfigOfAttributes() }) {


This block is basically hoisted to parseMemberBlockItem()

To align with parseCodeBlockItem() scheme

Don't want to attach ; to IfConfigDeclSyntax

rintaro · 2025-09-27T04:08:23Z

Sources/SwiftParser/Declarations.swift

-      let isProbablyVarDecl = self.at(.identifier, .wildcard) && self.peek(isAt: .colon, .equal, .comma)
-      let isProbablyTupleDecl = self.at(.leftParen) && self.peek(isAt: .identifier, .wildcard)
-
-      if isProbablyVarDecl || isProbablyTupleDecl {
-        return RawDeclSyntax(self.parseBindingDeclaration(attrs, .missing(.keyword(.var)), in: context))
-      }


Factored out to atBindingDeclarationWithoutVarKeyword() and move the handling to recoveryResult above, aligning with atFunctionDeclarationWithoutFuncKeyword().

rintaro · 2025-09-27T04:09:36Z

Sources/SwiftParser/Declarations.swift

-
-      if atFunctionDeclarationWithoutFuncKeyword() {
-        return RawDeclSyntax(self.parseFuncDeclaration(attrs, .missing(.keyword(.func))))
-      }


This is already handled in recoveryResult above.

rintaro · 2025-09-27T04:46:12Z

Sources/SwiftParser/Statements.swift

    if self.atStartOfStatement(preferExpr: true) || self.atStartOfDeclaration() {
      return false
    }
+    if self.atStartOfLine && self.withLookahead({ $0.atStartOfSwitchCase() }) {


Previously case was a start of statement, but not anymore. This prevents the second case below is parsed as the returning expression.

case .foo: return case bar: ...

rintaro · 2025-09-27T04:50:59Z

Sources/SwiftParser/TopLevel.swift

+
+    let item: RawCodeBlockItemSyntax.Item
+    let attachSemi: Bool
+    if self.at(.poundIf) && !self.withLookahead({ $0.consumeIfConfigOfAttributes() }) {


Inlined parseItem() because we want to choose whether to attach ; (attachSemi) depending on what we parse.

rintaro · 2025-09-27T04:54:43Z

Sources/SwiftParser/TopLevel.swift

+    } else {
+      // Otherwise, eat the unexpected tokens into an "decl".
+      item = .decl(
+        RawDeclSyntax(
+          self.parseUnexpectedCodeDeclaration(allowInitDecl: allowInitDecl, requiresDecl: false, until: stopCondition)
+        )
+      )
+      attachSemi = true


Previously in parseItem(), we tried atStartOfStatement(allowRecovery: true) and atStartOfDeclaration(allowRecovery: true) for recoveries.
Now unexpected tokens at this position are always parsed as UnexpectedCodeDeclSyntax until we find a decl, statement, or expression.

rintaro · 2025-09-27T04:58:04Z

Tests/SwiftDiagnosticsTest/ParserDiagnosticsFormatterIntegrationTests.swift

      1 | func o() {
      2 | }👨‍👩‍👧‍👦}
-        |  |`- error: extraneous braces at top level
+        |  |`- error: unexpected braces in source file


This message change is because previously the UnexpectedNodesSyntax was stored as SourceFileSyntax.unexpectedBetweenStatementsAndEndOfFileToken, but now it's UnexpectedCodeDeclSyntax in SourceFileSyntax.statements.

rintaro · 2025-09-27T05:01:38Z

Tests/SwiftParserTest/AttributeTests.swift

        DiagnosticSpec(
          locationMarker: "2️⃣",
-          message: "unexpected code '))' before macro"
+          message: "unexpected code '))' in class"


This and similar changes are because the unexpected code are now separate UnexpectedCodeDeclSyntax in the statements.

rintaro · 2025-09-27T14:05:59Z

@swift-ci Please test

rintaro · 2025-09-27T19:21:33Z

swiftlang/swift#84553
@swift-ci Please test

rintaro · 2025-09-28T03:09:18Z

swiftlang/swift#84553
@swift-ci Please test

rintaro · 2025-09-28T04:08:13Z

swiftlang/swift#84553
@swift-ci Please test

rintaro · 2025-09-28T04:08:48Z

swiftlang/swift#84553
@swift-ci Please test Windows

rintaro · 2025-09-28T05:06:25Z

swiftlang/swift#84553
@swift-ci Please test Windows

rintaro · 2025-09-28T13:14:06Z

swiftlang/swift#84553
@swift-ci Please test Windows

rintaro force-pushed the parser-recover-stmtlevel branch from c706d4c to 2ca55fe Compare September 26, 2025 23:27

rintaro commented Sep 27, 2025

View reviewed changes

rintaro force-pushed the parser-recover-stmtlevel branch from ef1bf91 to dc4e479 Compare September 27, 2025 03:58

rintaro commented Sep 27, 2025

View reviewed changes

rintaro added 11 commits September 27, 2025 08:38

[SwiftParser] Improve statement level recovery

bda7f10

[SwiftParser] Improve '#if' recovery

41286de

tbs

6026c78

tbs

91f9d63

tbs

ad00841

tbs

130a51b

cleanup

bfddfd6

tbs

0eccca1

comment

6d34216

cleanup

7d97ad9

format

a61a1a7

rintaro mentioned this pull request Sep 27, 2025

[ASTGen] Update for UnexpectedCodeDeclSyntax addition swiftlang/swift#84553

Open

fix

f9a97cf

rintaro force-pushed the parser-recover-stmtlevel branch from 5ceb6e8 to f9a97cf Compare September 28, 2025 03:08

format

229d5dc

[SwiftParser] Improve statement level recovery #3156

Are you sure you want to change the base?

[SwiftParser] Improve statement level recovery #3156

Conversation

rintaro commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rintaro Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rintaro commented Sep 27, 2025

Uh oh!

rintaro commented Sep 27, 2025

Uh oh!

rintaro commented Sep 28, 2025

Uh oh!

rintaro commented Sep 28, 2025

Uh oh!

rintaro commented Sep 28, 2025

Uh oh!

rintaro commented Sep 28, 2025

Uh oh!

rintaro commented Sep 28, 2025

Uh oh!

Uh oh!

rintaro commented Sep 26, 2025 •

edited

Loading

rintaro Sep 27, 2025 •

edited

Loading