Skip to content

search: refactor front end parser types

Administrator requested to merge rvt/refactoring-parser-types-range into main

Created by: rvantonder

Set up for using the current parser as a tokenizer for further parsing.

By commit:

  1. Pulls apart the "list" type (Sequence) from true Token types. I think the distinct separation of Token and Sequence is clearer, and though I'm intrigued by the use of the Typescript utility types, think the removal of Exclude<Token, Sequence> improves things. The union of Token and Sequence I'm calling Term

  2. This commit pulls range out of ParseSuccess. In general range I felt range is better suited attached to Token/Term rather than the result type (and parent to the Token/Term). At the "cost" of adding range as a member to each token, it simplifies all the cases where we do Pick<ParseSuccess<Token>, 'range' | 'token'> and just use Token directly instead.

  3. This updates references to token, and tests. There's quite a bit of shrinking and simpler code because the above two commits flatten range into tokens, and removes the token prop that nests tokens in sequences.

Expand for old context

High level context, but: I'm strongly considering reuse of our existing parser for lexing (generate sequence of tokens), and then feed those tokens into the parsing routine similar to https://github.com/sourcegraph/sourcegraph/pull/15013. I sense it'll separate concerns for what we need in the frontend better than putting it into one routine (can go into details later).

In this PR I'm refactoring some parts of the parser (i.e., maybe lexer-to-be :-)) that helps my understanding and push along the above, and also, I ran into a Typescript question that baffles me.

By commit:

  1. Pulls apart the "list" type (Sequence) from true Token types. I think the distinct separation of Token and Sequence is clearer, and though I'm intrigued by the use of the Typescript utility types, think the removal of Exclude<Token, Sequence> improves things. The union of Token and Sequence I'm calling Term

  2. This commit pulls range out of ParseSuccess. In general range I felt range is better suited attached to Token/Term rather than the result type (and parent to the Token/Term). At the "cost" of adding range as a member to each token, it simplifies all the cases where we do Pick<ParseSuccess<Token>, 'range' | 'token'> and just use Token directly instead.

  3. This updates references to token, and tests. There's quite a bit of shrinking and simpler code because the above two commits flatten range into tokens, and removes the token prop that nests tokens in sequences.

Now the one thing that baffles me: after updating references in (3), I get:

Unsafe member access FOO on an any value.

but only in some places, and not others. So I don't know why I see this and how to fix it. I can't wait to hear the answer to why this is. See commit and inline comments.

Note: Build fails to something unrelated in to this PR, it fails for me branching off main, based on some extension snapshot, not the changes in this PR. rebased

Merge request reports