This fixes the line highlight offsets that we use to render the highlights in the native code preview area.
Some notes about the issue:
Previously we were using byte length (String#length) to sum up the length for all previous rows and code pointer size for the offset in the current line. This is incompatible and caused the offsets to be invalid.
Java does not understand Unicode code pointers well and the only API we have to set highlights in the code preview requires absolute byte offsets.
So to fix the issue, this PR attempts to convert the offsets returned by the Sourcegraph API (code pointers) to the UTF byte offsets.
The conversation works by finding the offset in the line based on the unicode code pointer and then counting the bytes up to that offset.
A note about symbol search
Problem 1: Highlight offsets are broken
Symbol search does not return offsets from the API. Instead, we rely on the offsets encoded in the URL. Unfortunately it seems that these are not working for unicode characters and the offsets returned are larger than even the byte offsets of that row.
An example is the following search query: repo:^github\.com/0x03f3/php-emoji-reverse-shell$ 😍 type:symbol
If you inspect the link (ending in ?L3:72-3:76), it describes offsets from 72 to 76. However the byte size of this line does not exceed 72:
So this is unfortunately an issue from the backend that we can not easily work around with.
Problem 2: Emojis in symbol name crash the program.
Symbol search was also broken in a different way. Since the symbol name was part of the serialized preview payload but can contain emojis, it was not deserializing properly in the Java side and would cause the app to crash.
I fixed this issue by applying the same base64 encoding that we already apply to the file content to the symbol name.
Test plan
One emoji before the match r:leonardomso/33-js-concepts test
Multiple emoji before the match NestJs + TypeScript + TypeORM + Redis + MySql + Vue + Element-UI
Regular results console.log
Emoji symbol result repo:^github\.com/0x03f3/php-emoji-reverse-shell$ 😍 type:symbol
Note: As explained above, the highlight offsets for these are unfortunately broken for lines that contain non ASCII chars.
