Fix duplicated completion candidates to solve #6

[dezzw]

Description
This PR fixes an issue where the language server would return duplicate completion items for the same identifier. The problem was particularly noticeable with identifiers like broken-function appearing multiple times in completion suggestions.

Root Cause
The find-available-references-for function was collecting all references to an identifier without deduplication, resulting in multiple identical completion items when an identifier was defined in several places or imported from multiple sources.

Solution
Applied ordered-dedupe to filter identifier references before converting them to completion items:

• Added deduplication at the source of available references in the completion function
• Used the eq? equality predicate on identifier-reference-identifier to identify duplicates
• Applied the fix for both normal reference gathering and the prefix-filtered case

Implementation

[whole-list
        (if (null? target-index-node)
          '()
          (ordered-dedupe
            (if (equal? "" prefix)
              (find-available-references-for document target-index-node)
              (filter 
                (lambda (candidate-reference) 
                  (string-prefix? prefix (symbol->string (identifier-reference-identifier candidate-reference))))
                (find-available-references-for document target-index-node)))
            (lambda (a b)
              (eq? (identifier-reference-identifier a)
                   (identifier-reference-identifier b)))))]

Benefits

• Cleaner completion results without duplicates
• Better user experience with more concise completion lists
• Maintains original ordering via ordered-dedupe
• More efficient handling of completion requests

Related Issues
Fixes #6

[ufo5260987423]

Well, I don't think you can use ordered-dedupe here... because the result of find-available-references is not ordered.
And, you may do more work to find out : why here're so many duplicate results....
The root cause always is more important than a temporary patch.

[dezzw]

Thank you for the valuable feedback. You're absolutely right that we need to focus on the root cause rather than applying temporary patches. After analyzing the debug-recursion.log (1M+ lines), I might identify the fundamental issues causing the exponential duplication:

Root Cause Analysis

The core problem could be in find-available-references-for's recursive design:

For debug usage, I restructure the first case of the function find-available-references-for as

 [(document current-index-node)
      (let* ([local (index-node-references-import-in-this-node current-index-node)]
          [local-identifiers (map identifier-reference-identifier local)]
          [exclude (index-node-excluded-references current-index-node)]
          [parent-refs (if (null? (index-node-parent current-index-node))
                        (document-ordered-reference-list document) 
                        (find-available-references-for document (index-node-parent current-index-node)))]
          [filtered-parent-refs (filter 
                                  (lambda (reference)
                                    (not (member (identifier-reference-identifier reference) local-identifiers)))
                                  parent-refs)]
          [combined-refs (append local filtered-parent-refs)])
        (with-output-to-file "debug-ref-sources.log"
          (lambda ()
            (display "=== Reference Sources Debug ===\n")
            (display (format "Local refs count: ~a\n" (length local)))
            (display (format "Parent refs count: ~a\n" (length parent-refs)))
            (display (format "Filtered parent refs count: ~a\n" (length filtered-parent-refs)))
            (display (format "Combined refs count: ~a\n" (length combined-refs)))
            (display (format "Final refs count after exclude filter: ~a\n" 
                      (length (filter (lambda (reference) (not (member reference exclude))) combined-refs))))
            (display "\nLocal ref identifiers (first 10):\n")
            (let loop ([refs local] [count 0])
              (if (and (not (null? refs)) (< count 10))
                (begin
                  (display (format "  ~a\n" (identifier-reference-identifier (car refs))))
                  (loop (cdr refs) (+ count 1)))))
            (display "\nParent ref identifiers (first 10):\n")
            (let loop ([refs parent-refs] [count 0])
              (if (and (not (null? refs)) (< count 10))
                (begin
                  (display (format "  ~a\n" (identifier-reference-identifier (car refs))))
                  (loop (cdr refs) (+ count 1)))))
            (display "\n"))
          'append)
        (filter
          (lambda (reference) (not (member reference exclude)))
          combined-refs))]

Based on the log, I assume the following reasons:

1. Exponential Reference Multiplication

• Each recursive call up the parent chain re-includes all references from lower levels through append
• No cycle detection prevents infinite loops in complex inheritance/import chains
• Result: References get duplicated 2^n times where n is recursion depth

Evidence from Debug Log:

DUPLICATES DETECTED in recursive result:
  zero?: 11 times
  write-char: 11 times  
  with-syntax: 11 times

Results growing from ~1,575 → 2,265 → 10,412 references per call.

Also, this could be the reason why the server repeatedly cancels/timeouts some requests of textDocument/documentSymbol:

{"jsonrpc":"2.0","id":20,"method":"textDocument/documentSymbol","params":{"textDocument":{"uri":"file:///home/dez/scheme-langserver/test.ss"}}}

{"jsonrpc":"2.0","id":19,"result":[{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"

{"jsonrpc":"2.0","id":19,"error":{"code":-32800,"message":"textDocument\/documentSymbol"}}

{"jsonrpc":"2.0","method":"$/cancelRequest","params":{"id":20}}

2. Ineffective Deduplication Strategy

• Current filtering only removes based on local exclusion lists
• No global deduplication across the entire recursion chain

Evidence from Debug Log:

=== Reference Sources Debug ===
Local refs count: 0
Parent refs count: 3863
Filtered parent refs count: 3863
Combined refs count: 3863
Final refs count after exclude filter: 3863

Local ref identifiers (first 10):

Parent ref identifiers (first 10):
  $primitive
  $primitive
  $system
  $system
  &assertion
  &assertion
  &assertion
  &condition
  &condition
  &condition

Whole-list count: 40
Duplicated identifiers in whole-list:
  broken-function: 33 times
  brok: 7 times

The analysis may not be correct and needs to be explored deeper, but this is what I can find out so far, and no ideal/effective fix yet.

[ufo5260987423]

You should minimize your debug-recursion.log and this approach may help find out the real cause. Because find-available-references-for is triggered in many scenarios in scheme-langserver. And only those directly correspond to #6 are truely related to the bug.

[dezzw]

The original duplications appear both in textDocument/completion and textDocument/documentSymbol, which shows in the scheme-lsp.log,

Logs of showing textDocument/documentSymbol duplicates:

{"jsonrpc":"2.0","id":20,"method":"textDocument/documentSymbol","params":{"textDocument":{"uri":"file:///home/dez/scheme-langserver/test.ss"}}}

{"jsonrpc":"2.0","id":19,"result":[{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}},{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"

Logs of showing textDocument/completion duplicates:

{"jsonrpc":"2.0","id":4,"method":"textDocument/completion","params":{"textDocument":{"uri":"file:///home/dez/scheme-langserver/test.ss"},"position":{"line":13,"character":12},"context":{"triggerKind":1}}}
{"jsonrpc":"2.0","id":4,"result":[{"label":"brobroken","insertText":"n","sortText":"brobroken"},{"label":"brobroken","insertText":"n","sortText":"brobroken"},{"label":"brobroken","insertText":"n","sortText":"brobroken"}]}

The problem should be resolved by ending with scheme-lsp.log

{"jsonrpc":"2.0","id":3,"result":[{"label":"b","insertText":"","sortText":"b"},{"label":"base-exception-handler","insertText":"ase-exception-handler","sortText":"base-exception-handler"},{"label":"begin","insertText":"egin","sortText":"begin"},{"label":"bignum?","insertText":"ignum?","sortText":"bignum?"},{"label":"binary-port-input-buffer","insertText":"inary-port-input-buffer","sortText":"binary-port-input-buffer"},{"label":"binary-port-input-count","insertText":"inary-port-input-count","sortText":"binary-port-input-count"},{"label":"binary-port-input-index","insertText":"inary-port-input-index","sortText":"binary-port-input-index"},{"label":"binary-port-input-size","insertText":"inary-port-input-size","sortText":"binary-port-input-size"},{"label":"binary-port-output-buffer","insertText":"inary-port-output-buffer","sortText":"binary-port-output-buffer"},{"label":"binary-port-output-count","insertText":"inary-port-output-count","sortText":"binary-port-output-count"},{"label":"binary-port-output-index","insertText":"inary-port-output-index","sortText":"binary-port-output-index"},{"label":"binary-port-output-size","insertText":"inary-port-output-size","sortText":"binary-port-output-size"},{"label":"binary-port?","insertText":"inary-port?","sortText":"binary-port?"},{"label":"bitwise-and","insertText":"itwise-and","sortText":"bitwise-and"},{"label":"bitwise-arithmetic-shift","insertText":"itwise-arithmetic-shift","sortText":"bitwise-arithmetic-shift"},{"label":"bitwise-arithmetic-shift-left","insertText":"itwise-arithmetic-shift-left","sortText":"bitwise-arithmetic-shift-left"},{"label":"bitwise-arithmetic-shift-right","insertText":"itwise-arithmetic-shift-right","sortText":"bitwise-arithmetic-shift-right"},{"label":"bitwise-bit-count","insertText":"itwise-bit-count","sortText":"bitwise-bit-count"},{"label":"bitwise-bit-field","insertText":"itwise-bit-field","sortText":"bitwise-bit-field"},{"label":"bitwise-bit-set?","insertText":"itwise-bit-set?","sortText":"bitwise-bit-set?"},{"label":"bitwise-copy-bit","insertText":"itwise-copy-bit","sortText":"bitwise-copy-bit"},{"label":"bitwise-copy-bit-field","insertText":"itwise-copy-bit-field","sortText":"bitwise-copy-bit-field"},{"label":"bitwise-first-bit-set","insertText":"itwise-first-bit-set","sortText":"bitwise-first-bit-set"},{"label":"bitwise-if","insertText":"itwise-if","sortText":"bitwise-if"},{"label":"bitwise-ior","insertText":"itwise-ior","sortText":"bitwise-ior"},{"label":"bitwise-length","insertText":"itwise-length","sortText":"bitwise-length"},{"label":"bitwise-not","insertText":"itwise-not","sortText":"bitwise-not"},{"label":"bitwise-reverse-bit-field","insertText":"itwise-reverse-bit-field","sortText":"bitwise-reverse-bit-field"},{"label":"bitwise-rotate-bit-field","insertText":"itwise-rotate-bit-field","sortText":"bitwise-rotate-bit-field"},{"label":"bitwise-xor","insertText":"itwise-xor","sortText":"bitwise-xor"},{"label":"block-read","insertText":"lock-read","sortText":"block-read"},{"label":"block-write","insertText":"lock-write","sortText":"block-write"},{"label":"boolean=?","insertText":"oolean=?","sortText":"boolean=?"},{"label":"boolean?","insertText":"oolean?","sortText":"boolean?"},{"label":"bound-identifier=?","insertText":"ound-identifier=?","sortText":"bound-identifier=?"},{"label":"box","insertText":"ox","sortText":"box"},{"label":"box-cas!","insertText":"ox-cas!","sortText":"box-cas!"},{"label":"box-immutable","insertText":"ox-immutable","sortText":"box-immutable"},{"label":"box?","insertText":"ox?","sortText":"box?"},{"label":"break","insertText":"reak","sortText":"break"},{"label":"break-handler","insertText":"reak-handler","sortText":"break-handler"},{"label":"broken-function","insertText":"roken-function","sortText":"broken-function"},{"label":"buffer-mode","insertText":"uffer-mode","sortText":"buffer-mode"},{"label":"buffer-mode?","insertText":"uffer-mode?","sortText":"buffer-mode?"},{"label":"bwp-object?","insertText":"wp-object?","sortText":"bwp-object?"},{"label":"bytes-allocated","insertText":"ytes-allocated","sortText":"bytes-allocated"},{"label":"bytes-deallocated","insertText":"ytes-deallocated","sortText":"bytes-deallocated"},{"label":"bytevector","insertText":"ytevector","sortText":"bytevector"},{"label":"bytevector->immutable-bytevector","insertText":"ytevector->immutable-bytevector","sortText":"bytevector->immutable-bytevector"},{"label":"bytevector->s8-list","insertText":"ytevector->s8-list","sortText":"bytevector->s8-list"},{"label":"bytevector->sint-list","insertText":"ytevector->sint-list","sortText":"bytevector->sint-list"},{"label":"bytevector->string","insertText":"ytevector->string","sortText":"bytevector->string"},{"label":"bytevector->u8-list","insertText":"ytevector->u8-list","sortText":"bytevector->u8-list"},{"label":"bytevector->uint-list","insertText":"ytevector->uint-list","sortText":"bytevector->uint-list"},{"label":"bytevector-compress","insertText":"ytevector-compress","sortText":"bytevector-compress"},{"label":"bytevector-copy","insertText":"ytevector-copy","sortText":"bytevector-copy"},{"label":"bytevector-copy!","insertText":"ytevector-copy!","sortText":"bytevector-copy!"},{"label":"bytevector-fill!","insertText":"ytevector-fill!","sortText":"bytevector-fill!"},{"label":"bytevector-ieee-double-native-ref","insertText":"ytevector-ieee-double-native-ref","sortText":"bytevector-ieee-double-native-ref"},{"label":"bytevector-ieee-double-native-set!","insertText":"ytevector-ieee-double-native-set!","sortText":"bytevector-ieee-double-native-set!"},{"label":"bytevector-ieee-double-ref","insertText":"ytevector-ieee-double-ref","sortText":"bytevector-ieee-double-ref"},{"label":"bytevector-ieee-double-set!","insertText":"ytevector-ieee-double-set!","sortText":"bytevector-ieee-double-set!"},{"label":"bytevector-ieee-single-native-ref","insertText":"ytevector-ieee-single-native-ref","sortText":"bytevector-ieee-single-native-ref"},{"label":"bytevector-ieee-single-native-set!","insertText":"ytevector-ieee-single-native-set!","sortText":"bytevector-ieee-single-native-set!"},{"label":"bytevector-ieee-single-ref","insertText":"ytevector-ieee-single-ref","sortText":"bytevector-ieee-single-ref"},{"label":"bytevector-ieee-single-set!","insertText":"ytevector-ieee-single-set!","sortText":"bytevector-ieee-single-set!"},{"label":"bytevector-length","insertText":"ytevector-length","sortText":"bytevector-length"},{"label":"bytevector-s16-native-ref","insertText":"ytevector-s16-native-ref","sortText":"bytevector-s16-native-ref"},{"label":"bytevector-s16-native-set!","insertText":"ytevector-s16-native-set!","sortText":"bytevector-s16-native-set!"},{"label":"bytevector-s16-ref","insertText":"ytevector-s16-ref","sortText":"bytevector-s16-ref"},{"label":"bytevector-s16-set!","insertText":"ytevector-s16-set!","sortText":"bytevector-s16-set!"},{"label":"bytevector-s24-ref","insertText":"ytevector-s24-ref","sortText":"bytevector-s24-ref"},{"label":"bytevector-s24-set!","insertText":"ytevector-s24-set!","sortText":"bytevector-s24-set!"},{"label":"bytevector-s32-native-ref","insertText":"ytevector-s32-native-ref","sortText":"bytevector-s32-native-ref"},{"label":"bytevector-s32-native-set!","insertText":"ytevector-s32-native-set!","sortText":"bytevector-s32-native-set!"},{"label":"bytevector-s32-ref","insertText":"ytevector-s32-ref","sortText":"bytevector-s32-ref"},{"label":"bytevector-s32-set!","insertText":"ytevector-s32-set!","sortText":"bytevector-s32-set!"},{"label":"bytevector-s40-ref","insertText":"ytevector-s40-ref","sortText":"bytevector-s40-ref"},{"label":"bytevector-s40-set!","insertText":"ytevector-s40-set!","sortText":"bytevector-s40-set!"},{"label":"bytevector-s48-ref","insertText":"ytevector-s48-ref","sortText":"bytevector-s48-ref"},{"label":"bytevector-s48-set!","insertText":"ytevector-s48-set!","sortText":"bytevector-s48-set!"},{"label":"bytevector-s56-ref","insertText":"ytevector-s56-ref","sortText":"bytevector-s56-ref"},{"label":"bytevector-s56-set!","insertText":"ytevector-s56-set!","sortText":"bytevector-s56-set!"},{"label":"bytevector-s64-native-ref","insertText":"ytevector-s64-native-ref","sortText":"bytevector-s64-native-ref"},{"label":"bytevector-s64-native-set!","insertText":"ytevector-s64-native-set!","sortText":"bytevector-s64-native-set!"},{"label":"bytevector-s64-ref","insertText":"ytevector-s64-ref","sortText":"bytevector-s64-ref"},{"label":"bytevector-s64-set!","insertText":"ytevector-s64-set!","sortText":"bytevector-s64-set!"},{"label":"bytevector-s8-ref","insertText":"ytevector-s8-ref","sortText":"bytevector-s8-ref"},{"label":"bytevector-s8-set!","insertText":"ytevector-s8-set!","sortText":"bytevector-s8-set!"},{"label":"bytevector-sint-ref","insertText":"ytevector-sint-ref","sortText":"bytevector-sint-ref"},{"label":"bytevector-sint-set!","insertText":"ytevector-sint-set!","sortText":"bytevector-sint-set!"},{"label":"bytevector-truncate!","insertText":"ytevector-truncate!","sortText":"bytevector-truncate!"},{"label":"bytevector-u16-native-ref","insertText":"ytevector-u16-native-ref","sortText":"bytevector-u16-native-ref"},{"label":"bytevector-u16-native-set!","insertText":"ytevector-u16-native-set!","sortText":"bytevector-u16-native-set!"},{"label":"bytevector-u16-ref","insertText":"ytevector-u16-ref","sortText":"bytevector-u16-ref"},{"label":"bytevector-u16-set!","insertText":"ytevector-u16-set!","sortText":"bytevector-u16-set!"},{"label":"bytevector-u24-ref","insertText":"ytevector-u24-ref","sortText":"bytevector-u24-ref"},{"label":"bytevector-u24-set!","insertText":"ytevector-u24-set!","sortText":"bytevector-u24-set!"},{"label":"bytevector-u32-native-ref","insertText":"ytevector-u32-native-ref","sortText":"bytevector-u32-native-ref"},{"label":"bytevector-u32-native-set!","insertText":"ytevector-u32-native-set!","sortText":"bytevector-u32-native-set!"},{"label":"bytevector-u32-ref","insertText":"ytevector-u32-ref","sortText":"bytevector-u32-ref"},{"label":"bytevector-u32-set!","insertText":"ytevector-u32-set!","sortText":"bytevector-u32-set!"},{"label":"bytevector-u40-ref","insertText":"ytevector-u40-ref","sortText":"bytevector-u40-ref"},{"label":"bytevector-u40-set!","insertText":"ytevector-u40-set!","sortText":"bytevector-u40-set!"},{"label":"bytevector-u48-ref","insertText":"ytevector-u48-ref","sortText":"bytevector-u48-ref"},{"label":"bytevector-u48-set!","insertText":"ytevector-u48-set!","sortText":"bytevector-u48-set!"},{"label":"bytevector-u56-ref","insertText":"ytevector-u56-ref","sortText":"bytevector-u56-ref"},{"label":"bytevector-u56-set!","insertText":"ytevector-u56-set!","sortText":"bytevector-u56-set!"},{"label":"bytevector-u64-native-ref","insertText":"ytevector-u64-native-ref","sortText":"bytevector-u64-native-ref"},{"label":"bytevector-u64-native-set!","insertText":"ytevector-u64-native-set!","sortText":"bytevector-u64-native-set!"},{"label":"bytevector-u64-ref","insertText":"ytevector-u64-ref","sortText":"bytevector-u64-ref"},{"label":"bytevector-u64-set!","insertText":"ytevector-u64-set!","sortText":"bytevector-u64-set!"},{"label":"bytevector-u8-ref","insertText":"ytevector-u8-ref","sortText":"bytevector-u8-ref"},{"label":"bytevector-u8-set!","insertText":"ytevector-u8-set!","sortText":"bytevector-u8-set!"},{"label":"bytevector-uint-ref","insertText":"ytevector-uint-ref","sortText":"bytevector-uint-ref"},{"label":"bytevector-uint-set!","insertText":"ytevector-uint-set!","sortText":"bytevector-uint-set!"},{"label":"bytevector-uncompress","insertText":"ytevector-uncompress","sortText":"bytevector-uncompress"},{"label":"bytevector=?","insertText":"ytevector=?","sortText":"bytevector=?"},{"label":"bytevector?","insertText":"ytevector?","sortText":"bytevector?"}]}

{"jsonrpc":"2.0","id":6,"result":[{"label":"brok","insertText":"","sortText":"brok"},{"label":"broken-function","insertText":"en-function","sortText":"broken-function"}]}

{"jsonrpc":"2.0","id":7,"result":[{"name":"x","kind":13,"range":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}},"selectionRange":{"start":{"line":4,"character":16},"end":{"line":4,"character":17}}}]}

The duplications both in textDocument/completion and textDocument/documentSymbol are filtered.