Code intelligence tracking issue 3.13

[chrismwendt]

Theme Increase user confidence in code intelligence provided by LSIF and make setup easier.

Sourcegraph project roadmap

Chris

• https://github.com/sourcegraph/sourcegraph/pull/7909 Atomic migrations (on-call work)
• https://github.com/sourcegraph/infrastructure/commit/09f43d04b6fef785936752a0272416dcce599d2d fixed pod-tmp-gc which was causing numerous out-of-disk errors (on-call work)
• https://sourcegraph.atlassian.net/browse/SG-17 Customer needs help publishing to private extension registry
• Add extension copy command src-cli#82 Add src extension copy command (for customer 👆)
• Dart LSIF indexer
  • Fork crossdart
  • sourcegraph/lsif-dart@30c192a Upgrade to latest Dart SDK 2.7
  • sourcegraph/lsif-dart@32931fa LSIF output mode: initial structure
  • sourcegraph/lsif-dart@bb70758 LSIF output mode: skeleton for project and docs
  • sourcegraph/lsif-dart@809e542 LSIF output mode: emit hovers/defs/refs
  • sourcegraph/lsif-dart@32b1402 Fix --dart-sdk argument
  • sourcegraph/lsif-dart@60edf10 Fill in hover with docstrings
  • sourcegraph/lsif-dart@dbed91f Fix hover highlighting by computing token lengths
  • Recognize more tokens
  • Recognize parameters
  • https://github.com/sourcegraph/customer/issues/5 Delivery plan
• https://github.com/sourcegraph/lsif-demos sample code that has LSIF data uploaded for demo purposes
• Make @sourcegraph/code-intel the sole CODEOWNERS of code intel extensions code (Slack thread):
  • CODEOWNERS: make @sourcegraph/code-intel the sole owners code-intel-extensions#204
  • CODEOWNERS: make @sourcegraph/code-intel the sole owners sourcegraph-go#120
  • CODEOWNERS: make @sourcegraph/code-intel owners sourcegraph-typescript#312
  • chore: CODEOWNERS: make @sourcegraph/code-intel the sole owners lsp-client#81
• fix: do not register providers inside of withConnection lsp-client#42 Fix infinite loop due to duplicate registrations
• Annotator lsif-test#9 LSIF annotator to aide dev/debugging sourcegraph/lsif-test@ad820e3

Eric

• Update code intel regression test suites
  • https://github.com/sourcegraph/sourcegraph/pull/7874
  • https://github.com/sourcegraph/sourcegraph/pull/7890
  • https://github.com/sourcegraph/sourcegraph/pull/7845
• Replace repository names with ids in lsif-server (issue LSIF server repository names #6278)
  • https://github.com/sourcegraph/sourcegraph/pull/7828
  • https://github.com/sourcegraph/sourcegraph/pull/7896
  • https://github.com/sourcegraph/sourcegraph/pull/7910
• Add additional dev documentation for LSIF
  • https://github.com/sourcegraph/sourcegraph/pull/7931
• Add precise code intel action telemetry (RFC 100)
  • Get approvals from @nicksnyder and @dadlerj on RFC
  • Add telemetry command to sourcegraph for extensions https://github.com/sourcegraph/sourcegraph/pull/7939
  • Validate json arguments https://github.com/sourcegraph/sourcegraph/pull/7989 and https://github.com/sourcegraph/sourcegraph/pull/7995
  • Send arguments back to logEvent https://github.com/sourcegraph/sourcegraph/pull/7988
  • Add db query methods to get event counts by period https://github.com/sourcegraph/sourcegraph/pull/7992
  • Add db query methods to get latency percentiles by period https://github.com/sourcegraph/sourcegraph/pull/7994
  • Ensure ping payloads are not too big for request query string https://github.com/sourcegraph/sourcegraph/pull/7993
  • Calculate code intel usage from DB https://github.com/sourcegraph/sourcegraph/pull/8029
  • Add code intel usage data to GraphQL https://github.com/sourcegraph/sourcegraph/pull/8000
  • Add code intel usage data to ping requests https://github.com/sourcegraph/sourcegraph/pull/7942
  • Emit telemetry events in Emit telemetry events code-intel-extensions#193
  • Update BigQuery schema and ETL functions to consume new data
• Refactor code intel extensions to make adding events easier
  • Consolidate common logic Activation refactor code-intel-extensions#198
  • Go extension Activation refactor sourcegraph-go#117
  • TypeScript extension Activation refactor sourcegraph-typescript#308
• Refactor lsp features in basic-code-intel
  • Follow Go extension Use lsp-client sourcegraph-go#36
  • Follow Typescript extension Use @sourcegraph/lsp-client sourcegraph-typescript#106
• Continuous delivery of extensions (RFC 106)
• Overlapping LSIF roots (RFC 107)

Michael

• Add basic code intelligence support for:
  • Objective-C, Enable search-based intelligence for Objective C. code-intel-extensions#197
  • GraphQL, Add basic code intelligence for GraphQL schema files. code-intel-extensions#206
  • Verilog and VHDL, Add basic code intelligence for Pascal, Verilog, and VHDL code-intel-extensions#200
  • Pascal, Add basic code intelligence for Pascal, Verilog, and VHDL code-intel-extensions#200
  • Groovy, Add basic code intelligence for Groovy files. code-intel-extensions#209
• Create a multi-language indexer that auto-detects project structure.
  • Write a draft of RFC 101 Precise Indexing Configuration

External

• C# LSIF indexer (Thorsten, slush project) too busy

[chrismwendt]

Week 2 summary

• Wrapped up on-call work (atomic DB migrations)
• Implemented hovers/defs/refs in the Dart LSIF indexer

Week 3 plan 2020-01-27

• Polish and fixes for the Dart LSIF indexer
• Notify customer that it's ready to try out
• Miscellaneous UX improvements to the C++ LSIF indexer

[efritz]

Week 2 summary

• Implemented all of the required (known) work for adding code intel usage data to database, graphql resolvers, and updatecheck (blocked on @dan's approval)
• Started exploratory work on semanticdb -> lsif converter (not prioritized, picked it up out of interest while blocked on other tasks)

Week 3 plan 2020-01-27

• Refactor basic code intel to have all logic for LSIF (remove logic from go/ts extensions)
• Drag code intel telemetry over the finish line

[creachadair]

Week 2 summary

• Implemented Objective C support for basic code intelligence.

Week 3 plan 2020-01-27

• Add the remaining basic intel languages on the roadmap (Groovy, Verilog, VHDL, Pascal, GraphQL).
• Research and write into the currently-empty skeleton RFC 101 (LSIF autoconfiguration).

[creachadair]

Week 3 summary

• Implemented basic code intelligence for GraphQL, Groovy, Pascal, Verilog, and VHDL.
• Wrote an initial draft of RFC 101 (WIP), Precise Indexing Configuration.

Week 4 plan 2020-02-03

• Out Monday, Tuesday, Wednesday (3–5 Feb).
• After returning: Address comments or concerns on the RFC 101 draft, schedule a review.

[chrismwendt]

Week 3 summary

• Dart: fixed some UX issues
• Dart: populated hover tooltips with docstrings
• In aid of the Great Refactor: fixed an infinite loop related to duplicate registrations
• Created an LSIF annotator that helps you debug ranges in LSIF dumps

Week 4 plan 2020-02-03

• Add defs/refs to lsif-annotate
• Add more node types to the Dart LSIF indexer (currently only supports simple identifiers)
• Notify customer that it's ready to try out

[efritz]

Week 3 summary

• Transformed basic-code-intel into a monorepo for code intel extensions
• Wrote RFC 106 to provide automatic+continuous delivery for extensions
• Wrote RFC 107 to fix oversight in LSIF root handling

Week 4 plan 2020-02-03

• Try to get https://github.com/sourcegraph/sourcegraph/pull/7942 approved by @dadlerj
• Update basic-code-intel extensions to use lspClient
• Add testing infrastructure to basic-code-intel

[efritz]

Week 4 summary

• Fixed analytics pipeline (after breaking it)
• Discussed RFC 107 to sanity check proposal for overlapping LSIF roots
• Continued on The Great Refactoring

Week 5 plan 2020-02-10

• Finish The Great Refactoring
• Implement RFC 107
• Get https://github.com/sourcegraph/sourcegraph/pull/8140 approved by @dadlerj

[creachadair]

Week 4 summary

• Addressed issues with basic code intel additions (GraphQL extension not enabled globally for all users on sourcegraph.com #8172, Definition of scalar type not found in GraphQL schema #8183, Enable basic code intel for more languages by default. #8314).
• Discussed and addressed most of the comments on the RFC 101 draft.
• Discussed and agreed with the RFC 107 draft.

Week 5 plan 2020-02-10

• Quick experiments to check whether the proposed RFC 101 implementation strategy is viable on popular CI stacks.
• Get RFC 101 approved (in some form) and start implementing its key features.
• Finish up code intel issues from last week (e.g., GraphQL hovers have no syntax highlighting #8185, GraphQL find references does not find reference to interface if type implements multiple interfaces #8186, Go to definition on GraphQL reference in schema pops up panel with results in Go code #8177).

[chrismwendt]

Week 4 summary

• Dart: added support for many more tokens and language constructs
• Dart: added definition preview when docstring is missing
• Dart: added to lsif.dev
• Dart: notified Workiva (following delivery plan)

Week 5 plan 2020-02-10

• Next item on the roadmap (will discuss in team sync)
• Address any feedback from Workiva

[uwedeportivo]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.13 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you