Company case
GitHub: Code assistants
What the published sources confirm
GitHub ran a controlled experiment with 95 professional developers writing an HTTP server in JavaScript, with and without GitHub Copilot, plus a survey of more than 2,000 developers.
Business problem and AI concept
- Problem
- Developers spend time on repetitive code; measuring productivity is difficult.
- Concept
- Context-aware code completion, and how to measure its effect with a controlled experiment.
Reported results
The Copilot group completed the task 55% faster on average (95% confidence interval 21%–89%); completion rate was 78% versus 70%.
GitHub blog, Sep 2022. Caveat: One task in one language; does not measure long-term code quality.
Suggested workflow
Written for learning, not the company's system. This flow shows how you could build something similar. It is not a description of GitHub's private architecture.
Step 1: Read permitted repository context near the cursor
Step 2: Retrieve related symbols and files
Step 3: Generate a suggestion with a model
Step 4: Developer accepts, edits or rejects
Step 5: Compile and run tests in a sandbox
Step 6: Track acceptance and test results
Suggested technologies and alternatives
| Layer | Suggested | Alternatives |
|---|---|---|
| Context | Roslyn for C# symbols | tree-sitter |
| Retrieval | Embeddings over code + docs | BM25 over identifiers |
| Sandbox | Docker with no network | Ephemeral CI runner |
Small example
- Input
- Method signature `decimal CalculateVat(Order order)` and existing tests.
- Output
- Suggested implementation plus two new unit tests; tests run and pass in a sandbox.
Rebuild it yourself
Build a repository Q&A and test-suggestion assistant for a small sample .NET solution.
Definition of done
- Answers cite file paths and lines
- Generated tests compile and run in an isolated container
- Measure on 10 tasks with and without the tool
Security and privacy risks
- Leaking private code to third parties
- Insecure suggested code
- Licence issues in suggestions
Sources
Listed by title and publisher. This guide contains no external links; search for the title if you want to read the original.
- Research: quantifying GitHub Copilot's impact on developer productivity and happiness. GitHub blog, 7 Sep 2022. Company research.
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (Peng et al.). arXiv, 2023. Research paper.