How GitHub Copilot really influences the efficiency of software development

In recent years, GitHub Copilot has established itself as a promising tool in software development. The idea that an AI-supported solution can automatically generate complex sections of code sounds appealing. The introduction of AI-supported tools such as Github Copilot is supposed to boost productivity in software development.

But what is the reality? Does Copilot actually increase developer productivity? We conducted an experiment to measure the real-world efficiency gains from using Github Copilot. In this article, we present our results and give an insight into the advantages and disadvantages of Copilot in practice.

What is GitHub Copilot?

GitHub Copilot is an AI-powered programming assistant that helps developers create code. Based on the context of the project and the lines of code written, GitHub Copilot automatically suggests the next line of code or even entire blocks of code. The tool uses the huge amount of publicly accessible code to recognise patterns and generate suitable solutions.

The experiment: Github Copilot in practice

In order to evaluate the increase in productivity through GitHub Copilot, we had two developers implement different user stories with and without Copilot. Our aim was to determine realistic efficiency gains and to check in which use cases Copilot is particularly useful.

Before deciding to test with Github Copilot, we investigated similar tools such as Duet AI (based on Google Gemini), CodeGeeX, and Jetbrain’s AI Assistant. In the end, we decided in favour of GitHub Copilot, the market leader at the time, as it seemed to be the most promising tool. Nevertheless, we would like to mention that there are also alternative tools that can be considered depending on your requirements and preferences.

GitHub Copilot Usecase 1: Adding a team member

As part of the app development for our summer party this year, we found an exciting use case for GitHub Copilot. One of our user stories was: As a participant in the summer event, I want to be able to add a team member to my group if someone doesn’t have a smartphone.

The requirements were clear: an easy way to add team members by name without exceeding the maximum team size of four people. The functionality also needed to be understandable for users without technical expertise. To implement this functionality, two of our developers worked on the same task – one with Copilot, the other without.

The result: the developer with Copilot needed around 25 per cent less time than his colleague without the AI support and a workload of 5.5 hours.

Quality comparison with and without GitHub Copilot

• Edge cases and error handling: GitHub Copilot also supported development in catching edge cases, error handling at the state management level, and API calls. A positive impact on code quality is possible, as Copilot pointed out aspects that the developer might have missed.
• UI (user interface) level: GitHub Copilot adapts to the developer’s style because existing code is sent along as context. There was hardly any influence on code quality at the UI level. Depending on the framework, the model is trained differently. The Flutter (Dart) framework is a young language and is developing rapidly, which is why outdated code suggestions can also occur.

GitHub Copilot Use Case 2: Displaying more information about the app

In the next use case, we focused on improving the transparency of an app. Our user story aimed to provide users with more information about the app and the developers behind it. This was based on complex and irregular data structures.

The result: When working with irregular data structures, as they were in this particular case, Github Copilot was unable to achieve a significant increase in efficiency and had little impact on the productivity of the software developers.

GitHub Copilot Use Case 3: Display a list of all Pokémon names

In this GitHub Copilot use case, we set ourselves the task of creating a Pokémon app. A central function was a dynamic list that displays more and more Pokémon names as you scroll. When clicking on a name, a detailed overview with all relevant information should appear. We used GitHub Copilot to realise this part of the project.

The result: With GitHub Copilot, we were able to speed up development by around 20 per cent and save a total of 1.25 hours.

Lessons learnt from using GitHub Copilot in software development

The integration of GitHub Copilot into our development processes has led to a number of significant improvements:

• Increased efficiency: Automated code completion and the generation of code snippets have reduced the average development time per project by 20-25 per cent. This allows our developers to concentrate on more complex tasks and deliver high-quality software more quickly.
• Improved error handling: Copilot has proven to be particularly helpful in identifying and fixing errors and handling edge cases. The AI-supported analysis of the code helps to identify and rectify potential problems at an early stage.
• High developer satisfaction: The positive feedback from our developers emphasises the added value of Copilot. They appreciate the intuitive operation, the time savings, and the overall improved developer experience.

Strengths of GitHub Copilot

• Productivity boost: Fast generation of boilerplate code and automatic completion of functions.
• Real-time suggestions: Inline recommendations make programming easier and reduce search effort.
• Code improvements: GitHub Copilot can be asked for optimisation options in the code.
• Documentation: Automatic creation of code documentation for functions.
• Test data and unit tests: Copilot can generate test data and unit tests and improve test coverage.
• Standard tasks: Excellent for routine programming tasks.
• New programming languages: Makes it easier to get started with new programming languages if knowledge of a technology stack is already available.
• Code explanations: Useful for junior developers, QA engineers, or similar who need explanations for the code.
• Error prevention: Helps with catching edge cases and error handling.

Limitations of GitHub Copilot in software development

Like any AI model, GitHub Copilot is heavily dependent on its training data. In the case of rare or very specific data structures that are not included in the training data, the model can only provide limited support. In such cases, the AI can only achieve minimal increases in productivity. Outdated information with which the AI was trained can also lead to outdated code suggestions or practices that are no longer recommended. Complex tasks remain the domain of human developers – this is where GitHub Copilot reaches its limitations.

Conclusion: How GitHub Copilot really affects development time

Our tests show that GitHub Copilot reduces development time for standardised tasks by an average of 20-25 percent. However, not every task is equally affected by this increase in efficiency. The increase in efficiency is highly dependent on the complexity of the task.

GitHub Copilot is definitely a helpful tool for developers, but it is no substitute for human expertise. It therefore still needs capable developers who understand what they are doing.

Comparison with GitHub’s own experiment

GitHub itself conducted a similar experiment and published its results. Their study showed that the developers who used Copilot not only experienced an increase in productivity, but also greater satisfaction with their work. This is in line with our results, which also point to significant time savings and improved developer satisfaction.

Is the introduction of GitHub Copilot worthwhile in practice?

Based on our experiments, we can recommend the introduction of GitHub Copilot in many use cases. The significant time savings and support for error handling make GitHub Copilot a valuable tool for developers. However, the limitations and specific use cases where Copilot is less effective should be considered.

Advantages:

• Time savings: Up to 25 per cent faster development time.
• Improved error handling: Support for handling edge cases and API errors.
• Cost efficiency: The time savings can lead to lower costs in many projects.
• Satisfied developers.

Limitations and risks:

• Licence fees: The use of Copilot is subject to licence fees, which is why a cost-benefit analysis should be carried out to assess its cost-effectiveness. Despite the costs of integrating Copilot, we believe that the benefits of increased efficiency and improved code quality outweigh the costs in many projects.
• Risk of erroneous suggestions: This is an AI system in which errors can also occur. The proposals are based on open-source code, which may contain errors, bad practices, or outdated information.
• Limited usability: Irregular data structures: The AI may be less useful with irregular or specific data structures.
• Over-reliance: Developers may rely too much on Copilot and not review suggestions sufficiently.

By using GitHub Copilot in a targeted manner and being aware of its limitations, you can utilise the full potential of this tool and significantly optimise your software development projects.