What is GitHub Copilot?
GitHub Copilot is a tool developed by OpenAI and GitHub that aims to write code when given either already written code or natural language.
Copilot is based on OpenAI Codex, a LLM (Large Language Model) built on GPT-3 and trained on both natural language and code. Simply put, it is a tool that uses a pre-trained machine learning model to try to complete the text and code already entered.
In this TechUp, I will illustrate how well, but also how poorly something like this can work.
Copilot was announced in June 2021 and has been available to all GitHub users since October 20, 2021. It was initially free to use, but Copilot now costs $10 per month.
Requirements
-
Supported IDE: To use GitHub Copilot you need a supported IDE in addition to a GitHub account. Besides the most common IDEs like Visual Studio Code and IntelliJ, Atom, Emacs and Vim are currently supported as well.
-
Connection to GitHub: The IDEs must be connected to GitHub in order for Copilot to send the entered text and code snippets to Copilot.
-
GitHub Copilot Account: Of course you need to sign up for GitHub Copilot, currently you can take advantage of a 2 month free trial before it becomes chargeable.
My experience with GitHub Copilot
I tested Copilot in IntelliJ, as well as in VS-Code, the experiences were different depending on the IDE.
In VS code, Copilot’s completions seemed quite snappy, in Intellij on the other hand, Copilot’s code completions overrode IntelliSense’s completions, sometimes resulting in a very unpleasant user experience. The quality of the suggestions differed significantly depending on the situation. I will illustrate what working with Copilot looks like with a few examples.
GitHub Copilot basically tries to further complete the code that has already been entered. The input of the LLM used by Copilot is therefore the already existing code, the output of the model is always the next token, or the next code snippet. Since these suggestions of course depend directly on the readability and the given context of the input, the results here differ considerably. By pressing Tab, Copilots suggestions can be inserted directly into the code.
Figure: Successful example
One error that LLMs can have due to this kind of completion is that recursion occurs; the input in this case generates an output, which taken as input suggests the same output again. When working with Copilot, such incidents occur from time to time. Since this Problem can be hard to capture, I was only able to get a screenshot when trying it in german:
Figure: Faulty example (german)
Especially in situations where a certain basic structure is standard, such as in tests, (standard) configurations or well-structured code such as in CSS or HTML, Copilot works optimally. The suggestions regarding translations of text modules (i18n files) were very pleasant.
In general, it can be stated that Copilot can especially complete repetitive code structures well.
Figure: Repetitive code structure example with i18n
Figure: Repetitive code structure example with HTML
When it comes to meaningful completion of more sophisticated logical structures, the quality differs significantly. Especially if you are currently looking for a good solution yourself and use standard tools from e.g. Intellij to find the possible functions on an object, it often happens that Copilot tries to invasively complete the code based on the context. In the worst case, this blocks the workflow completely.
The current solution that has helped me in these situations is to disable Copilot for the time being until I find a solution. This is possible in Intellij as well as VS code with two mouse clicks.
On the other hand, the completion of structures like lambda expressions in Java is very good, as well as the completion of bracket substitution and semicolons, which Intellij on its own does not offer you in this form. Sometimes suggestions are partially correct, so you accept and adjust them briefly, which for me personally didn’t really affect the workflow.
I came across one use case of Copilot around New Year’s, when I spent a few days working on text compression. I wanted to analyze a text with Python for symbol frequencies and a few other statistics, but since my Python skills were a bit rusty, several common commands didn’t occur to me at first. Since Copilot was enabled, I came up with the idea to first write my intention as a comment and then complete the individual steps with Copilot:
Of course, in this case it was very simple syntax, yet Copilot sped up the workflow significantly, as I didn’t have to leave the IDE or pick up another tool to find the commands I needed. You can clearly see from this example that completion is useful, but not necessarily perfectly accurate, as Copilot suggested reading in the wrong file.
Another interesting insight came up while writing this TechUp, which I wrote in VS Code with Copilot enabled. Some of the suggestions were helpful, some were misleading, and some made you wonder at what point you might be plagiarizing without realizing it.
What I found very interesting were suggestions in which keywords appeared that could certainly be good entry points into this same topic for someone new to it. Like this example:
Pros and cons
Technologies such as Copilot are currently still in their infancy, but offer a very interesting insight into future developments in terms of the possibilities of using LLM and AI-generated code, as well as text.
In terms of the usefulness of this tool in active software development, I currently see the advantages in speeding up repetitive or highly structured tasks, which usually do not require much creative input. Depending on the IDE, I found Copilot to be too invasive, which could stall the workflow.
Besides the active development, I see the interaction with tools like Copilot at the current time as very interesting and instructive, because on the one hand it creates an understanding of the underlying technology and its limitations, but on the other hand it can also generate new ideas and solutions by creatively dealing with the possibilities of such a tool.
The current popularity of chat GPT for completing and generating code is a good example of the possibilities that arise from the use of AI-generated code, which enable a completely different approach to interacting with AI.
Conclusion
For me, a fitting comparison to the current state of Copilot is the emergence of the first car navigation devices. Each week there were newspaper articles about people who trusted their sat nav and found themselves either on the wrong side of the highway or in a pond. Copilot, in my opinion, is currently in exactly this position. As a user, you have to rely on your own critical examination of its suggestions in order to take advantage of this tool.
Furthermore, while Copilot can take simple code off your hands, your own thinking and understanding of the underlying concepts is definitely a prerequisite to working with Copilot.
Tools like Copilot will certainly not replace developers in the future, but I can see a slight increase in productivity at the moment. I will definitely continue to follow further developments in this field and am excited to see what the future brings. Stay tuned! 💪
If you’re interested in the topic of AI, checkout Stefans TechUp: The basics of neural networks - easily explained 🔥