Are algorithmic coding tests dead? An inside look at ChatGPT & GitHub Copilot

The ChatGPT launch has resulted in an outburst of discussions on social media and various articles and comments in the traditional media. One of the areas that generated a lot of interest was the potential use of ChatGPT in software development and how coding tests – especially for junior positions – might become irrelevant in the upcoming future.

Contrary to the common sentiment in the industry, we at DevSkiller, are very excited about the possibilities that ChatGPT and other AI coding solutions like GitHub Copilot offer in testing and evaluating software development skills. Through this article, we will discuss why we are so enthusiastic about the potential of AI, and how it can help us better evaluate software development skills.

What is ChatGPT?

ChatGPT, or OpenAI’s Generative Pre-trained Transformer, is a powerful form of Artificial Intelligence (AI) created by OpenAI. It was developed as a part of their mission to develop advanced AI technologies for everyone to use. ChatGPT uses natural language processing and deep learning to generate text that is nearly indistinguishable from that written by a human.

ChatGPT utilizes natural language processing (NLP) and deep learning algorithms to generate text that can mimic real-life conversations. By taking into account the context of a conversation, it can generate text that reads like it was written by a human. For example, if asked what the weather is like, it can answer with a response that seems natural and logical.

How does ChatGPT work?

ChatGPT works by taking input from the user, analyzing it, and generating a response based on its understanding of the conversation. It can also access other sources of data such as past conversations, stored information, and even outside sources like the internet. By combining all these data sources, ChatGPT can respond to queries in a way that is more intelligent and accurate than ever before.

What makes ChatGPT so powerful is its ability to learn and adapt quickly. The more conversations it has, the better it gets at understanding the context of each conversation, so it can respond accurately and appropriately.

What is GitHub Copilot

GitHub Copilot is a pair programming feature offered by the popular version control platform, GitHub. It provides an AI-assisted approach to code collaboration that can help speed up development cycles. Pair programming is a software development technique in which two programmers work together on one project at the same time. With GitHub Copilot, the AI takes over many of the more tedious and mundane tasks associated with coding, freeing developers up to focus on higher-level design considerations.

GitHub Copilot uses artificial intelligence to identify common patterns and patterns unique to each developer’s workflow. This helps automate repetitive tasks such as code reviews and refactoring, allowing developers to make changes quickly without having to manually review every single line of code. Additionally, it enables two people to write code in parallel with real-time updates from both parties reflected across their screens so they can easily collaborate even if they are not in the same physical space.

By using AI-driven pair programming, teams are able to significantly improve efficiency and increase their productivity without compromising quality. Not only does this benefit developers directly but also customers who benefit from shorter delivery times and faster problem-solving cycles. Moreover, since much of the grunt work has been automated, there is more time for creativity and experimentation which may ultimately lead to higher-quality products overall.

ChatGPT vs coding tests

“It is unlikely that OpenAI will kill coding tests. While OpenAI is a leading organization in the field of artificial intelligence and machine learning, it is not focused on replacing coding tests. Coding tests are a common way for employers to evaluate the programming skills of job applicants, and they are not likely to go away anytime soon. Additionally, while machine learning and AI can automate some tasks, they cannot yet fully replace the critical thinking and problem-solving skills that are essential for writing good code.”

The statement above is not us. It’s ChatGPT 3 answering a simple question:

But jokes aside…

ChatGPT vs DevSkiller’s RealLifeTesting™

While most comments in the IT industry focused on the negative effects of ChatGPT launch on IT skills assessment companies, we at DevSkiller are really enthusiastic about the potential of AI solutions. Why? The answer is simple: our RealLifeTesting™ methodology. In short: we stay away from even complex algorithmic tasks that were asked and solved hundreds of times. Instead, we endorse problem-based, real-life assignments. Such tasks require unique industry experience and problem solving, rather than creative copy-pasting.

From the beginning, the key advantage of the DevSkiller platform was that it was supposed to mirror a real-life work environment, without any limitations. Since all professional developers – including members of the DevSkiller team – use solutions that speed up the code creation process on a daily basis (e.g. GitHub Copilot), we are fine with the tested candidates doing it as well. Of course, only as long as they aren’t cheating, by blatantly copying the same solution from one another. Which we also try to discover using advanced anti-plagiarism detection engine.

OpenAI will definitely kill algorithmic challenges. But to solve DevSkiller work-sample tests, it would have to read and understand the whole codebase, dependencies, and  business requirements and then implement a solution. Instead of asking about trivial academic stuff like implementing Fibonacci or inverting binary trees, our tests are based on full-blown projects. 90% of our tasks are GPT-proof while the competitors have most of the tasks that can be easily solved with OpenAI.

Jakub Kubryński, DevSkiller CEO & Founder

In general, we strongly believe that no tool can replace human creativity and problem-solving skills and that’s what we aim to be testing. Sure, AI tools can generate large chunks of code, but the overall design, structure, and other large-scale constructs are and will be the domain of a thinking human being – a software developer.

Tools like GitHub Copilot (and ChatGPT to some extent) are a great aid to help developers. Not a threat to us, IMHO. I actually find Copilot quite useful but not something that can entirely replace writing code. ChatGPT can actually produce nonsense as long as your question sounds smart enough, while the DevSkiller platform prevents hiring people who only use AI-generated responses/solutions.

Tomasz Nurkiewicz, DevSkiller CTO

The developers seem to approve:

Source: Reddit

What does the future hold?

Even though we don’t see AI tools as a threat to our business model, we’re concerned with a couple of things related to their proliferation.

First of all, algorithmically generated code sometimes makes no sense or guesses incorrectly  the intent of the person initiating its creation. A careless developer may even introduce security vulnerabilities by accident this way. This was already observed when code sample corpuses contained malicious code frequently enough. There’s also a broader copyright issue. Technically, we don’t copy code from anybody else. Nevertheless, it’s not generated out of thin air either.

However, it doesn’t change the fact that code testing based on real-life programming challenges – the philosophy that laid the DevSkiller foundation – is here to stay.

Within a few years, AI code generation tools will become ubiquitous like Stack Overflow. Rather than preventing using them, we should highlight the fact that DevSkiller is AI-friendly and even encourage candidates to use such tools as part of their normal workflow. We have tasks that you can solve 80% with Copilot (like in an ordinary job), whereas the remaining 20% need your real skills – what we test accurately. At the same time, the competitors’ tasks can be solved 100% using Copilot, which says nothing about a candidate, and a lot about the quality of their tasks.

Tomasz Nurkiewicz, DevSkiller CTO

Summary

At DevSkiller, we understand that Artificial Intelligence (AI) tools like Chat GPT technology have the potential to revolutionize the way software developers are tested. However, we also believe that to properly assess the skillfulness of a candidate or an employee, we need to simulate the real-life working environment they’ll be working in. This means, allowing for use of AI – be it ChatGPT, GitHub Copilot, or anything else – in the tests. All in all, those tools are present in the daily work of a programmer in real life.

Nevertheless, no AI tool will be able to fully replace a human being – not only in the software development industry but also in other areas.