Can We Really ‘Ignore’ GenAI’s Vulnerable Code? This Security Pro Says Yes (Sort Of)

Let’s face it: Software developers don’t necessarily get the most extensive training. So they’re sometimes writing insecure code while feeling the pressure to work at faster-than-human speed. Now, in the era of AI, they’re leaning more on tools that can generate code snippets faster, and they – make that all of us, actually – tend to trust that code more than human-generated code.

That speed and trust is not a great recipe for app security.

The higher code velocity enabled by large language models (LLMs) and generative AI (GenAI) translates into higher vulnerability velocity, which is now generating a host of new challenges to security testing, remediation and smart AI governance.

The risks from third parties and software supply chains are rising. Know your vulnerabilities – in seconds.

As we approach Black Hat USA 2024, one of the most anticipated sessions is “From HAL to HALT: Thwarting Skynet’s Siblings in the GenAI Coding Era” by Chris Wysopal, a respected leader in application security and co-founder of Veracode.

In an exclusive briefing just before Black Hat USA 2024, Focal Point sat down with Wysopal to discuss the crucial topics he will cover in his session. Here are some key insights from our conversation, providing a preview of what attendees can expect to learn about the intersection of generative AI and cybersecurity.

This interview has been lightly edited for length and clarity.

Your Black Hat session namechecks HAL and Skynet – some of AI’s most infamous Hollywood moments. Short of Arnold Schwarzenegger busting into our offices, how much change do you think is really headed our way?

The idea of the session is to talk about how LLM-generated code is probably one of the biggest usages of LLMs today. It’s growing and has amazingly huge potential. It’s going to change the way software is built.

LLM-generated code is probably one of the biggest usages of LLMs today. It’s growing and has amazingly huge potential.

For me, as someone who’s been in application security for over 25 years, this is a huge change to software development. Typically, when there are changes to software development, there are changes to application security. So, this talk is about how generated AI, generated code, is going to change application security.

[Read also: A procrastinator’s guide to Black Hat USA 2024 – and how to watch later]

You’ve mentioned that LLM-generated code can be as insecure as human-generated code. What are the implications of this for companies relying on tools like ChatGPT or Microsoft Copilot?

There are a few different studies from Stanford, Wuhan University, and Carnegie Mellon that looked at this problem and basically found that these LLMs are generating code that’s just about as insecure as human-generated code. Garbage in, garbage out. If you learn how to do something one way, you’re probably going to do it that way. So, that’s sort of like the baseline starting point. And then it’s like, well, what do we do about this?

How can companies leverage GenAI to their benefit while ensuring secure software development?

[Instead of training] a foundation model on how to write code, we can fine-tune it and give it examples of what bad code looks like. And if you see this bad code and you’re told to write good code, this is what the good code would be.

[Instead of training] a foundation model on how to write code, we can fine-tune it and give it examples of what bad code looks like.

One of the things we think is a solution is to go ahead, use your ChatGPT, your Code Llama, your Copilot, because it’s going to have the best knowledge on how to write different types of code for many different languages and coding problems. And ignore the fact that it’s writing vulnerable code, just like you’re ignoring the fact that your developers are writing vulnerable code. Because, frankly, you’re not sending them to such an intense secure coding training that they’re writing great secure code all the time. You’re basically saying, I know my developers are writing insecure code.

So you still have to test for security issues, and you still have to fix security issues.

How do you see the role of human expertise evolving in this AI-driven coding era? Is there still a place for manual code review and security testing?

We already have a problem of finding more issues in human-developed code than we can fix, right? So you have this growing security debt. You already aren’t committing enough resources to fix all the things that you’re finding with your automation.

So that’s like the problem we’re starting with. And now we’re going to amplify that problem by writing more code faster with essentially the same security quality. Even if it was twice as good, even if it had half the vulnerability density, you’d still be ending up with a bigger problem if you were running three times as much code per developer. So we’re generating more vulnerabilities using LLMs in development.

So yeah, we need fast tools in the CI/CD pipeline that will find the problem, but we also need fast tools that will fix the problems.

[Read also: Ultimate guide to AI cybersecurity – benefits, risks, and rewards]

You’ve talked about the potential for recursive learning and the degeneration of knowledge in AI models. Can you elaborate on this issue and its implications for cybersecurity?

Yeah, this is starting to become something that people are worried about. There are degenerative effects on knowledge when you’re learning from AI-generated content because you’re learning those hallucinations. So the idea is that, what if most code being written is written by LLMs, and those LLMs, we all know, get some things wrong. Does that cause a problem for the next generation of code that it’s going to be learning from?

Go ahead, use your ChatGPT, your Code Llama, your Copilot… And ignore the fact that it’s writing vulnerable code, [because] you still have to test for security issues, and you still have to fix security issues.

You know, that next generation of LLMs – in three years, when it’s learning, is it going to be learning on maybe 10% of the code in the world being written by LLMs? And will that start to increase? Maybe someday we’ll be almost always writing code with LLMs, and there’ll be barely any human interaction. What does that mean if the LLMs are learning from others, so we’re just learning from the output of other LLMs?

We’re just starting to understand how this has a degenerative effect on knowledge, and it might be that we have to mark code generated by LLMs so that we’re not training future LLMs with that code. Or something like that.

What ethical concerns arise from using GenAI for coding, especially regarding code attribution and copyright issues?

You have the same problem that you have with the New York Times articles or like with Sarah Silverman’s book – she is suing because she wrote some jokes in a book that she’s never performed anywhere. It was just in the book, and it’s showing up in ChatGPT responses like, you know, “Write me a joke about this,” and it’s using her jokes.

So it’s clear it’s just using my content and my copyrighted content. So that’s absolutely happening with code.

[Read also: Here’s another legal quagmire that has CISOs (and enterprise owners going all in on AI) worried – and how to steer clear]

Most of the code it’s learned on is GPL [General Public License] code. So it requires attribution, and, with some of it, the GPL 3 would require you to publish the code of your software, if you’re including it. So, yeah, that’s something that we haven’t seen any case law about, but I can tell you that lawyers are very worried about this, and what they’re doing is analyzing code with code snippet analysis to try to match it up, to see if it’s matchable with open source, and if it’s a big enough block of code, that would be an infringement. Like, one line of code you can’t infringe on. But if it’s a whole routine… If it’s a whole class, that’s definitely infringing.

As we wrap up, what are you most looking forward to at Black Hat, and what do you hope attendees take away from your session?

What I really want to say is that generative AI is impacting coding in a huge way. Just like moving to the cloud or something like that, it’s a fundamental change for applications and we need to think about how this is going to change the security of software going forward.

We need processes in place so that this doesn’t make it worse. Like automated fixing – I think that’s a good one. Our job is to make sure, you know, if we’re a bank or a software company, that whatever we’re putting out in software is secure, and if generative AI is causing it to be less secure, we need a response. That’s my main message.

LEARN MORE AT BLACK HAT USA 2024

Don’t miss this chance to hear from one of the leading minds in cybersecurity and join the conversation on the future of AI and application security at Black Hat USA 2024. Chris Wysopal’s session at Black Hat USA 2024 is a must-attend for all IT professionals – and enterprise leaders looking to gain sharper insight on a major obstacle to AI security.

• Session: From HAL to HALT: Thwarting Skynet’s Siblings in the GenAI Coding Era
• Date: Wednesday, August 7
• Time: 1:30 – 2:10 p.m. PT
• Location: Oceanside A, Level 2