Ep. 10: Obsessed with ChatGPT? Here’s the Hype and the Hope

Summary

ChatGPT took the world by storm when it debuted last November. Politicians are puzzled, scientists are worried, and consumers still can’t get enough. There’s something surreal and Willy Wonka-ish about the whole thing. (It’s worth noting Wonka wasn’t exactly to be trusted.) We ask an expert how worried she is on a scale of 1 to 10. Her answer comes without hesitation.

HOST: Shawn Surber, senior director of technical account management, Tanium
GUEST: Krystal Jackson, junior AI fellow, Center for Security and Emerging Technology, Georgetown University

To learn more about this issue

Check out these articles in Focal Point, Tanium’s new online cyber news magazine, and two recent ChatGPT alerts from our cyber threat intelligence (CTI) analysts.

• Can We Actually Pause AI? 1,100 Experts (and, yeah, Elon Musk) Say Yes
• Yes, ChatGPT Will Turbocharge Hacking—and Help Fight It, Too
• CTI Roundup: Hackers Use ChatGPT Lures to Spread Malware on Facebook
• CTI Roundup: Threat Actors Exploiting ChatGPT

Interview transcript

The following interview has been edited for clarity.

Krystal Jackson: We are definitely having this kind of GPT moment right now. And definitely as someone who has been in the AI space and aware of generative AI and its capabilities for a little bit, I didn’t quite get it at first—why this model this time was getting so much attention. But I really think part of it is that OpenAI kind of cracked the code on the chat part of ChatGPT.

Shawn Surber: It can pass graduate-level exams, build code for websites, and even craft the perfect pickup line just based on your dating app profile. It can also increase cyberattacks, making it a lot easier for just about anybody to create their own malware, phishing emails, and other dangerous content.

Hi, I’m Shawn Surber, and today on Let’s Converge, we’re talking ChatGPT—the hype and the hope.

People are obsessed with ChatGPT and other new chatbots powered by artificial intelligence [AI]. They’re also a little freaked out. In fact, we’re recording this on the day that Elon Musk and Apple co-founder Steve Wozniak and a group of other prominent scientists published a petition calling for a six-month pause in the AI race. They say tech companies are moving too fast and they’re rolling out technologies we don’t fully understand.

The fears surrounding these chatbots are not unfounded, but here’s where the hope comes in: The same tool that can be used by cybercriminals can also fuel a firestorm of new defensive technologies to protect businesses, government agencies, and enterprises large and small.

Joining us today is Krystal Jackson, junior AI fellow at Georgetown University’s Center for Security and Emerging Technology. Krystal’s research centers on the future of weaponized AI and the ethics of technology, which sounds absolutely fascinating to me. Welcome to the podcast, Krystal.

Jackson: Thank you so much for having me.

Surber: Absolutely. So, a new study came out this month about phishing emails—you know, those clever messages that trick us into clicking on dangerous links and downloading malware. The researchers studied 53,000 email users in more than 100 countries and found that phishing emails created by humans generated a click rate of 4.2%, while ChatGPT emails only got a clickthrough of about 2.9%. So humans are still better at tricking other humans. I don’t know if that’s cause for celebration or not, but should we feel hopeful about this? As advanced as these chatbots are becoming, are humans still better at crime than machines?

Jackson: Yeah, I think that statistic really speaks volumes. You know, at the heart of phishing scams and phishing messages is the intent to deceive. And that’s something that I don’t think we’re seeing quite yet with these early iterations of chatbots in particular. There’s still a very general-purpose technology meant to generate general texts, and so that doesn’t completely surprise me that they’re not as effective. I think part of the concern is around the volume and the scale and just the sheer amount of phishing messages that you might be able to get out. But certainly the effectiveness piece is really important as well.

Surber: I agree, and I think it actually makes sense at this point that professional red-teamers are going to be better at generating phishing emails than chatbots that aren’t exactly designed for it. But what about the concept of an AI-in-the-middle, like chatbot interception attack? I’m probably not the first person to call it AI-in-the-middle, but I think I should trademark it anyway. But you know, when you go to a website and it pops up and says, ‘Hey, do you want some help?’ What about malicious actors using something like ChatGPT to intercept those connections and allowing that AI to quickly and effectively respond while feeding the victim bad information, bad links, et cetera?

Jackson: That’s really interesting. I think that in a lot of ways that might be some of the direction that we’re going kind of unintentionally. So one of the things I’m concerned about most is ChatGPT and these other chatbots being integrated with other real-world systems. There’s a really big push right now to get ChatGPT integrated with systems that allow you to book a flight or order groceries. Very similar to how Amazon Alexa kind of helps you order stuff as well, but for the entire internet. So I can definitely imagine something like this turning into a prime target for attackers. If every insecure payment system and website suddenly has a chatbot attached to it that has the potential to generate malware, string together exploits in unique ways, that seems like we better have the security of those kinds of systems pretty high before we start plugging them in. And definitely, people can take these systems and weaponize them. That’s definitely something I think we need to keep our eye on going forward.

Surber: I’ve actually been playing around with a variety of the ones that are available now. They’re good at small tasks, but if you try and get ‘em to write an entire program, they wander off the rails. But OpenAI recently released ChatGPT-4. So what’s new in 4 and how do you think that’s going to affect things?

Jackson: GPT-4 is definitely bigger and badder in most ways. Some of the highlights is it’s multimodal, which means that you can now input images as well as text. It still doesn’t output images, but this definitely gives a larger array of things that it can intake as information. And it can output and intake larger pieces of information. So to your point about current capabilities, it seems like you have to prompt ChatGPT a lot to get it to string together little pieces of code to put together what would be like a full piece of malware. With GPT-4, it really opens up the ability to put out larger pieces of code, summarize larger pieces of text.

For some context, GPT-3 used about 175 billion parameters in its training, and GPT-4 uses trillions, which means it’s really orders of magnitude better at doing those learning tasks, the complex problem-solving and translation kind of skills. It’s just better at those.

Surber: Now, is it true that ChatGPT exists kind of in a bubble in time, as it were? It’s only looking at content that was fed to it as part of the training? Whereas I’m seeing things like the new AI bots from Microsoft and Google are touting the fact that they’re looking live at the web?

Jackson: Yeah, that’s definitely one of the limitations of some of these systems and something that different developers are trying to address in different ways. So there was indication that, for instance, at some coding tasks ChatGPT seemed to have a cutoff. It was obviously looking at some of the popular platforms where you could test your coding skills and kind of upload them and compare them to other people. And there was a sharp cutoff at I think about September 2021 or about that time where it was no longer able to compete on the new coding challenges, which would indicate that, you know, it just wasn’t trained on that data. And so it’s doing something closer to recalling information than really being creative or smart in any way. But some developers, like you mentioned, they’re trying to address that by having continuous learning, and I think that’s something that’s really exciting, but it’s also something that gives us less control over what information these models do and don’t have. So that’s, you know, more consideration.

Surber: So why do you think people are so fixated on these AI chatbots? You know, everybody’s talking about it, it’s generating a lot of excitement, but in your opinion, is it primarily hype or is this a significant step in the evolution of our interaction with the digital world?

Jackson: I think it’s a little bit of both. We are definitely having this kind of GPT moment right now, and definitely as someone who has been in the AI space and aware of generative AI and its capabilities for a little bit, I didn’t quite get it at first—why this model this time was getting so much attention. But I really think part of it is that OpenAI kind of cracked the code on the chat part of ChatGPT. So to me it’s very conversational in nature. It’s able to pull together all of these disparate pieces of information that we haven’t really seen done in this way before in a really nice, easy-to-use interface. And because of that, you want to actually use it. It’s engaging, it’s conversational. So for me it’s all those little elements that really make it kinda the phenomenon that we’re seeing right now.

Surber: At this point, no one really knows for sure how much these chatbots can do, right? So what should enterprise leaders and cybersecurity professionals focus on? What are the areas of legitimate concern?

Jackson: That’s a really good question. You know, essentially it does three things really well. That would be filling in the information gap. So it has the potential to assist someone who maybe has a lower level of knowledge and would normally not be able to pull off some of these more sophisticated attacks and really elevate them into a more sophisticated adversary, filling in their knowledge on different subjects really quickly and efficiently—more than just, you know, going on Google or taking a course would allow you to do.

The second thing is it really helps with the generating and automating of this code and text production. We saw a couple months ago this research group that was able, with a little bit of prompting, to get the platform to create a polymorphic virus, and it created code for everything from the DLL [dynamic link library] injection all the way to the code for finding and encrypting files. And so we could definitely see that that capability is there, but it took a team of pretty advanced researchers to be able to get the prompting right, and they did hit the chatbot safeguards, but they were able to ultimately bypass them.

The third is something that I think is on the horizon, which is tailored models by organizations that maybe, again, do not care so much about safety and security, that might actually want to create on the backbone of things like ChatGPT models that are specifically designed for hacking and for malicious activity. So I think that those are some of the things that when you kind of break it down that way, it gives you a clearer sense of what the concerns are. And I think for all those cases we still have the tools to be able to address them.

Surber: So when all is said and done, scale of 1 to 10, your professional opinion, 1 being you’re totally chill on it, 10, you’re being pretty freaked out on it, where do you stand on AI bots like ChatGPT?

Jackson: I would say that I’m maybe a 4. I’m not gonna lose any sleep over it and I’m gonna say I’m pretty excited. There’s gonna be a wave of really cool applications that pop up as a result of all of this. And although we recognize that we have a lot of hurdles to overcome to see the real benefits, I’m feeling pretty confident that we can get there.

Surber: Good, good. That makes me feel more confident. So that’s a good thing. You know, it’s interesting, what I’m seeing is these AIs are really good at rapid response, analyzing massive data sets, that sort of thing, which strikes me as being of greater value to defenders than attackers. You know, as they say, the attackers only have to be right once, the defender has to be right every time. So being able to analyze your closed environment more effectively, more rapidly and looking for malicious attacks, I think AI’s got some really interesting value there. What are your thoughts? Do you think this is going to be better in the long run for defenders than attackers?

Jackson: That’s a really interesting question. It’s definitely one that I think about all the time. It’s hard to say either way, but I have to say that definitely as defenders we’re not losing out. Right? So you hinted at all those things that AI really gives an advantage for. And one thing that’s really interesting that just popped up that’s new is Microsoft Security Copilot as a new GPT-4 AI assistant for cybersecurity tasks. It’s able to do some of those things you’re saying, like analyze code snippets. And really one of the big things that I’m excited about is automation. Usually, especially if you’re a smaller organization, automating some of your security processes can be really time-consuming and costly, and this seems like it could really give a big advantage to those kinds of organizations, having their analyses or reverse-engineering processes, or some of things that are a little bit more complex, have automation steps along the way.

And so those kinds of features I think are really exciting. And again, it’s a general-purpose technology, so as quickly as they can make malicious applications, we can make defensive applications. I think we have a little bit more of the advantage in terms of the structures and people that we have on our side trying to build those.

Surber: That’s good. I like that . So tell us a little bit, if you can, about your research, investigating the potential of weaponizing AI.

Jackson: Some of my research in the past has kind of looked at that offensive-versus-defensive question that you asked before. I tried to do some basic mathematical modeling of how AI might impact different areas of cybersecurity, like the ability to exploit patch vulnerabilities. Phishing. Currently, though, I’ve kind of transitioned into using reinforcement learning for autonomous cyberdefense. So again, giving defenders those tools to kind of uplift cybersecurity and give us an advantage. That’s my main focus right now.

Surber: You just raised a really interesting point in my head, using AI for education. For certification exams, a long time ago, they went to more of an interactive model—as you’re getting questions right, it skips sections of questions. Is there a space for not just cybersecurity education, but education in general, where students could interact with an AI to learn faster, more effectively, and you know, more completely than they’re getting from textbooks and looking things up on Wikipedia—which I’m sure they’re not supposed to do.

Jackson: Yeah, absolutely. I think one of the things that’s really interesting about this kind of GPT moment we’re having right now is we’re kind of narrowed in on all these little sub areas, or examples of things like, yes, it might make adversarial behavior better or worse, et cetera. But I think there’s the larger question of how will these models change how we do education? How will they change how we think about cybersecurity overall? It’s really making us rethink a lot of these things, you know, just not to the point of, can it help us be better at learning and getting through education, but also finding out that a lot of the things we do in the education space are not maybe as effective or are not going to be able to be the standards that we had before. You know, we’ve seen so many articles, as much as I’ve seen the articles about malicious uses, I’ve seen students are cheating. I think about half of students are now using GPT in some way. So it’s definitely making us have this larger conversation of,like, what is the role of these models in society and in everyday life? And we’re definitely gonna have to rethink some of these things.

Surber: That’s a big deal. In fact, it was one of the first things I was thinking about as I started plugging around here and I started using ChatGPT to just generate some text for me—’Hey, you know, write this story. Give me this, this idea.’ When I delved deeply into technical stuff, I found it to be of average use. But the ability to generate unique text, it was very skilled at, so that was my first thought—’Oh man, is this basically gonna be a way around plagiarism checkers or will teachers in the future be able to feed the student’s text back into ChatGPT to see if it’s something that ChatGPT or another bot created? Is that something that would even be possible? Interesting questions.

Jackson: Yeah, absolutely. As far as I know, that’s not possible as of right now. But I could definitely see specific applications for that becoming really popular in the future, especially as this becomes like a larger problem in the education space.

Surber: What’s your outlook for, say, the next five years?

Jackson: It’s really early days for this technology right now. We have this very myopic view of what’s it doing right now? But we want to stay creative, we wanna stay optimistic. I think that’s the way some of this generative AI is going to end up serving us the best. So I don’t have really specific answers to that question, unfortunately, but I think it’s definitely gonna have huge impacts for the kinds of things that people just create. That’s what I’m most excited for, in the next five years and going forward.

Surber: Well, here I was hoping you’d give us some hard predictions that we could hold you to, but I can’t blame you for that. Everything’s evolving way too quickly for it. So as we’re wrapping up, is there anything else you’d like to share with us that I haven’t asked you about?

Jackson: I guess the last thing that I would just say is like a final word of caution, something I don’t think maybe is getting as much press: I really hope that, as amazing as all this generative AI is, that we don’t let it completely replace human decision making. I hinted that before, like we’re integrating it into all these different systems, and the last thing we want is to lean so heavily on the technology and into the hype that we forget it’s only trained on the information that it was provided and was already out there. Some of the information on the internet’s good, some of it’s really bad, and we need to keep a real premium on critical thinking. Now more than ever, as we kind of learn how to use this tech and want to evolve.

Surber: Wise words, wise words. You can’t replace humans, as much as sometimes we’d like to. And so I like that, I like that perspective. There’s a lot of things that I think that ChatGPT and other AI bots can do to help make life easier, make work easier, hopefully not make school easier, but in the end, human creativity I think is going to be something we can’t just replicate with AI, at least not anytime in the near future. So thank you so much, Krystal. Really appreciate it.

I’ve been talking today with Krystal Jackson, junior AI fellow at Georgetown University’s Center for Security and Emerging Technologies.

If you’d like to read more about ChatGPT, check out Focal Point, Tanium’s online cyber news magazine. We’ve got links to several articles in the show notes, or just visit tanium.com. Make sure to subscribe to Let’s Converge on your favorite podcast app. And if you liked this episode, please give us that five-star rating.

Thanks for listening. We look forward to sharing more cyber insights on the next episode of Let’s Converge.