Down the Security Rabbithole Podcast (DtSR)
This is Cybersecurity's premier podcast. Running strong since 2011 Rafal Los, James Jardine, and Jim Tiller bring a no-nonsense, non-commercial approach to our profession. DtSR brings interviews and discussion with people you want to meet, and stories you have to hear. So whether you're just starting out, or are decades deep into your career, you'll always learn something on this show.
On Twitter/X: https://twitter.com/@DtSR_Podcast
On YouTube: https://www.youtube.com/@DTSR_Podcast
On LinkedIn: https://www.linkedin.com/company/down-the-security-rabbithole-podcast/
Down the Security Rabbithole Podcast (DtSR)
DtSR Episode 730 - Matt Coons Agentic Detection Engineering
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
TL;DR: Stop vibe-coding your detections! This week's pod features Matt Coons at GitLab as we talk about detection engineering, but with AI agents.
Guest
Description
Your detection backlog is endless, your alert queue never stops, and “just tune it out” quietly creates blind spots. From the floor of BSides Atlanta, we sit down with Matt Coons from GitLab to talk about a different path: agentic detection engineering that moves faster while staying provable, testable, and human-owned.
We get concrete about what changes when AI agents enter the loop. Matt explains how end-to-end validation can work in practice, from running real API calls to ensuring the right events land in the logging platform and trigger the intended rule. We also unpack the difference between detection engineering and detection tuning, why time-bound tuning beats “mute forever,” and where human judgment still matters when an agent suggests exclusions that might hide real malicious behavior.
A big theme is designing security operations for scale. We explore signals versus detections, grouping signals by entity to expose patterns, and using multi-agent triage to reduce SOC burnout without losing visibility. Matt shares how model tiering can control cost, why newer frontier models can deliver more production-ready detection content, and why the real bottleneck is shifting toward human approval and accountability. We also hit prompting discipline, skill files, and how to get summaries that are actually shorter than the original.
If you care about AI in cybersecurity, SOC automation, threat hunting, MITRE ATT&CK coverage, and reducing false positives while catching more real incidents, this one is for you. Subscribe, share it with a teammate, and leave a review, then tell us: what part of your detection workflow would you trust an agent to handle first?
Have something to say? Let's hear it.
>>> Please consider clicking the link above to support the show!
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
YouTube home: https://youtube.com/playlist?list=PLyo0dkKRvfVtWXjRxNISrhme1MgBj3C2U&si=scHDiTuLXSEQ9qHq
LinkedIn Page: https://www.linkedin.com/company/down-the-security-rabbithole-podcast/
X/Twitter: https://twitter.com/dtsr_podcast
Cold Open And Welcome
SPEAKER_03They say they say we should have known better than two. So deep down, deep down into this rabbit hole. And uh let's think. Hi, this is Nikolai, and you're listening to my rabbit revolve, co-host James Curity, and featuring Mr. Jim Tillers in the one and only Young Security Rabbit Hole podcast. Again, listen up and enjoy.
Live From BSides Atlanta
SPEAKER_01Here on October 30th 30th. Wow, I'm getting ahead of myself. On October 3rd, and we're here at the uh Georgia Tech Hotel and Conference Center. Thank you to Mr. Green, Professor Green, buddy Scott, and of course Steve for having us here. I get a chance to sit down with some really interesting people. I've had Mehur Van Car on the future of cyber defense in the age of AI. Phil Bratz talked about LLMs and the changing nature of pen tests. OS Widener talked about open weight models, and that one got a little philosophical. I like it. Man, it was a treat having Xavier Ash here on talking about uh stuff from the front lines and his colorful career. And now we wrap it up with the one and only uh Matt Coons. And of course, Matt, we're gonna talk about agentic detection engineering, the sweet smell of agents in the morning. Oh yes. Late afternoon, as is the case. All right, give us a quick introduction about yourself.
SPEAKER_00Yeah, for sure. Really excited to be here and chat with you. So I have been at GitLab about four years, and I lead detection engineering, uh trust and safety, and also threat hunting in security operations. Love myself some good detection engineering.
SPEAKER_01Yeah, so all right, where what's the angle you're
AI First With Guardrails
SPEAKER_01coming at this from? Are you a skeptic? Are you a are you a uh AI first somewhere in between?
SPEAKER_00Yeah, for sure. Great question. You know, I would say I'm definitely AI first, however, also want to make sure that we uphold our standards. Like to me, like being AI first, but just like vibe coding detections is never a good look. How do we know our detections are gonna work? How do we know like we're being effective? And so it's you know, I think that agentic detection engineering can like, you know, significantly increase our velocity, but we've got to make sure that we are uh doing it appropriately, responsibly, and more, you know, we have controls that we have to uh keep checking.
SPEAKER_01This is one of those topics where I literally now work in a company that we we this is what we do, right? Which is which is interesting, and gotten the pleasure of of seeing this from the inside out, in addition to the outside in for many years. But the the technical engineering has traditionally been either something companies simply don't do because they don't have the talent for it, or they do poorly because they're missing the key component, which was that like that feedback loop, which you just talked about, right? Does this does this thing we just cranked out work or does it create more noise? And if you don't have that loop, you might you it's actually probably better that you don't do any of this. Yeah, how has that evolved pre-AI over the last you know decade or so? And then like AI shows up and what's different?
End To End Detection Testing
SPEAKER_00Yeah, for sure. So I think you know, pre-AI, um everyone was you know realized that they had to build detections to identify a malicious activity, but it took a lot of work to validate the detections that you wrote were were accurate. So you had several different ways you could do it. You could go out and try to simulate the activities yourselves, maybe work work with your red team or do like a purple team exercise or something like that. You also could like try to build some process or some product product that would do it for you, or like go out and buy a BAS solution or something like that, a breach attack simulation platform. Post AI, I think like that whole loop becomes a lot easier. As long as you have the connections to the right tools, you and and you set up the foundations, you can use Agentec AI to like build the test cases. And like that's that's one of the things we have internally. It's it's a kind of a program called Oops, and it's like an offensive simulation tester. Oops. Yeah, exactly. I like it. Um yeah, so uh great name. And so the the platform is really designed to go out and validate that our detections are working correctly, and we have the connections to the different systems. It doesn't just you know simulate log events, it actually goes out and runs the API calls, does the actual thing to make sure that when the API call runs, the event goes into this logging platform, and then that triggers the detection that it should trigger. So it's end-to-end.
Engineering Versus Tuning Decisions
SPEAKER_01Okay, so what's I'm gonna take you just half a step back because I think there's some something I want to clear up. Detection tuning versus detection engineering, same, similar, totally different?
SPEAKER_00Yeah, so I would say that typically owned by the same team. However, in my mind, you know, they are different processes. Okay. You know, with with detection engineering, you're writing the new detections, you're going out, you're making sure you have coverage against the threat actors that are relevant to you and the threats that are relevant to you. The tuning aspect comes in to make sure that you're reducing the noise so you have an acceptable true positive, false positive ratio. It's interesting, you know, with agentic AI, it also makes tuning something that's easier because agents are pretty good at identifying like this is a specific pattern that's triggering a lot, I can exclude that. However, I think where the humans still need to make the critical decision is is that a tuning event that I want, is that a tuning that I'm gonna accept, right? The agent is great at suggesting the changes and it's probably gonna be really accurate. But do I want to exclude that? Or is that actually an indication of something malicious happening or something weird happening? And I don't want to exclude that. So that's where the humans really have to be the decision makers in the process.
SPEAKER_01How do how do you see this? Because I think when we did when we talked about detection tuning, the natural inclination is this is too noisy, turn it off, just you know, like make it go away. Right. But sometimes there's value in that noise. Yes, for sure. Right? So just excluding classes of attacks. Yes. So this requires uh previously, and maybe still, this requires institutional knowledge, experience, dare I say. Yes, right. So that you're like, okay, I know this fires 10 times a minute, but the 11th time it fires maybe something real. How do I how do I tune that down? And by tuning, I don't mean turn it off, but make it more relevant, right? Right. Is it is is AI helping you with that?
SPEAKER_00Yeah, for sure. And so I would say yes. We have, you know, we try to be more, you know, I would say customized with how we do tuning. So we can at GitLab, we will tune um a specific behavior for a certain amount of time. So you have the ability to tune out for a period versus just tune out for infinity. I think that's one of the things we often do is we we tune out for infinity when maybe it's a short-term process that's gonna that's not gonna become a problem in a in a couple of weeks. Right. So we have the ability to tune for a period of time.
Signals Grouped By Entity
SPEAKER_00We also have this concept of uh signals versus detections. So signals essentially don't alert directly, but they get grouped together based on entity. So I have different acceptable threshold levels for signals versus detections. For a signal, if it's triggering a thousand times a week, I could care less because no one's looking at those individual signals directly. I've got agents on retro hunting across the signals, but I don't have the sock, you know, who's like, you know, burdened by all of these signals. They're telling me interesting things that are happening. And when you group them by entity, then you see patterns. You see, wow, this one entity has all these different signals that are firing. That's anomalous. Is that something brand new then?
SPEAKER_01Because that's that feels like it's something that humans weren't able to do before. So at least not in not in a meaningful way, because agents don't sleep. They don't care how much data you throw at them. Right. You could throw the same attack 24,000 times a day. Right. And the person will look at it and go, it's exhausting.
SPEAKER_00I don't know if I can do anything with this. Agents don't care. Right. And so I would say, to a certain extent, like I would say the idea of like risk-based alerting was supposed to solve that, where you use risk as a quantifier and signals will, you know, like introduce risk, add risk to an entity. However, there's the chance that you might have one signal that by itself is really interesting and never hits those thresholds. And so we actually had some real world examples where we have an agent that goes out and says, This one signal looks really interesting because these things happen that's anomalous. Hey, go look at that. And then it'll raise that one signal and have a human go look at it. And so there's value. And like, you know, on the on the incident response side, they have a full like, you know, end-to-end multi-agent triage system. And that is actually finding really interesting things and reducing the noise, right? So we're doing the tuning on the detection interference side, but on the incident response side, on the cert side, we have like kind of a tierless sock. On that side, they're doing the same thing to only escalate the interesting things to the humans that really need to be looked at. So where's the biggest advantage then?
SPEAKER_01That is it in scale? Is it in in is it in the actual processing?
SPEAKER_00Is it in all of those? Yeah.
Velocity Scale And Human Approval
SPEAKER_00So on the on so let's talk about it on on two sides. On the detection engineering side, I definitely think it's velocity and scale. And so I definitely so one of the things we're building is like an end-to-end detection, you know, kind of capability where I give this this platform an idea. It's a GitLab issue or some threat, it goes out with multiple agents, validates the detections, goes and runs them against our logging platform, and then gives me like you know production-ready detections. The the bottleneck is changing. The bottleneck used to be writing detections. I used to have a backlog of all of these ideas. Right. Someday I'll get to this. Yes. The bottleneck now is the human approval, right? Because it's done all the work, it gives you the MRs and says, these are ready to go. Now the human needs to go and say, okay, are these up to my standards? Did it do everything that I wanted it to do? So the j so what you're saying is the jobs not going away, it's shifting. It's shifting. We're becoming architects. We're becoming the ones who sit at a higher level and validate the results of what the agents are doing. But that's what you want anyway. For sure.
SPEAKER_01We don't want to be doing the detection thing. Well, tur turning them, turning that crank is like it it's fun at first, but then it like just gets tedious over time. Yeah. And and I mean, look, machines are good at finding signal and noise when you tell them what to look for. For sure. And then you've got the this notion of anomalous activity, right? Which is what ML is fantastic for. Look at 10,000 machines talking and tell me which ones looks different. As a human, you're like, yeah, right. No chance. Especially if you track ports, protocols, right? Packet size, packet contents. Right. Right? Time share. Like, no chance. Yep. But machines are fantastic for that. Yes. Is that is the efficacy of like a detection engineering team tied to any like the the the frontier model newness awesomeness, or is it like one of those where like it doesn't really matter? You could use an older or open weight model, for example?
SPEAKER_00It's a good question. I'm actually, and in like kind of, you know, some of this may tie into what I'm doing in trust and safety as well. We also do detections in trust and safety. We're detecting malicious behaviors for a different reason and we auto-mitigate it. But I'm actually a big fan of like uh model tiering. Okay. So I use cheaper models for the more like simple tasks. Uh-huh. And then the more interesting things get sent to a more expensive model, right? So in the grand scheme of things. That makes sense. Yeah. In the grand scheme of things, I'm I'm saving money, but you know, using the you know, the Opus, you know, the five or whatever, using the more expensive models for the things that really need that more advanced knowledge.
SPEAKER_01Are you finding that there's that much difference between what's what's released today and what was released six months ago? Like what I'm asking is, do you think it's worth it?
SPEAKER_00I would definitely say the newer frontier models are really good at giving me detections that are ready to go, production ready. Okay. So much so that I want us to get to a point where my signals go straight to production without a human in their own. Really? Because signals, we're not reviewing them directly, right? They're getting grouped on entity. So the idea would be my agents are running full-time, building new signals, identifying new interesting tripwires, right? They're laying that groundwork. And then the more interesting aggregated detections are the ones that, like, you know, are the criticals that I really want a human to look at. We're reviewing those because those are the ones that we need to make sure 100% right. But the lower level ones, go to product.
SPEAKER_01So I think that's an interesting detect distinction calling out signal versus detection, right? Things that it it's interesting, right? How often somebody logs in, where they log in from, how many packets they send. I always found the like the back when I was doing apps, like one of the interesting things, truly interesting things was like, okay, you know, when we look at a a database server, like rows return for a query, right? For sure. You your your average is like, you know, a hundred, suddenly somebody's doing a million. You're like, hold on, right. I know what that is, right? That doesn't take rocket science. For sure. Either I remember when I can't think of his name right now, and I was just gonna say it. He was the uh CISO for Etsy. He wrote trip trip yarns.
Tripwires And Promoting To Detections
SPEAKER_01Okay. Zane Lackey. Okay. Zane's uh this concept of like tripwire, the enemy knows they've tripped it, right? Because something goes boom or something goes click, and you're like, uh oh crap, I gotta hurry up and get out of here. Yep. A trip yarn is not something that they fall over, it's it's a log that gets done. I think his example was like after every reboot, you pull the D message off of the machines, and if there's a hundred machines that were cloned from each other, they should all look identical. Right. One of them holds a new module, you're like, not necessarily bad. Right. But that's kind of interesting. Right, for sure. 100%, right? So you guys find the same thing. How do you go from how do you promote something, or how do you know when something is promoted from just signal to a detection? Like that's an interesting bridge.
SPEAKER_00For sure. Yeah, and so the way I think of it is signals are more atomic in nature. Okay. Detections, I want them to be behavioral. So I don't want detections to be just like a pretty simple thing. I want it to be a pattern of activity or something that tells more of a story. And then the interesting thing is on the on the incident response side, on the on the on the you know, the sock side, they're actually able to use a genetic AI to go out and validate a lot of like they'll get a detection from us, and then they'll use agents to go out and like gather all this context. What's the what's the user? You know, who's the user? What's their normal access patterns? Get all this additional context and make a determination, yeah, this is weird, or no, this is just like, you know, the this is Corpsec doing Corpsec stuff, they're changing Octa policies again. Yeah. And so and and so we've been able to, which is really cool. We have the we have this concept of um you we call it UIM, user attestation module, where if there's an interesting thing, but we don't want to detect on it specifically, we'll send it to the user and have them validate. We'll send them a DM and say, was this you yes or no? And they can click yes or no. With agentic AI, we've been able to intercept that and say, okay, are you we gonna even send this to the person? Right? Interesting. And so we've actually had like our corpse team just recently, they reached out to us and said, We're not getting notifications anymore when we change octa policies. What's going on? And we said, Oh, guess what? Now we're having our our agents identify is this probably legit or not? Because you're always doing it from the same IP, you have an access request, you're doing all of everything by the book, why tell you about it?
SPEAKER_01Because it looks like everything else you've done, right? Exactly. And then you can you can the odds of it being something bad are so much lower. Not zero, but not so significantly lower. Yeah. That's interesting. I I think I think the future of this this particular space is so promising because when humans had to create detections and we had to process signal, noise. Well, once we you know I say once once we got the noise down to something manageable, that that's never that's actually never been a thing. It's never really been manageable though, right? Because noise is just noise, it happens. And I think noise for a lot of socks, it burned out a lot of people because that detection cue just it never ended.
SPEAKER_00Yes, for sure. I mean, and and that and that is the promise of like you know what we're able to do with with agents, and like I would say, even beyond that, what is amazing to me is what like one engineer with a problem that they know needs to be solved can do with like Claude as an example, right? Like, you know, an example like on our on our incident response side, like you know, we were able to build this multi-agent triage process. One engineer did it, right? The engineer is like, hey, I got a problem, I want to go solve it, I'm gonna do it, right? And that's totally possible in today's like you know, world where I'm not necessarily having I don't have to know how to write all of the code. I need to have a really good idea of what I want, what my problem is, and how to solve it. And I can use agents to do a lot of that work for me.
SPEAKER_01How has the how has the that that loop that we talked about a little bit ago between I need to I see something, it's not quite right, or I need to create something, then you gotta essentially stage it, test it, make sure it doesn't blow anything up, make sure it doesn't blow your mailbox up, right, right, because suddenly you're detecting 10,000 things a second, like that's not what I wanted. Right. Not helpful, yeah. To something that's useful, and then because just because it's useful now doesn't mean five minutes from now it doesn't somehow become a deluge of more nonsense, right? Right. So there's there's that like that continuous feedback loop. Where is where have you put AI into this process that and how does it change for sure?
Continuous Testing In Non Prod
SPEAKER_00Yeah, and so like you know, right now we're using AI heavily to write our new detection rules, but the the thing that I'm very cognizant of is, you know, I really don't want my detection rules to really be static. You know, techniques are gonna change, things are gonna change, and so I was talking about that oops product we built earlier. I really want to be able to use that to test adjacent techniques, right? I have a I have a detection for technique A, but guess what? There's three different ways to exploit that same thing using different APIs or hitting different endpoints. And so I want to be able to like, you know, non-deterministically do more testing, adjacent testing, to identify, hey, we're covering this, but we should be also covering these over here. And I can test this out. I can go run the API calls, show what it looks like in the logs, and then build detections for that.
SPEAKER_01Does AI give you the ability to test a detection without putting it into this into like production streams?
SPEAKER_00So if like synthetic data or whatever. Right. So you so you could do synthetic data or you could connect to a non, you could connect your AI agents to a non-production system, right? So if you have a non-production system, you can you can do it that way. Also with GitLab, we have uh what's called like a GitLab development kit, GDK, so which you can essentially run GitLab, like the entire GitLab product locally. Okay. They're actually standing up one like in Kubernetes. So you can run like a GitLab instance in Kubernetes. So you could you know use the agents to test it in that Kubernetes environment, validate what the logs look like, take those logs and write detections from them, and then put those in the prod. And that kind of bypasses the test and prod problem. Okay. What's the what what's next?
SPEAKER_01What what what are you guys doing next with the with this technology?
SPEAKER_00What is AI going to do next for you? That's a great question. You know, one of the things I think is really cool is we have like a long laundry list of like, you know, detections we want to write, you know, areas we want to cover, and it just allows us to move so much faster. And you know, one of the trade-offs we've had to make in the past is where do we spend our fo or where do we focus our time? You know, and and inevitably we always want to focus our time on protecting the company, protecting our our resources. But now with the additional cycles and resources we have, we have time to do things like build customer-facing detections, build detections for our, you know, anyone out there who wants to know how do I write GitLab detections. Like we have the ability to move significantly faster and you know, be able to cover a lot more ground, which is really cool. I also think that instead of being stuck in the detection mindset, we'll be able to move toward we'll be able to shift left. Move more toward defense, right? Okay. And yeah, it can help us get there as well, which is really cool.
Metrics That Prove Value
SPEAKER_01Is is are you what kind of metrics do you track? I'm just kind of curious. And I don't want, don't give me any secret sauce. But like, what do you track when it comes down to like are we doing when somebody says are you doing better than you did last month? Yes. Yes. What do you tell them?
SPEAKER_00Like, what's the metric that you track? Yeah. So, you know, I mean, I think one of the strongest metrics of how well are we doing is tracking the number. Of incidents that were detected because of my program. Because of the detection program. Okay. Because ultimately incidents are like, you know, what we end up spending our time on. And those are the real exposures. So what percentage of those came from a detection that we have internally? Okay. And tracking that and increasing that number. I want that to be a pretty high number. I don't want it to be, you know, 10, 20%. I want it to be significantly higher than that.
SPEAKER_01Okay. Are you are you are you decreasing, are you measuring things like the decrease in noise and false positives?
SPEAKER_00Correct. Yeah. So like, you know, true positive, false positive ratio is is a pretty pretty uh standard one. We're tracking that. We're also tracking like, you know, our coverage against things like miter and and you know and stuff like that, you know, number of detections. But I think, you know, if if I was to compare, you know, percentage of incidents that we catch compared to noise, I would really focus on number detector number of incidents that we detect because today we're using AI to really reduce the burden of noise for our engineers, right? So yes, noise is something we always want to reduce, but what do I care more about? Catching more incidents or reducing noise? I'm always gonna go for catching more incidents.
SPEAKER_01Right, right. But I mean, like, I think what we're learning is with this technology, you don't it does it's not an either or anymore. For sure. It's both. It is both. Which is which is kind of freeing. So now you can actually go be go do the job that that you actually that your creative mind is able
Prompting For Useful Summaries
SPEAKER_01to do. Are we are we maybe a physical philosophical question, but are we asking, broadly speaking, as an industry AI to solve the right problems for us or address the right issues? Because you feel it seems like you've created a you've got it focused on like the right things, right? The the reducing the the burdensome turning the crank kind of work while the humans focus on the things that machines can't do, and I would argue hopefully don't ever get to do, which is being creative, like actually creative. Right.
SPEAKER_00Yeah, and I I think I think that's the right mental mindset to be in. I'll also say we we use I AI heavily to go out and find the interesting things that are happening so we can kind of analyze like, is this relevant to us, right? Like I've I've got like these these agents that go out and like kind of you know peruse like the news and all of the different you know sources out there and and match it up to what's relevant to GitLab and what we care about, right? And so doing that type of like mass analysis is something that like you'd have a person do in the past and no one liked doing that. It was sort of a tedious manual job. Yeah, hey, I can automate that and then I can look at a summarization and say that looks really interesting. I want to dive into that one and learn more.
SPEAKER_01How do you get summaries that aren't more than the original thing? Just because one of the things I I keep saying, uh people are like, 'cause I've asked for summaries like over of email of just run my regular email. Like, hey, summarize this email for me. And it it's longer than the original email. Like that's not a summary.
SPEAKER_00Yes, for sure. You've made it longer. This isn't helpful for sure. And I would say it's all about being really specific what you want. Okay. You know, treat it like an eager, uh, an eager, you know, tier one analyst or an eager, you know, someone right out of college, and they just want to, you know, please you with whatever you're asking for, not necessarily thinking about like, you know, do I want a B L U F update or something like that? And so an example of what one of the things that I do that works quite well is I have a I use Cloud Cowork, I have a Cloud Cowork automation that goes out, looks for all the interesting relevant updates, you know, in the news, and then summarizes them for me and says, okay, are these relevant to threats that I care about? And I have given it very specific guidance on, you know, give me a bullet point format, you know, no more than two to three sentences per bullet point. Okay, you know, organize them based on criticality, and attribute it to what's relevant to me and you know our organization. And that does really well because I give it very specific guidance, you know, open-ended question, very specific guidance on what I want back.
SPEAKER_01So that that's that seems to be like the that seems to be the winner is ask, ask exactly what you want, and you'll get exactly what you ask. If you're generic, the response is often unpredictable. Yes, sure. So that that makes us that makes it kind of interesting because I think the the the who the the winner, shall we say, or the person that's best or the team that is best utilizing stuff is the one that understands its limitations and capabilities the best. Yes. Not necessarily the most talented.
SPEAKER_00Does that make sense? Correct. Yeah, I think that's fair. And I mean it's so much of it is about prompt engineering, you know, like building your skill files. Like, you know, with with like my AI agents that I'm using every day, I have like, you know, a directory of skill files based on the different teams, what I want for each team, you know, kind of where our resources are, what our standards are, and all this type of stuff. And so, you know, it's really about being very specific, and that produces more deterministic results. The the the better you are with you know the description of what you need, the better, the the more repeatable and you know, like uh anticipated the results. Uh yeah, uh predictability is good in this field.
SPEAKER_01Do you feel like and this is something that's been bandered about too?
AI As Superpower Not Crutch
SPEAKER_01Do you feel like with the with with I don't know the all the agents that we can give to somebody let's let's take the example of like, hey, I'm uh I'm just getting into this field. Yeah. Is this going to can can we keep AI from being a crutch and be a superpower instead? Because it's easy to just rely on it and actually not not do the work. Right. Because it'll do the work for you, but it's not really, because you just said a whole bunch of stuff that like if you don't actually have the knowledge, you're not gonna get good results back.
SPEAKER_00And then that that's what I think is really important for like you know, new graduates and and new professionals in the field not to go down the road of you know, vibe vibe coding, vibe detecting their way to detecting, you know, vibe detecting their their way to success. Like if you lose the ability to really know what you're doing, then you don't know how to validate that the results AI is giving you is accurate and what you really want. And so that that's the thing that I think it's still really, really, really important for you to know what you're asking for, for you really to know how to validate what it's returning. Because if you lose that, then what's what's the value of us humans, right?
SPEAKER_01Well, and and if you can't like, and that's that's what happens is you're and when we did this to ourselves a little bit in the early 2000s, where we're like, oh, just offshore everything to somebody that this how hard can this be? Just give them the script to follow. You're like, yeah, but you can't you can't outsource critical thinking. Like you can't give that to somebody else to do on your behalf. And this is what we're finding is I I think I agree with with where you're going with it too, is like I I like giving the people that that start a really, really big push, yeah, but then being super careful that they don't then say, well, this is all I need to do, and like just let the computers do it for me. That's a that's a fast that's like that's like saying I'm training for a marathon by getting on the moving staircase stairway or moving moving sidewalk. You're like, well, this is good enough. I can move for it.
SPEAKER_00I was gonna say, I'm I'm a big fan of the work that your agents do is still attributable to you. That's that's that's you cannot, you know, stand up and say, no, I think this is the right thing, this makes sense, you know, what it did make sense, and I validated the results, then you shouldn't be, you know, producing it, even having your agents do it.
SPEAKER_01If you can't understand and explain what it is your agents produce, who's running who here? Exactly. 100%. 100%. Yeah.
SPEAKER_00And honestly, if you get in the real world, you should expect your manager or your leadership to say, what is this? Does this make sense and ask questions? And if you can't, if you can't, you know, reason through the logic of what you're doing, yeah, that that's a tough place to be.
SPEAKER_01Then you're then yeah, then you're just a really, really bad that that goes back to like bad bad engineers or just bad engineers sometimes. Right. No matter how much superpowers you give them for sure. All right, last question because I'm
What Threats Look Like Next
SPEAKER_01running out of time. Yeah. Look down the look down the road a little bit. What do we what's what's coming in in your space? Like, how does this get better? Does it get better?
SPEAKER_00It's a great question. I think you know, AI really helps us move with more velocity. But one thing that I think is really interesting is we I think that the attack surface and the types of you know vulnerabilities, the types of threats that we're gonna see is only gonna continue to increase. Like me and my boss just gave a talk about you know protecting your source code and keeping it safe from attack. You know, GitLab in the last three months paid out over half a million dollars in bug bounties, right? So that's significantly increased over the normal baseline. So, I mean, so all of that to say, you know, our attackers are using magentic AI, our legitimate, like our ethical hacker one researchers are using magentic AI, our the people who work at our company are using magentic AI, everyone is using that. So it the the chance of you know unintentional like disclosures of information, you know, oh my goodness, by mistake, I wrote something that was vulnerable and it introduced uh you know an attack on my system.
SPEAKER_01That's the nightmare.
SPEAKER_00There are so many new vectors that I don't think we even realize yet that are coming and are already like starting. And so I think like detection engineering is the scope is just gonna continue to increase and it's gonna be so much more.
SPEAKER_01Huh. Well, this has been fun. I enjoyed it. It was great. Thanks for coming on, man. This has been good. I'm glad to have you. I certainly learned a little bit, and this wraps up the five-part miniseries, I guess, from B Sides of Atlanta. Folks, thank you for listening. Matt, thanks for obviously thanks for joining us. Uh, keep up the good work. I look forward to having you back on, talk more about this in the future. Folks, thank you for
Wrap Up And Listener Requests
SPEAKER_01listening. Hey, subscribe to the podcast if you haven't already. Check out the YouTube page I have. I don't do a great job maintaining it, but if you all like the videos, I'll I'll put more effort into it. And then, you know, like uh if you get a minute, we'd love a review if you enjoy the content. That's how others find it. So until next time, catch you another time, another place on another down the security rabbit hole podcast. Ciao, everybody. Bye-bye.
SPEAKER_02This has been a month. Thanks for listening. Don't forget to leave my dad a review and share this with your friends. Bye.