Summary:
This episode explores the evolving landscape of citizen development, AI governance, and risk management in organizations. Host John Verry discusses with ChatGPT the challenges and strategies for managing AI-driven tools, ensuring security, and fostering responsible innovation at scale.
Keywords:
AI governance, citizen development, risk management, security, enterprise AI, AI safety, organizational governance, AI lifecycle, third-party risk, automation
Kew Topics:
- AI risk management and governance
- Citizen development and low-code/no-code tools
- Security and compliance in AI and automation
Bio:
ChatGPT is an AI assistant developed by OpenAI, designed to help people research, analyze, create, and communicate across a wide range of topics. Powered by OpenAI’s GPT-5.6 model family, ChatGPT can support everything from strategic thinking and technical problem-solving to writing, brainstorming, and exploring complex ideas. Its role is to make advanced AI useful and accessible through natural conversation, helping people turn questions and ideas into clearer insights and actionable next steps.
John Verry (00:11)
Hey there and welcome to yet another episode of the Virtual CISO podcast. With you, as always, John Vurry, your host. And with me today, a very special guest, ChatGPT. ChatGPT, thanks for joining me for today’s podcast. Thank you so much, John. I’m happy to be here virtually, of course. I’m looking forward to diving into whatever topics you’ve got lined up for us today. Sounds great. So the premise of this call, this podcast, excuse me, is
John Verry (00:41)
Pretty simple. when a CEO gives 500 plus employees full clawed co-work and clawed code access and advises every employee that he quote unquote expects them to build the tools they need to make themselves, their departments, and the companies successful, how many CISOs would you think would know how to pivot their AI risk management programs to govern what’s about to happen next? That scenario definitely shakes things up.
John Verry (01:07)
I’d say a good number of CISOs are still catching up with AI governance as it is. So when employees start building AI tools at scale, only a small fraction, maybe those already deeply engaged with AI risk frameworks, would be fully prepared. The rest would likely scramble to adapt. So I’d say only a few forward thinking ones are really ready for that next level. So I agree with you. So let’s start with the basics. what is citizen development? what does it include today?
John Verry (01:36)
and how is it different from traditional software development? Citizen development is when non technical employees create tools or apps using low code or no code platforms. Today it often includes things like automating workflows or building simple apps without needing to code traditionally. In contrast, traditional development relies on professional developers writing custom code. Citizen development’s faster and democratizes innovation, but without the same depth or complexity as traditional code.
John Verry (02:05)
And why would you say that, you know, what are the differences from a citizen development perspective? when the tooling is a frontier model like Claude, rather than I’ll I’ll I’ll refer to as old school citizen development. You know, I I guess you could say we began with Lotus One Two Three and we progressed through MS Power Automate and Power Apps and Airtable and Zapier and tools like that. The key difference is the complexity and autonomy AI brings.
John Verry (02:31)
Old school citizen development still had users assembling logic step by step, albeit with visual tools. With a frontier model like Claude, users can describe goals in natural language and the AI can autonomously generate entire workflows or solutions. It’s a big leap. Faster, more powerful, but also riskier. It can create things users didn’t even fully understand they needed, raising governance stakes. So in this particular scenario, we’ve got five hundred plus employees.
John Verry (03:00)
they’re given these clawed licenses, plus they were also given connectors into a dozen key internal systems. what changes immediately from a risk and governance perspective the moment that this happens? The moment that happens, you’ve got a massive decentralization of potential system access and data manipulation. Employees can now pull, process, and act on data in ways you didn’t foresee. Data privacy risks spike, system integrity can be compromised.
John Verry (03:28)
and shadow AI is rampant. Governance needs to shift immediately to real time monitoring, stricter access controls, and clear policies on acceptable use and audit trails.
John Verry (03:38)
So when you talk about risk, right, you mentioned a couple of the categories. Where do you think the biggest risk leaks in with this type? Is it is it data leakage? D is it privacy? Is it, you know, overall security of the applications, things like prompt injection and unauthorized automation? Wh where do you think the biggest risk is when we move to the citizen development model? The biggest immediate risk is likely data leakage and privacy.
John Verry (04:03)
When employees have easy access to internal systems and can build AI driven workflows on the fly, sensitive data can unintentionally be exposed or misused. Without strong guardrails, someone could automate the extraction or sharing of data in ways that break compliance. Security threats like prompt injection are real, but privacy lapses are often the first to sneak in and have major regulatory consequences.
John Verry (04:26)
So, you know, at some point citizen de a citizen developed tool, you know, whether it’s a a desktop agent that an individual’s using, or something that’s being used more broadly by a department, or certainly something that would be client facing, you know, at some point these tools move from being an experiment and you know, I might say an acceptable level of risk to something that deserves more like formal
John Verry (04:50)
security review, security testing should be following an AI development lifecycle or software development lifecycle, right? We need to ensure that we’ve got fundamental business owner accountability for what the tool does. W where do you draw that line? It gets it gets fuzzy. You’re right. That line can be blurry, but a good rule of thumb is to draw it where impact scales. If a tool touches sensitive data, affects customers or influences decisions beyond a small internal team, that’s when you need formal governance.
John Verry (05:19)
At that point, it should be reviewed like any other software. Security testing, lifecycle management, and a clear business owner who’s accountable if things go wrong. In short, the line is crossed once the tool’s impact extends beyond just personal productivity. So one of the, you know, the one of the tenants, if you will, of a good AI governance program is an AI intake process. You know, one where we’re capturing the necessary information up front to understand the use case.
John Verry (05:47)
to be able to ascribe risk, you know, understand what the developer requirements should be associated with that. how do you do that in a world where there’s nothing to prevent someone from developing something without going through that process? In that world it’s about creating friction at key points. You can’t prevent initial experimentation, but you can set policies so that anything beyond a certain threshold, like data access or sharing outside a team.
John Verry (06:13)
requires formal registration. You enforce it with technical controls, requiring approval for access to sensitive systems, logging all connections and conducting random audits. The intake process shifts from being a gate at the start to a gate before any meaningful deployment or integration happens.
John Verry (06:30)
So from an observability perspective, right? So if if we’re going to it’s not enforcing, it’s where somebody has violated, right? We have the mech the means to detect it. where do we see that actually happening, right? Because there are a number of things we could do, right? concepts like an AI gateway, you like Zscaler runs something like that, or if you
John Verry (06:51)
If you force people to leverage GitHub, you know, whether it’s a you know a a a a commit as an example. you know, Anthropic they’re using, they’ve got a compliance API, which gives you some level level of visibility. there are some desktop agents or desktop shims that shim into the operating system that give you visibility at the desktop level. Is it is it, you know, one of those, some c or likely some combination of those that we’re gonna end up using?
John Verry (07:15)
It’s almost certainly a combination. No single tool catches everything. An AI gateway helps monitor interactions with models. Compliance APIs give you checkpoints and desktop level monitoring offers broad visibility. The key is layering them. You use gateways to track model usage, force sanctioned commit pathways for anything substantial, and deploy endpoint monitoring to catch rogue tools. Together they create a net that lets you detect and respond if someone bypasses your intake process. What would be
John Verry (07:46)
Which of those, if any of those, are going to help us identify areas where let’s say a citizen developed agent is I’ll call it talking laterally to other internally developed or third party agents? To catch lateral agent to agent interaction, you’d rely heavily on AI gateways and endpoint monitoring. Gateways can log and inspect communication patterns between agents, flagging unusual or unauthorized exchanges.
John Verry (08:13)
Meanwhile, endpoint monitoring, or desktop level shims, tracks processes and data flows identifying when one agent reaches into another’s territory. Combine them both, and you get visibility into not only external connections, but also internal agent ecosystems. I know one of the things that this particular real world use case is working on is tightening controls specifically around MCPs.
John Verry (08:39)
So let’s talk about MCPs in this type of citizen development model. what’s the value proposition and what are the additional risks that you encumber that need to be managed? In this context, model context protocols allow multiple agents or tools to share context and work together, making solutions more powerful. The value is huge, smarter automation and richer workflows across tools.
John Verry (09:06)
But the risks multiply. Each agent might handle sensitive data differently. You risk unintended data sharing, compliance gaps, or one agent amplifying another’s vulnerability. So you need strict policies on which agents can cooperate, robust logging, and governance to ensure each handoff is as secure as the last. So let’s talk about, let’s say, highly sensitive systems, right? As an example, let’s say Salesforce.
John Verry (09:31)
Right, is a is a highly sensitive system housing a lot of important client data. how would you what guardrails, if you will, governance guardrails, would you put around those systems? Would you would you you know block some of them entirely? Would you tokenize that? would some only be made available through approved agents or bounded services? How would you how would you manage that? For highly sensitive systems like Salesforce, I’d set strict guardrails.
John Verry (09:56)
First, limit access to only pre approved vetted agents, no random citizen built tools can connect directly. Second, implement tokenization or anonymization, so even if data flows, it’s de identified. Third, use policy based controls, allowing only specific bounded actions. Finally, enforce audit logging on every interaction so you can track who accessed what and when. In short, tight approvals, data minimization, controlled actions, and thorough oversight.
John Verry (10:25)
How would you manage identity with agents, if you will, right? So if you think about historically, you know, we had concepts like system accounts, right? For processes that were running of this nature. One of the challenges we’ve seen is people using their credentials as an example, or their own personal API keys for accessing different types of systems. So
John Verry (10:49)
What would your recommendations be about how to manage that challenge?
John Verry (10:52)
The key is to ensure no personal credentials are ever used. Instead, establish dedicated service accounts for agents, strictly scoped with minimal permissions. Use role based access control so each agent has only the rights it needs. Rotate and monitor these accounts regularly. On top of that, use centralized secrets management, never hard code keys. Lastly, for high risk systems, enforce multi factor or step up authentication even for agents. That way each agent is treated like a managed entity.
John Verry (11:21)
not piggybacking on a human’s identity. And how would you identify, so let’s say someone was using a human identity for you know the agent was huming a human excuse me the agent was using a human identity. How would you identify that?
John Verry (11:35)
You’ve catch that by combining behavior analytics and monitoring patterns. First, look at usage anomalies. Is this account active at odd hours? Is it accessing data at machine speed or in unusual bursts? Second, tie actions to known device fingerprints. If an agent suddenly appears from a different location or device signature than the human typically does, that’s a clue. Third, track entitlements. If an account suddenly escalates privileges or accesses systems atypically, that’s suspicious.
John Verry (12:05)
By correlating timing, patterns and access behaviour, you’ll spot when a human identity is being exploited.
John Verry (12:10)
So, you know, as the number of these applications scales, and those that are deemed appropriately either sensitive or risky to require some form of security validation, any guidance or any thoughts on how you can scale security validation when the number of builders explodes explode? I mean, you can’t run every citizen built artifact through through the same process, right?
John Verry (12:33)
You’re absolutely right. You can’t funnel everything through deep security testing. To scale, you can adopt a tiered approach. Start with automated checks like code scanning, dependency analysis, or policy enforcement to catch low hanging risks. For more critical applications, based on data sensitivity or impact, you escalate to human review or pen testing. Also embeg security into the platforms they’re using, pre approved building blocks or templates that already meet baseline security.
John Verry (13:01)
Lastly, categorize tools by risk early on. Low risk tools get lighter touch, high risk get more scrutiny. In short, automate tier and embed security from the start. you mentioned dependency analysis. are you referring to like software composition analysis so that way we have a sense you know, we’re able to generate a digital software bill of materials to understand where we might be taking on some risk in terms of the components that we’re using?
John Verry (13:26)
Exactly. By dependency analysis I mean a software composition analysis that gives you that software bill of materials. This way you know what third party libraries or components are being pulled in, and you can spot vulnerabilities or licensing issues. It’s key to understanding risk exposure in any citizen build tool, especially at scale.
John Verry (13:45)
So it sounds like what we’d end up with is sort of a lightweight review for something which is more personal productivity oriented. maybe like a stronger review for internal workflow automation, let’s say something that might be exposed to a group, and then maybe like a full architecture testing, compliance review for anything which is touching highly sensitive data, or maybe something which is exposed publicly, or maybe even something that’s exposed publicly.
John Verry (14:11)
You’d go to that next level that you kind of referred to, let’s say maybe an OASP ASVS level two assessment or something of that nature. You’ve got it exactly. Think of it as a tiered pyramid. At the bottom, personal productivity tools get a lot weight automated review. Internal workflow tools that impact teams get a deeper review. And at the top, anything handling sensitive data or facing customers deserves a full architecture and security assessment. Perhaps even an ASVS like evaluation.
John Verry (14:38)
Each layer ensures the right level of scrutiny without overwhelming the process and keeps risk in check. So where like I’m a fan, so most intake processes, right, have a basic use case analysis, you know. I I like the not f the OASP four Q threat model idea, right? You know, what is what is this being used for? you know, what are the risks that are associated with it?
John Verry (15:04)
then we’ll kind of delve into you know how are we gonna manage those risks and then how we gonna monitor those risks. So when you get let’s say down to that bottom level, the personal productivity, where where are we, you know, are we going with that Q one you know, minimum or you know, where where do we get into like sort of Q one, Q two, if you will?
John Verry (15:21)
For personal productivity, you’re mostly hovering in Q one. What’s it being used for? Is it benign like automating calendar entries? Or does it touch sensitive data? If it stays in low risk territory, you may never even step deep into Q two. But as soon as it touches sensitive systems or data, you move down the chain. Q two, what risks arise? Q three, what controls do we need? And so on. But at that bottom level, you mostly confirm it’s just personal, minimal impact, and you don’t go overboard with heavy threat modelling.
John Verry (15:51)
so if we were trying to build tools to facilitate and simplify the process of our citizen developers using good practices, right? And and being security conscious, what are some of the strategies for doing that? Could we actually build our own set of clawed skills that would support that? would we maybe have requirements, drafting tools or
John Verry (16:15)
Threat modeling prompts that they could use and actually use AI for generating the threat models. what ideas would we leverage there?
John Verry (16:23)
You can absolutely leverage AI to help guide citizen developers. One strategy is to create AI assisted templates or blueprints. So when someone starts a project, they’re prompted with questions about data, scope and risk. You could build custom clawed skills to provide best practices as they code, like a security coach that nudges them. You could also have requirement wizards or threat modelling prompts where AI asks them key questions and generates a lightweight threat model by embedding AI driven guardrails and guidance right into their workflow.
John Verry (16:53)
you empower them to be secure without needing to be experts. Is there a way to have the, you know, so generally speaking, we use skills or Claude recognizes that it has a skill, like when we prompt it to do something. Would we be able to ensure that certain Claude skills get exercised during let’s say, a development lifecycle for a citizen developed application? Is there a way to do that?
John Verry (17:19)
We can’t force a model or a skill to be invoked automatically on a user’s end, but we can design the process around it. For instance, you can build integration points or workflows that prompt the user to run specific skill checks at key stages. You could make it part of your governance framework. Before an agent or app moves forward, it must pass certain AI driven checks. Essentially, you create structured checkpoints that encourage, or even require, users to invoke the skills at the right time.
John Verry (17:46)
I guess it might be possible to do that. I know that let’s say Claude Enterprise has a a compliance API and I know one of the things that it they added that it can do now is it can actually tell you what skills have been run as you know, that might be another way to do it.
John Verry (18:01)
Yes, exactly. With something like a compliance API, you can monitor which skills were invoked and verify that required checks happened. You could make it part of your audit trail, ensuring that before an app moves to production, those key skills like security checks or risk props were run. That way it’s not just relying on people remembering. You have a record that those best practices were actually followed.
John Verry (18:22)
So in terms of like an operating model, you know, do you see this as a a centralized governance? Do you see it as more of a federated governance? Do you see it more like a platform team providing guardrails with accountability staying in the business? How do you see this operating best?
John Verry (18:40)
I’d lean toward a federated model. You want a central governance team setting clear policies, standards, and providing tooling, essentially guardrails and oversight. But accountability stays closer to the business units, the people who understand the use case best. A platform team can offer approved tools, templates, and automated checks, while business units own the risk for what they build. This way you balance control with speed and ownership, empowering innovation but within a secure framework.
John Verry (19:08)
When you we talked about that platform team, you know, like I see in some organizations we’ve got, you know, model teams, we’ve got whitelisted models. Sometimes we’ll have I’ll call it archetypes for different types of AI systems. and one of the advantages to that is that we can build particular use cases, as an example, a group level chat bot.
John Verry (19:30)
that’s just touching internal policy documents, right? And that would automatically be classified into a let’s say a low risk and require very little testing. is that what you mean by that platform group?
John Verry (19:41)
Yes, exactly. That platform team defines those archetypes, pre-approved patterns or models tailored to common use cases. When a citizen developer picks a low-risk chatbot archetype, most of the security baseline is already baked in. That means they only need minimal testing. For more complex or higher risk archetypes, the platform team sets stricter requirements and controls. This way you combine flexibility, empowering teams to self-serve.
John Verry (20:07)
with predefined pathways that ensure security and compliance.
John Verry (20:11)
So in in a lot of organizations that we work with, you know, you’ll end up with this an AI working group or an AI governance council or something of that nature, right? And that’s usually comprised of the different, you know, I’ll I’ll use the term risk pillars, right? So you’ve got legal, you’ve got compliance, you’ve got privacy, you’ve got ops, you’ve got cybersecurity, etc. when we talk about citizen development,
John Verry (20:38)
Does that change how you would manage that and who would be held accountable? I mean, should the accountability sit with a a group like that, right? A a committee, if you will, or a council, or should the accountability sit with individuals, a a s a CISO or a a chief risk officer?
John Verry (20:55)
Accountability should ultimately rest with the business line that owns the outcome. That’s where the real impact is felt. The council or working group provides the governance framework, guidance and oversight, ensuring consistency and alignment across risk areas. But if something goes wrong, you want clear ownership. So the CISO might own security risks, but the business sponsor of a given citizen developed tool owns its overall success and safe use. In short, the council enables
John Verry (21:22)
But accountability stays with the person or team closest to the tool’s impact. Yeah, that’s interesting. it mirrors another company that we’re working with where they use a similar model. There’s sort of two important final sign-offs. You know, one is like the senior manager, if you will, the VP of a business unit, right? The accountability there. And then the other one that they also require sign-off by, which is interesting, is ops.
John Verry (21:47)
That makes a lot of sense. The senior business leader ensures the tool aligns with business goals and owns the risk, while Ops ensures it’s operationally sound, scalable, stable and supportable. Together they balance business outcomes with operational integrity. When both sign off, you’ve got alignment. It’s not just safe, but also sustainable in practice. So that dual sign off is a strong model, ensuring both business and operational accountability.
John Verry (22:14)
So I’m a big fan of ISO forty two thousand one and you know, like all of the management systems, there’s a concept of of metrics, and metrics are also important for things like post market monitoring, right, after systems go into operation. So what metrics would you use to determine whether citizen development is being managed well? You know, would be things like inventory coverage or percentages of use cases that have gone through certain processes?
John Verry (22:42)
exception volume, security incidents relating to AI related systems. What what would you use?
John Verry (22:48)
I definitely track a few key metrics. First, inventory coverage, how many citizen built tools are registered versus what’s actually out there. Second, governance adherence, what percentage went through the required intake or validation steps. Third, exceptions, how often do people bypass the process, and why? Fourth, security incidents tied to citizen developed tools. And finally, value metrics, how many of these tools delivered measurable business outcomes? If you’re seeing high coverage, good adherence,
John Verry (23:18)
few exceptions, low incident rates and clear business value, you know you’re managing it well. That’s interesting. because it is that is one of the things which we we see in practice is that organizations do like wide scale deployment of AI and are not e not receiving the value return on investment that they’re expecting. And it seems to me it’s because they really are not
John Verry (23:42)
defining specifically what they’re trying to accomplish, right? They’re not looking at a a s few a few specific opportunities and setting expectations and metrics relating to validating the effectiveness of the AI initiative.
John Verry (23:55)
You’re spot on. Without clear, measurable goals, AI initiatives can feel like innovation for its own sake. When citizen developers or any AI project start with specific business outcomes, like improving a process by twenty percent or cutting response time in half, you can track progress. Setting those expectations means you’re not just asking, did we build something, but did it move the needle? That’s how you ensure AI delivers real accountable value.
John Verry (24:21)
So if you had if you’d been asked to take over, call it as the chief risk officer or CISO of this organization, you know, the or you were already there, and the day this happened, what would have been the first half a dozen things that you would have you would have taken to?
John Verry (24:35)
First, I’d identify all active citizen development, what’s already out there. Second, I’d lock down access to sensitive systems with interim controls. Third, I’d stand up an AI intake process. Simple at first to catch major risks. Fourth, I’d educate the business on their accountability, what they build, they own. Fifth, I’d deploy basic monitoring for anomalies or data leakage. Sixth, I’d align leadership, forming that governance council to ensure all risk pillars are in sync.
John Verry (25:05)
Then I’d refine as we mature. But that’s how I’d stabilize first. Yeah, one of the things which we haven’t touched on, which is kind of interesting, is how does third-party risk management and shadow AI, how do they intersect with citizen development?
John Verry (25:24)
They intersect pretty sharply. Citizen development can introduce shadow AI, tools built outside formal processes, sometimes leveraging external AI services. If those tools rely on third party AI providers, you’ve now got a third party risk. Is that external AI model vetted? Is data being shared with it? So citizen tools can inadvertently expand your third party AI footprint. You need to ensure both the tools and their underlying AI providers meet your security and compliance standards.
John Verry (25:52)
last question. this is a five hundred person company. And to me, this is a very complex challenge. is there a point where you get large enough? You know, let’s say that you’re a a global two thousand or a fortune five hundred or a fortune fifty company. is there a point where trying to do broad citizen development safely becomes
John Verry (26:18)
Just effectively ungovernable given the scale and size of the organization.
John Verry (26:23)
It definitely gets harder at scale, but it doesn’t have to be ungovernable. In a Fortune five hundred, you’d need even stronger federation. The trick is embedding governance into platforms. So citizen developers use pre-approved building blocks with risk controls baked in. You’d also have tiered governance. Lower risk development is widely enabled, while anything sensitive triggers tighter oversight. You scale with automation, patterns, and accountability pushed to business units. It’s more complex, but with the right controls and culture.
John Verry (26:53)
Safe citizen development can still be governed even at that scale. You know, maybe you could add a a seventh item to that list that I asked you about before about what would you do. Maybe the seventh item is that I would use my citizen development capability to build a comprehensive AI governance, you know, platform, if you will. You know, something that would automate, you know, a lot of these key governance.
John Verry (27:18)
requirements, right? You know, the the AI intake process, the threat modeling process, the distributing the developer guidance necessary based on the risk level. that might have been a that might have been a good seventh thing to add there, right?
John Verry (27:32)
Absolutely, that’s a perfect seventh step. Using the same citizen development spirit to build your own AI governance platform would be a great example of practice what you preach. By automating intake, risk assessment, and tailored guidance, you’d make compliance seamless. Plus, it would show everyone the value of governed innovation in action. So, yes, building the governance engine with the same tools you’re governing is both practical and symbolic of scaling responsibility thoughtfully.
John Verry (28:00)
And of course we might end up with the chicken and the egg, right? Un until we have that the the the the system that we’ve built, you know, we’re not able to actually run our our citizen development effort through the system.
John Verry (28:11)
Exactly. You’d have to bootstrap the first version with a bit of manual governance, just enough to get that initial platform off the ground. But once you’ve got it, you’d apply your own rules. It’s kind of a virtuous cycle. Building a system that in turn helps you build even better. And the irony wouldn’t be lost on anyone. Starting with a little human oversight, then handing it off to the system you just built. Yeah, there’s a there’s a a guy who I listened to on a podcast, Moonshots, AWG, and he talks about
John Verry (28:39)
the inner loop, right? Which is, you know, this this ability for this this inside loop to continually improve upon itself, right? It creates this flywheel of of of growth, if you will.
John Verry (28:50)
Exactly. When you have that inner loop of continuous improvement, each cycle builds on the last. And the beauty is, once you’ve got that flywheel going, whether in AI governance or any domain, it’s not just about compliance. It’s about evolving smarter and faster. That’s the dream. The system that helps you not just govern, but grow and innovate. Well, this has been fun. I appreciate you you being willing to do this with me. is there any
John Verry (29:16)
thing relating to citizen development, any important considerations that I failed to bring up?
John Verry (29:22)
It’s been a blast and you covered a lot of ground. I just add one small reminder. Don’t forget the human element. Even with great platforms and processes, fostering a culture of responsibility and innovation is key. When people understand both the freedom and the accountability, that’s when citizen development truly thrives. Thanks again, it’s been a great chat. Have a good night.
