Unmanaged AI use could trigger India’s next big data breach, warns IDfy COO

As enterprises accelerate GenAI adoption ahead of full DPDP enforcement, unmanaged AI usage and “shadow AI” pose growing privacy risks, believes Malcolm Gomes, Chief Operating Officer at digital identity verification firm, IDfy.

“The next major privacy breach may not begin with a hacker. It may begin with an employee pasting customer data into a public AI tool to get work done faster. The real risk is not AI itself, but unmanaged AI usage," said Gomes.

The warning comes as enterprises rapidly adopt generative AI across customer onboarding, fraud detection, identity verification and business operations, while simultaneously preparing for full implementation of the Digital Personal Data Protection (DPDP) Act before 2027.

The shift highlights a growing challenge for Indian enterprises: balancing AI-driven innovation with privacy and regulatory compliance. Industry experts say organisations are moving beyond traditional concerns around consent collection to broader questions about how data is used, stored and governed throughout the AI lifecycle.

According to Gomes, generative AI has outpaced enterprise governance frameworks, creating blind spots around where customer data is being shared and how it is subsequently handled by third-party AI platforms.

Once personally identifiable information (PII) enters an uncontrolled AI environment, companies often lose visibility over where that data resides, how long it is retained, whether it is used for model training, and how it can be deleted or recovered later, he said.

The concern is not merely theoretical. Citing industry estimates, Gomes said roughly 15% of employees have entered sensitive information into public large language models (LLMs), while nearly 40% of organisations have reported at least one AI-related privacy incident.

The growing use of what experts describe as "shadow AI" is emerging as a major governance challenge. Similar to shadow IT, employees can access AI tools without formal approval from procurement or IT departments, often using them for seemingly harmless tasks such as summarising documents, translating content or generating responses.

"Shadow AI is the new shadow IT. The difference is that it takes just one browser tab for sensitive customer data to leave the enterprise control environment," Gomes said.

He noted that existing cybersecurity controls, such as firewalls, endpoint protection and network security, are not designed to address situations where employees voluntarily share data with external AI platforms.

As a result, enterprises will need to implement new safeguards, including prompt-level monitoring, data classification systems, access controls, AI usage policies and privacy-by-design workflows.

The rise of GenAI is also forcing organisations to rethink their approach to DPDP compliance. Traditional privacy programs were built around structured data flows involving databases, applications and known vendors. AI introduces additional layers, such as prompts, chatbot logs, retrieval systems, AI agents, and model outputs that may not be captured in conventional data inventories.

For organisations preparing for DPDP compliance, key questions now include whether personal data used in AI systems aligns with the purpose for which consent was originally obtained, whether deletion requests can be honoured once data enters AI workflows, and whether adequate audit trails exist to demonstrate compliance.

Gomes argued that DPDP readiness can no longer remain a legal or compliance exercise alone.

"Organizations that get this right will not treat AI governance and privacy compliance as separate programs. They will build them together," he said.

Industry observers increasingly view privacy-first AI as a business differentiator rather than a regulatory obligation. As boards and regulators pay closer attention to AI-related risks, digital trust is emerging as a strategic priority alongside innovation.

According to IDfy, enterprises will require three foundational layers to scale AI responsibly: AI governance frameworks that define approved tools and use cases, consent orchestration mechanisms that ensure lawful and purpose-specific data use, and internal data controls covering masking, redaction, access restrictions, and audit logging.

With India's privacy regime maturing alongside accelerating AI adoption, organisations that can demonstrate transparency, traceability and verifiable control over how AI systems use personal data will be better positioned to earn customer trust and avoid regulatory scrutiny.

"The future of privacy governance will not be about slowing AI down. It will be about creating the control layer that allows AI to scale safely," Gomes said.