A credit-scoring system determines whether people are granted a loan. It makes recommendations that determines credit-worthiness. It does so by using historical data on people's profiles and a set of rules established by the financial institution. What if an attacker tampers with the training data that results in distortion of loan approval outcomes?
Compared to traditional software, AI specific cyber risks increase dramatically due to AI system dependency and reliance on data for training tasks. In addition, the AI supply chain risk become more complex due to increased interconnectedness of AI systems. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.
Trustworthiness
What does trustworthiness means in the AI context? According to NIST, the characteristics of trustworthy AI systems include: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed.
Identifying and managing AI risks and increasing trustworthiness requires broad focus on the AI lifecycle and actors that take part in it. Below is a view of AI lifecycle as defined by NIST.
The two inner circles show AI systems’ key dimensions and the outer circle shows AI lifecycle stages. Ideally, risk management efforts start with the Plan and Design function in the application context and are performed throughout the AI system lifecycle.
AI Cyber Risks to Consider
AI cyber risks need to be identified and mitigated throughout the lifecycle and relevant supply chains. Below are some cyber risks specific to AI systems and questions you can ask yourself to begin the journey of risk mitigation:
Data Poisoning: This involves tampering with the data used to train an AI system. AI trained on poisoned data increases the error rate and biased outcomes. An attacker can corrupt a data source and affect multiple AI systems that rely on it.
If your data is poisoned or tampered with, how would you know?
What kind of input validation/sanitization are you doing on that content?
How do you recover from adversarial contamination of your training data?
Model Stealing and Inversion Attacks: AI systems' training models can be targeted for stealing, copying, or reverse-engineering. An attacker could potentially extract not only the algorithms but also the training data used to train the model directly. This becomes a data privacy issue.
What is the impact of your training data being recovered by attacking/inverting your model?
How strong are the gates between the users and the model e.g. input validation on queries, returning minimal amount of useful information etc.
Can attackers recreate your underlying model by legitimately query the model?
Compromised Components: AI systems typically have numerous components – hardware, software, data sources, etc. – that are often sourced from different suppliers. If any of these components are compromised, it can put the whole AI system at risk.
Do you conduct rigorous due diligence for third parties performing model training to prevent trojaned models?
How dependent are you on your third parties for models and training data?
How secure is the third party training data provider?
Third-Party Software Risks: AI often relies on third-party software libraries and tools. Vulnerabilities in these tools can be exploited to attack AI systems.
Do you know third party party and open source dependencies present in your AI system?
Do performed software vulnerability testing for first and third party code?
Do you have secure software development lifecycle in place to mitigate traditional software supply chain threats?
Insecure APIs: APIs are often used for AI systems to interact with other systems and share data. If these APIs are not secured, they can become a point of entry for attackers.
Which customers/partners are authenticated to access your model or service APIs?
If you train against online data stores, what steps do you take to ensure the security of the connection between your model and the data?
AI Bias: If the AI supply chain has built-in biases (such as bias in data sets used for training), these biases can be exploited by attackers to manipulate the AI system's behavior. There is a risk that AI systems could potentially lead to unfairly biased outcomes for individuals and/or organizations. Furthermore, AI-driven unfairly biased outcomes could have privacy compliance implications, constitute regulatory, litigation and reputational risk, impact operations and result in customer dissatisfaction and attrition.
How do you handle properly formatted but overtly biased data, such as from trolls?
Lack of Transparency: AI systems, particularly those based on deep learning, are often "black boxes," meaning their decision-making process is not easily understood. This lack of transparency can make it difficult to detect when an AI system has been compromised.
Artificial intelligence (AI) technologies hold big promise for the future, but it also bring risks that must be addressed with the right governance approaches. To mitigate these risks, it is critical to implement strong cybersecurity measures across the entire AI supply chain. This includes secure coding practices, thorough testing, regular updates, use of secure data sources, monitoring for unusual activity, and educating all stakeholders about potential cyber risks.
Resources
NIST Artificial Intelligence Risk Management Framework (AI RMF 1.0) is a great resource for organizations who are designing, developing, deploying AI systems.
NIST Artificial Intelligence Resource Center includes the framework and playbook for achieving the outcomes laid out in the AI RMF.
Microsoft AI threat modeling guidance includes good guide for AI/ML-specific Threats and their Mitigations
The OECD Framework for the Classification of AI systems includes a detailed .pdf document on AI systems classifications including real life case studies.
Comments