Artificial intelligence (AI) is revolutionizing every sector. The complex AI training environments that power these advancements involve high computational power, large amounts of data, and sophisticated algorithms. However, the security of these environments is often overlooked.

This paper explains the essential elements of AI training and finetuning environments, followed by an examination of critical security concerns and best practices for securing these environments.

Understanding AI/ML Ops

Artificial intelligence/machine learning operations (AI/ML Ops) is an emerging discipline that brings together best practices from machine learning, DevOps, and data engineering to streamline and optimize the lifecycle of machine learning (ML) models.

AI/ML Ops Components

The AI/ML Ops framework takes an AI development project from the initial identification of a problem that needs solving, through testing and deployment, and finally to managing the scalability, costs, and performance of the model.

Model Development and Training

AI model training is the process of creating a model from scratch using a large dataset. The goal is to enable the model to learn patterns, features, representations, and relationships within the data. Training a generative AI model from scratch is resource-intensive and requires a substantial amount of data and computational power.

The essential tasks of this phase include the following:

• Problem Definition: Identify the business problem or research question the model will solve.
• Model Selection: Select an appropriate ML algorithm based on the problem type and the characteristics of the data.
• Feature Engineering: Create and select the most relevant features to improve model performance.
• Hyperparameter Tuning: Adjust the hyperparameters to optimize model performance.
• Experimentation: Run experiments and compare different models and configurations to identify the best performing model.
• Data Preparation: Clean and prepare data in the correct format for training.

Testing and Validation

Rigorous testing frameworks and validation techniques help in identifying and mitigating potential issues before deployment. Testing and validation methods are used to ensure that models perform as expected and meet the required metrics.

• Performance Evaluation: Assess the model’s accuracy, precision, recall, F1-score, and other relevant metrics. Learn more from these articles:
  • Accuracy vs. precision vs. recall in machine learning: what's the difference?
  • Accuracy, Precision, Recall or F1?
• Bias Detection: Check for biases in model predictions and ensure fairness.
• Robustness Testing: Evaluate how well the model performs under different conditions and with unseen data (data that was not part of the training and is new to the model).

Version control is an essential part of this process. Like software development, version control for AL/ML involves tracking and managing changes to the code, data, and configuration. This makes it easier to experiment and roll back to previous versions if needed. Best practices include the following:

• Track changes in the codebase using systems like Git.
• Track changes in datasets to ensure consistency and reproducibility.
• Log parameters, metrics, and artifacts from experiments using tools like MLflow or Data Version Control (DVC). MLflow is an open-source platform designed to manage the entire machine learning lifecycle. It provides a comprehensive set of tools and frameworks to streamline various stages of machine learning projects, from experimentation to deployment.

Note: Version control is a challenge often observed in open-source models that are available in sharing hubs such as Hugging Face.

Deployment

Once a model is ready, it needs to be deployed into a production environment that is either on premises or in the cloud. This process should be smooth and efficient to ensure minimal downtime and maximum reliability. Key activities include the following:

• Model Packaging: Prepare the model for deployment, often using containers like Docker.
• Infrastructure Setup: Configure the necessary infrastructure, whether on premises or in the cloud, to host the model.
• Continuous Integration/Continuous Deployment (CI/CD): Implement pipelines to automate the deployment process using tools like Jenkins, GitLab CI, or CircleCI. Additional tools for ML and AI include Vertex AI Pipelines, Prefect’s Marvin, Kedro-Airflow, and Metaflow.

Monitoring and Maintenance

After deployment, continuous monitoring is essential to identify shifts in data distribution that could impact model performance and to ensure that the model performs well over time. This includes tracking model accuracy and other performance metrics, monitoring operational issues, and implementing alerts for anomalies. For example, the mlflow.evaluate() API can perform some checks on the metrics that are generated during model evaluation to validate the quality of the model. Tools like Evidently can be used to evaluate, test, and monitor AI systems.

Scalability and Management

AI/ML Ops also involves scaling models to handle increased loads and managing the infrastructure needed to run these models efficiently. Horizontal scaling (adding more instances to handle increased traffic) and vertical scaling (enhancing the capacity of existing instances) can be used to efficiently manage computational resources to optimize cost and performance.

AI Training and Finetuning Environments

The environments used for training and finetuning AI models play a crucial role in determining their efficiency, accuracy, and reliability. AI training environments can be categorized into three types: cloud-based, on-premises, and hybrid. Each type offers advantages and challenges, and the choice of environment depends on an organization's specific needs, resources, and strategic goals.

On-premises environments are deployed within an organization's own data centers. These environments provide greater control over hardware, software, and data security. Cloud-based environments offer scalable, flexible, and cost-effective solutions for AI training and finetuning. Hybrid environments combine the benefits of both cloud-based and on-premises solutions, allowing organizations to leverage the flexibility of the cloud while maintaining critical operations on premises.

Components of AI Training Environments

The entire end-to-end training pipeline that involves the underlying AI hardware infrastructure requires a secure environment. From an AI hardware security and safety point of view, the integrity of the training environment is critical.

Regardless of the type of environment chosen, certain key components are essential for efficient AI training and finetuning. These components include graphics processing units (GPUs), tensor processing units (TPUs), data pipelines, and data storage.

GPUs and TPUs

GPUs are the powerhouses of AI training environments, offering parallel processing capabilities that significantly accelerate the training of complex AI models. Unfortunately, it is currently impossible to run GPU workloads in the cloud without trusting the provider, which eliminates cloud GPUs as an option for security-conscious users.

In the cloud, the provider controls the layer of privileged software responsible for management and provisioning. Even dedicated cloud instances (for example, Amazon’s EC2 dedicated hosts) run the provider’s virtualization software, making GPUs vulnerable to malicious or curious API calls to the client machine GPU. Virtualization software generally runs at a host machine’s highest privilege level, exposing a wide attack surface that includes GPU memory, execution context, and firmware.

Finally, unfettered visibility into host-device communication exposes both data and timing channels. Hence, it is important to secure the AI training pipeline from runtime and from the GPU/TPU hardware perspective as well.

Data Pipelines

Data pipelines are the infrastructure that moves data from its source to the AI training environment, ensuring that data is cleaned, preprocessed, and formatted correctly for training. This process includes the following steps:

• Ingestion: Collect raw data from various sources.
• Processing: Clean, transform, and prepare data for training.
• Storage: Store processed data in a format suitable for training.
• Monitoring: Continuously monitor data flow to ensure data quality and pipeline performance.

Figure 1: End-to-End Sequence Flow Illustration of a Secured AI Training Hardware Infrastructure

Data Storage

AI training requires access to large volumes of data, so the choice of storage solution impacts the speed and efficiency of data access and processing. The datasets that are used for AI training purposes are different from data that exists in other contexts in most organizations. This creates unique challenges for maintaining and managing the datasets in the AI training pipeline.

• Data volume: AI training data is typically massive, often requiring millions or even hundreds of millions of records. Securing and protecting such a vast amount of data is a significant task.
• Disparate data types: AI training data can encompass diverse types of information, including images, videos, audio files, and unstructured data like documents. This variety makes it challenging to assume uniformity across all data records or to support specific technologies without adaptation.
• Velocity: The speed at which data is generated, processed, and analyzed is important for organizations that require real-time or near-real-time data processing to make timely decisions. Velocity measures how quickly data is created and can vary widely depending on the source. For example, network infrastructure devices generate large amounts of data (logs) every second, while other sources might produce data at a slower pace.
• Non-continuous use: Unlike operational data, AI training data is only needed during active model training and intermittent retraining. Storing this data for future use in a cost-effective way is paramount.
• Sensitive information: AI training data often includes sensitive information, such as personally identifiable information (PII) related to customers, vendors, or employees. Proper security and compliance measures must be in place to protect this data from unauthorized access or misuse.

Cloud Storage vs. Local Data Center

In most cases, the cloud is the right place for your AI training environment. Lower up-front costs, the ability to scale up and down, and regional availability are compelling reasons for most startups and larger companies.

But there are a few exceptions to this rule:

• You are operating a very large-scale training process. It may be more cost-effective to run your own storage in your private data center. The exact costs vary based on geographic location and setup, but it typically requires infrastructure spending of more than $50 million per year.
• You need specific hardware that you can’t obtain from a cloud provider. For example, you need types of GPUs, memory, or storage that are not widely available.
• You cannot find a cloud that is acceptable for all the current existing geopolitical considerations.

Before building a data center, conduct a comprehensive price and performance analysis of GPUs for the setup (see Tim Dettmers’s analysis as an example). In addition to the cost and performance of the card itself, consider the costs of power, space, and cooling when making hardware selections. For example, two RTX 3080 Ti cards together have similar raw compute capacity to an A100, but respective power consumption is 700W vs. 300W. The 3,500-kWh power difference at market rates of $0.10/kWh over a three-year life cycle increases the cost of the RTX3080 Ti by nearly two times (approximately $1,000).

Cloud-Based Environments

Many organizations have cloud-based or hybrid AI training environments. The advantages of cloud-based environments include the following:

• Scalability: Easily scale resources up or down based on demand.
• Cost efficiency: Reduce upfront capital expenditure with pay-as-you-go models.
• Accessibility: Access the environment from anywhere with an internet connection.
• Integration: Seamlessly integrate the environment with other cloud services and tools.

Let's take a look at what the most popular cloud service providers have to offer.

Google Vertex AI

Vertex AI is an ML platform designed for training and deploying ML models and AI applications, including the customization of large language models (LLMs) for AI-powered applications. It integrates data engineering, data science, and ML engineering workflows to allow teams to collaborate with a unified toolset and leverage Google Cloud's scalability.

Vertex AI offers multiple training and deployment options. AutoML enables training on tabular, image, text, or video data without the need to write code or prepare data splits. Vertex AI also enables custom training and gives you complete control over the training process, which allows you to use preferred ML frameworks, custom training code, and hyperparameter tuning. Vertex AI also has a feature called Model Garden that allows users to discover, test, customize, and deploy Vertex AI models and selected open-source models.

Amazon Bedrock

Amazon Bedrock is a fully managed service that provides access through a single API to a selection of top-performing foundation models from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon itself. It includes a comprehensive set of features necessary for building generative AI applications with a focus on security, privacy, and responsible AI. You can test and evaluate these foundation models for your specific use cases, customize them privately using your data through methods like finetuning and Retrieval Augmented Generation (RAG), and develop agents that can perform tasks using your enterprise systems and data sources. Being serverless, Amazon Bedrock eliminates the need for infrastructure management and allows secure integration and deployment of generative AI capabilities into your applications using familiar AWS services.

Amazon Bedrock handles data processing within the AWS environment. It eliminates the need for users to manage infrastructure but gives users limited direct control over where their data is processed. Bedrock guarantees that user data is not used to train the foundational models and ensures that all data is encrypted and remains within the user's virtual private cloud (VPC). Organizations with very high security needs should consider these factors carefully.

Amazon SageMaker

Amazon SageMaker, on the other hand, is a comprehensive service designed for data scientists and developers to build, train, and deploy ML models for a wide range of use cases. It supports the entire AI/ML model lifecycle, offering tools to label and prepare data, select algorithms, train models, tune and optimize them for deployment, make predictions, and take actions based on those predictions. One notable feature of Amazon SageMaker is JumpStart, which provides a quick start to AI/ML projects by offering pre-built solutions and trained models. This capability accelerates project initiation and lets developers get started quickly without the need to develop everything from scratch.

Unlike Amazon Bedrock, Amazon SageMaker gives customers full control over their data and the underlying infrastructure. This control includes deciding where data is processed, encrypting data at rest and in transit, managing data access through identity and access management (IAM) roles, and complying with regulations using AWS’s extensive compliance offerings. Additionally, users can operate SageMaker within their VPC to maintain network-level control. This level of control is crucial for customers with stringent data security requirements.

Ultimately, both services provide good security features, but the choice between them will depend on how much direct control over data and infrastructure your organization needs.

Microsoft Azure AI Studio

Microsoft Azure AI Studio is a powerful platform designed for creating advanced AI solutions across various domains such as enterprise chatbots, content generation, and data analysis. It integrates a comprehensive suite of tools and makes them accessible from a centralized interface. These tools include the following:

• Azure Open AI
• Azure Machine Learning
• Microsoft Copilot for Security
• Azure AI Content Safety
• Azure AI Document Intelligence
• Azure AI Language
• Azure AI Translator
• Azure AI Search
• Azure AI Speech
• Azure AI Video Indexer
• Azure AI Vision

Users can easily deploy and test AI models — either pre-built models (such as OpenAI, Nvidia, Hugging Face, Mistral AI, and Deci AI, accessed through the model catalog) or custom ones created on top of existing frameworks. Training and evaluation are supported, with real-time feedback mechanisms for monitoring metrics such as accuracy, loss, and validation scores. This iterative approach helps enhance model performance through continuous refinement.

Developers can set up isolated test environments and orchestrate complex application scenarios for thorough testing and evaluation. They can integrate custom data sources such as databases, files, documents, and web addresses to enhance model prompting and interaction. A standout feature of Azure AI Studio is its support for prompt flows, which enables users to customize model behavior through intuitive flowchart-based interactions, enhancing usability and control. Additional features include an AI playground for experimenting with prompt flows, managing data, and customizing parameters like temperature (randomness when picking words during text creation) and top P (how many possible words to consider) to suit specific applications.

For seamless integration into applications, Azure AI Studio offers endpoint APIs for programmatic access to projects. In terms of security, Azure AI Studio incorporates robust features such as content safety filters and stringent access controls to protect sensitive data and ensure compliance with regulatory standards.

Microsoft Azure OpenAI

In Azure AI Studio, users can choose from various language models provided by companies like Mistral, Meta, and OpenAI. However, Azure OpenAI distinguishes itself by offering exclusive access to all OpenAI models. Both Azure AI Studio and Azure OpenAI support features such as finetuning and content filtering.

Azure OpenAI not only provides access to a wide array of OpenAI models but also offers features that enhance AI development and deployment. These features include advanced model customization options, seamless integration with Microsoft's broader ecosystem for enhanced scalability, and security features to safeguard sensitive data throughout the AI lifecycle. It also enables AI developers to easily incorporate AI capabilities into their applications through different APIs.

Note: The shared responsibility model applies to data protection in cloud AI services. According to this model, the cloud provider is responsible for protecting the global infrastructure that supports the cloud services. You, as the customer, are responsible for managing and securing your content hosted on this infrastructure. This includes configuring and managing the security of the cloud services you use. For more details on data privacy, refer to the cloud provider's data privacy documentation and guidelines.

Popular Finetuning Frameworks

There are several frameworks that are designed to streamline and optimize the finetuning process. Examples include Unsloth, Axolotl, and torchtune. Unsloth is an open framework that helps AI developers improve model performance while minimizing resource usage. It enables quicker iterations and experiments. It also significantly reduces memory requirements, making it possible to finetune large models with less computational resources. The article Fine Tuning AI Models—the Easy Way provides references to hands-on finetuning tutorials using some of these frameworks.

These frameworks allow you to configure checkpoints in the finetuning process. Checkpoints allow you to save the model's state at regular intervals during training. This is great for long training runs, as it prevents the loss of all progress if the process is interrupted. Checkpoints enable you to evaluate the model's performance at different stages of training, helping you understand how the model improves over time. By saving multiple checkpoints, you can compare different versions of the model and select the best performing one, which may not necessarily be the final iteration.

Regular checkpoints can be used to audit the training process, ensuring that the model's evolution follows expected patterns and hasn't been tampered with. If a security issue is detected in a later stage of training, you can roll back to a previous, secure checkpoint. In some cases, each checkpoint can be analyzed for potential security vulnerabilities or biases that may have been introduced during training, allowing for early detection and correction. In a secure AI pipeline, checkpointing can also ensure that you can reproduce the training process, demonstrating that the model was trained in a controlled and secure environment.

AI Cloud Data Protection Recommendations

For data protection, implement the following security best practices:

Protect Your Cloud Environment

• Safeguard your cloud account credentials.
• Configure individual user access using the cloud provider’s identity management solutions. Assign permissions based on the principle of least privilege.
• Enable multi-factor authentication (MFA) everywhere!
• Log API and user activity using the cloud provider’s logging services.

Protect Your Data

• Use TLS for secure communication with cloud resources. The provider may require TLS 1.2 and recommend TLS 1.3.
• Use the cloud provider’s strongest encryption solutions along with appropriate security controls to protect your data. If your environment requires validated cryptographic modules, use the relevant compliance endpoints provided by the cloud service.
• Leverage the cloud provider's managed security services to discover and protect sensitive data.

Be Smart!

• Don't put confidential or sensitive information—such as customer details—into tags or free-form text fields that may be used for billing or diagnostic purposes.
• Don't put credentials or any other sensitive information in URLs (even using encoding methods).
• Follow the specific security practices and recommendations provided by the cloud AI service.
• Know how the service handles your data.
  • Amazon’s Bedrock Security recommendations
  • Amazon SageMaker Access Control configuration guidelines
  • Google Vertex AI Security Controls for Generative AI

In general, cloud providers manage deployment environments and logs in a way that prevents any tenants from accessing your data.

Download a PDF of the recomendations checklist.

Securing AI/ML Ops

AI Architecture and Threat Models

Creating threat models involves identifying potential security threats to the AI training environment and developing a plan of action to mitigate these risks. First, determine the high-risk assets within the AI training environment, including data, models, and infrastructure. Understanding what needs protection helps in prioritizing security measures. Then, identify potential threats that could compromise the security of the AI training environment. Common threats include:

• Unauthorized access to or exfiltration of sensitive data
• Misuse of access by authorized users to manipulate or steal information
• Alteration of training data or model inputs to skew results
• Unauthorized changes to model parameters or algorithms
• Interception or alteration of data in transit, such as man-in-the-middle attacks
• Attacks aimed at disrupting the availability of training resources

Assess the vulnerabilities within the architecture that could be exploited by the identified threats. This includes evaluating security gaps in data storage, access controls, network configurations, and known vulnerabilities in open source and commercial third-party software.

Evaluate the potential impact and likelihood of exploitation for each vulnerability. Use that analysis to determine the severity of each threat and prioritize response efforts.

Network and Cloud Security Vulnerabilities

AI systems, like all digital systems, rely on network infrastructure for their operation. If a network is compromised, attackers can intercept data in transit, alter data, or manipulate the AI system's operations. This can result in inaccurate and biased model inference, manipulated outcomes, or even complete system failure.

Most network security concerns that affect general digital systems can also impact AI systems because AI models are typically trained and hosted on servers and operate over networks. Common network issues include unprotected communication channels, weak encryption protocols, and lack of proper access control.

By exploiting vulnerabilities in the network, attackers can manipulate the AI system’s operations. This could involve altering model parameters, injecting malicious data, or interfering with the model’s execution. Such manipulations can result in biased or incorrect outputs, which could undermine the trustworthiness of the AI system.

If encryption protocols that are used to protect data in transit are weak or outdated, attackers may be able to decrypt and access sensitive information. Implementing strong, up-to-date encryption standards is essential to safeguard data integrity and confidentiality.

Insiders with privileged access to the AI system may intentionally misuse their permissions to manipulate model parameters or alter training data. Such actions can introduce biases or security issues into the AI system, skewing results and undermining its effectiveness. Insiders with malicious intent might steal sensitive information that is stored within the AI system, such as proprietary algorithms, customer data, or trade secrets, causing significant financial and reputational damage.

NIST Secure Software Development Framework (SSDF) and AI

AI model creators, AI system developers, and other stakeholders can use the NIST Secure Software Development Framework (SSDF) Community Profile for AI Model Development throughout the training and finetuning lifecycle. Implementing SSDF practices helps creators minimize vulnerabilities in their AI models, mitigate the impacts of potential exploitation of unnoticed or unaddressed vulnerabilities, and address root causes to prevent future issues.

AI system developers can use the SSDF standardized terminology to discuss security practices with AI model creators and to integrate AI models into their software. AI system purchasers can also use SSDF terms to clearly express their cybersecurity requirements and needs to AI model creators and developers during the acquisition process.

Bear in mind that the NIST SSDF Community Profile for AI Model Development is not a prescriptive checklist but a foundational guide for planning and adopting a risk-based approach to secure software development practices for AI models. The guidelines in the Profile should be adapted and customized because not all practices and tasks are relevant to every use case.

Organizations should adopt a risk-based approach to determine which practices and tasks are suitable and effective in mitigating threats to their software development. Considerations such as risk, cost, feasibility, and applicability are crucial when deciding which practices and tasks to implement and how much time and resources to allocate.

Cost models may need to be updated to account for the specific costs of AI model development. As an organization adapts to new or increasing capabilities and risks related to AI models or systems, its risk-based approach to secure software development may evolve.

The NIST SSDF Community Profile for AI Model Development’s practices, tasks, recommendations, and considerations can be integrated into AI/ML Ops along with other software assets within a CI/CD pipeline. Implementing SSDF practices might involve multiple organizations. For instance, an AI model could be created by one organization, deployed in an AI system by another, and used by other entities. In such scenarios, there is likely a shared responsibility model among the model creator, the system developer, and the system purchaser. An AI system purchaser can establish agreements with AI system developers and/or AI model creators to specify responsibilities and how each party will confirm compliance with the agreement.

Note: A limitation of the SSDF and this Profile is that they address cybersecurity risk management only. Organizations should also manage other types of risks to AI systems as part of a comprehensive enterprise risk management program.

Application of SSDF to AI Training Environment Security

The NIST SSDF applies to AI training environment security by providing a structured framework and set of practices to address vulnerabilities and ensure robust security measures are in place.

The elements in the NIST SSDF are categorized into four groups, each with specific practices and tasks aimed at ensuring the security of software development:

• Prepare the Organization (PO) focuses on laying the groundwork for secure software development by establishing security requirements, roles, responsibilities, and processes.
• Protect the Software (PS) involves implementing measures to prevent unauthorized access and tampering with software and its components.
• Produce Well-Secured Software (PW) focuses on the processes and practices involved in creating secure software.
• Respond to Vulnerabilities (RV) focuses on identifying, addressing, and mitigating vulnerabilities in software throughout its lifecycle.

NIST Special Publication 800-218A explains how SSDF practices specifically apply to training environment security:

• Risk-Based Security Requirements (PO.1): Identify and document all security requirements for AI model development, including those specific to training environments. This ensures that all potential security needs are considered and addressed.
• Roles and Responsibilities (PO.2): Define clear roles and responsibilities related to AI model training. Provide role-based training to ensure that personnel understand how to secure training environments and handle AI models securely.
• Toolchain Integration (PO.3): Develop and implement automated toolchains to secure AI model training. Use tools to generate artifacts that document security practices, helping to ensure that training data integrity and provenance are maintained.
• Environment Separation and Protection (PO.5): Separate and protect AI training environments using sandboxing or containers to minimize the risk of contamination or unauthorized access. Monitor these environments continuously to detect and respond to security incidents.
• Data and Code Protection (PS.1): Protect training, testing, finetuning, and aligning data from unauthorized access and modification. Securely store this data and continuously monitor its integrity and confidentiality.
• Vulnerability Management (RV): Regularly scan and test AI models for vulnerabilities during the training phase. Implement processes for detecting and responding to security issues quickly to minimize the risk of exploitation.
• Continuous Monitoring (PO.5.3): Implement continuous monitoring of training environments to detect suspicious activities. Use alerts to trigger investigations when anomalies are detected, ensuring quick response to potential threats.

Figure 2 illustrates these concepts.

Figure 2: SSDF Practices and Training Environment Security

NIST Dioptra

NIST released a tool called Dioptra that can be used as a testing platform to evaluate the trustworthiness of AI models. Trustworthy AI encompasses attributes such as validity and reliability, safety, security, resilience, accountability, transparency, explainability, interpretability, privacy enhancement, and fairness, with a focus on managing harmful biases. Dioptra aligns with the Measure function of the NIST AI Risk Management Framework by offering tools to assess, analyze, and monitor AI risks.

Dioptra can be used for the following types of AI model testing:

• Internal (first party): Evaluate AI models throughout the development lifecycle.
• External (second party): Assess AI models during acquisition or in an evaluation lab setting.
• Auditors (third party): Conduct audits and compliance assessments of AI models.
• Research: Support researchers who are focused on trustworthy AI by assisting in tracking and analyzing experiments.

Dioptra also can be used for controlled AI red team engagements to test models and resources for vulnerabilities. NIST offers guidance on getting started with Dioptra.

Note: NIST AI Test, Evaluation, Validation AND Verification (TEVV) provides guidelines for assessing AI/ML systems. These guidelines focus on evaluating the accuracy, reliability, robustness, and security of AI systems while also highlighting the importance of fairness, transparency, and accountability.

Identity and Access Management in AI Training Environments

Managing who has access to AI training environments and what they can do is essential for protecting sensitive data, ensuring compliance, and maintaining the integrity of the AI models. The following are a few best practices for identity and access management (IAM) in AI training environments.

Core Principles of IAM in AI Training Environments

Authentication and Authorization

Authentication is the process of verifying the identity of a user or system that is attempting to access the AI environment. Passwords, biometrics, and cryptographic keys are examples of authentication methods. In advanced environments, multi-factor authentication (MFA) is often used to enhance security.

Authorization follows authentication and involves determining which resources an authenticated user is allowed to access and which actions they can perform. This is typically managed through access control policies, which are defined based on roles, groups, or individual user needs.

Role-Based Access Control (RBAC)

With RBAC, access permissions are assigned based on the roles of users within the organization. In an AI training environment, roles may include data scientists, machine learning engineers, IT administrators, and auditors, each with different levels of access and permissions.

For instance, data scientists may need access to large datasets and training models but do not require administrative access to the underlying infrastructure. Conversely, IT administrators need access to system configurations but may not require access to the datasets or AI models. This separation of duties is crucial to minimizing the risk of unauthorized actions and potential breaches.

Attribute-Based Access Control (ABAC)

ABAC extends the capabilities of RBAC by considering additional attributes such as user location, time of access, and the sensitivity of the data being accessed. In AI environments, ABAC can be particularly useful for enforcing fine-grained access controls. For example, access to certain datasets may be restricted based on the geographic location of the user, ensuring compliance with data residency laws.

Implementing ABAC requires a robust policy engine that can evaluate multiple attributes in real time to make access decisions. This approach provides greater flexibility and security in environments where access control needs are dynamic and context-dependent.

Examples of Access Control Configurations

The following links include examples of access control configurations in different platforms:

• Vertex AI access control with IAM
• Amazon AWS SageMaker Identity and Access Management
• Role-based access control in Azure AI Studio

Zero Trust in AI Training Environments

Applying Zero Trust principles to secure AI training environments involves implementing a security model that assumes no implicit trust within the network and enforces strict access controls and verification for every user and device interacting with the system. You can divide the AI training environment into smaller, isolated segments to limit the movement of potential threats. Each segment should have its own security policies, ensuring that access to sensitive data and resources is restricted based on the user’s role and needs.

Users, applications, and devices should have only the minimum level of access required to perform their functions. For AI training, this means restricting access to training data, model parameters, and computational resources based on the specific needs of each user or process.

Implement real-time monitoring and analytics to continuously assess the behavior and access patterns of users and devices. Any deviations from normal behavior should trigger alerts and require additional verification before granting access or executing actions. As explained earlier, use strong authentication and authorization mechanisms, including MFA, to verify the identity of users and devices.

Data Access and Security in AI Training Environments

Data is the lifeblood of AI training environments, and securing access to it is paramount. The following practices help ensure that data is protected.

Data Encryption

All sensitive data should be encrypted. For data at rest, use strong encryption standards such as AES-256. For data in transit, use secure protocols like TLS.

Encryption ensures that even if unauthorized access occurs, the data remains unintelligible without the correct decryption keys. Key management is a critical aspect of this process, requiring secure storage and rotation of encryption keys.

Encryption should be integrated with IAM policies to ensure that only authorized users can access or decrypt the data. For example, with attribute-based encryption, the keys required for decryption can be accessed only if the user has the required attributes and conditions.

Data Segmentation and Isolation

Data segmentation involves dividing datasets into smaller, isolated segments based on sensitivity or usage requirements. This approach limits the exposure of sensitive data by ensuring that only specific users or systems have access to the segments they need. In AI environments, segmentation can be implemented by organizing datasets into separate storage buckets or databases, each with distinct access controls.

Isolation also plays a role in securing data, particularly in environments where multiple AI models are trained simultaneously. Isolating training environments prevents one model's data or computations from inadvertently affecting another, maintaining the integrity and confidentiality of each model's data.

Identity Federation and Single Sign-On (SSO)

In modern AI environments, especially those that span multiple cloud platforms or on-premises and cloud hybrid setups, identity federation and SSO are critical for managing access.

Identity Federation

Identity federation allows for the integration of external identity providers (IdPs) such as Microsoft Active Directory, Google Workspace, or custom SAML-based IdPs. This enables users to access the AI environment using their existing credentials, reducing the need for multiple sets of credentials and simplifying identity management.

Federated identities are particularly useful in multi-cloud or hybrid environments, where users may need to access resources across different platforms. By federating identities, organizations can centralize authentication and ensure consistent access control policies across all environments.

Single Sign-On (SSO)

SSO streamlines the user experience by allowing users to authenticate once and gain access to multiple resources without needing to re-authenticate. This is especially beneficial in AI environments where users often interact with a variety of tools, platforms, and datasets.

Implementing SSO requires a robust identity management system that supports cross-platform authentication and integrates with various cloud providers. SSO also enhances security by reducing the attack surface associated with multiple login credentials and simplifies the enforcement of access policies.

Auditing and Monitoring

Continuous auditing and monitoring are essential for detecting and responding to unauthorized access attempts or unusual activity in AI environments.

Access Logs and Audit Trails

Detailed access logs should be maintained for all actions performed within the AI environment. These logs should capture information such as the identity of the user, the resources accessed, the actions performed, and the time of access. Audit trails provide a comprehensive record of all activities, which is crucial for forensic analysis in the event of a security incident.

Logs should be secured to prevent tampering and regularly reviewed to identify potential security threats. Automated tools can be used to analyze logs in real-time, flagging suspicious activity for further investigation.

Real-Time Monitoring and Alerts

Implementing real-time monitoring systems allows for the immediate detection of abnormal behavior within the AI environment. These systems can be configured to trigger alerts when certain thresholds are exceeded, such as repeated failed login attempts or attempts to access restricted data.

Integration with Security Information and Event Management (SIEM) systems can further enhance monitoring by correlating data from multiple sources to detect complex threats.

By employing a combination of role-based and attribute-based access controls, data encryption, identity federation, SSO, and continuous monitoring, organizations can build a robust security framework that protects sensitive AI data and resources from unauthorized access.

Supply Chain Security

Supply chain security in all software and technology solutions (including AI/ML) is a significant concern, affecting the integrity of the training data and ML models as well as the platforms on which they are deployed. Weaknesses in supply chain security can result in biased results, security breaches, or even total system failures.

While traditional security concerns have primarily focused on software components, ML introduces additional vulnerabilities. This is particularly true in the context of pretrained models and training data obtained from external sources, which are at risk of tampering and poisoning attacks.

Moreover, extensions to LLMs, such as plugin features, introduce their own sets of vulnerabilities. OWASP has provided documentation about AI supply chain security, insecure plugin design, model theft, and other threats in the OWASP Top 10 for LLM Applications.

Possible sources of security threats in the AI/ML supply chain include the following:

• Traditional third-party packages, especially those involving outdated or no-longer-supported components, may include unpatched vulnerabilities.
• Pre-trained models that are used as a basis for finetuning may be compromised.
• Crowd-sourced data that is used for training purposes may have been tampered with.
• Obsolete or unsupported models, which lack ongoing maintenance, may introduce security issues. There is no end-of-life or end-of-support standard for AI models, although OpenEoX is now in its infancy and may support AI models in the future.
• Unclear terms and conditions or data privacy policies set by model operators could result in an application's sensitive data being misused. Sensitive information may be used inadvertently in model training and subsequently exposed. Additionally, there is a risk of legal complications arising from the use of copyrighted material by the model supplier.

These concerns highlight the need for comprehensive security strategies that include all aspects of the AI/ML ecosystem, from data sourcing to model deployment and plugin integration.

Transparency and Traceability

Much like a traditional bill of materials in manufacturing that lists all the parts and components of a product, an AI bill of materials (BOM) provides a detailed inventory of all components of an AI system.

AI BOMs are different from traditional software bill of materials (SBOMs) which document the components of a software application. AI BOMs document the components of an AI system, including the model details, architecture, usage, training data, and more.

Note: Ezi Ozoani, Marissa Gerchick, and Margaret Mitchell introduced the concept of AI model cards in a blog post in 2022. Since then, AI BOMs continue to evolve. The concept of AI BOMs is defined in the paper, Toward Trustworthy AI: An Analysis of Artificial Intelligence (AI) Bill of Materials (AI BOMs).

AI BOMs have been supported in SPDX since version 3.0. Understanding the SPDX 3.0 AI BOM Support details the properties within the SPDX 3.0 AI Profile. AI BOMs have been supported in CycloneDX since version 1.6.

AI BOMs ensure that every element used in an AI solution is documented. This transparency encourages trust among users, developers, and stakeholders. With a detailed BOM, developers and auditors can assess the quality, reliability, and security of an AI system. They are also good for troubleshooting. In cases of system failures or biases, AI BOMs can facilitate the quick identification of the problematic component.

The following are some of the components of an AI BOM:

• Model Details: The model’s name, version, type, creator, and more.
• Model Architecture: Details about the model’s training data, design, input and output types, base model, and more.
• Model Usage: The model’s intended usage, prohibited uses, and potential misuse.
• Model Considerations: The model’s environmental and ethical implications.
• Model Authenticity or Attestations: A digital endorsement by the model’s creator to vouch for the AI BOM’s authenticity.

Note: The Coalition for Secure AI (CoSAI) is actively addressing the critical issue of software supply chain security for AI systems by fostering collaboration among industry leaders and academic experts. CoSAI is dedicated to enhancing the security of AI supply chains by developing robust methodologies, sharing best practices, and building technical solutions that ensure the integrity and trustworthiness of AI technologies. Cisco is one of the founding members alongside Anthropic, OpenAI, Amazon, Cohere, GenLab, Google, IBM, Intel, Microsoft, Nvidia, Wiz, Chainguard, and PayPal. Several other organizations have joined since the coalition's founding.

Appendix

Additional Resources

Coalition for Secure AI (CoSAI): https://www.coalitionforsecureai.org

MITRE Adversarial Threat Landscape for Artificial-Intelligence Systems: https://atlas.mitre.org

OWASP Top 10 for LLMs: https://genai.owasp.org

OpenID Connect & OAuth 2.0: https://oauth.net/

SAML 2.0 Technical Overview: http://docs.oasis-open.org/security/saml/Post2.0/sstc-saml-tech-overview-2.0.html

AI Security Reference Architectures: https://www.robustintelligence.com/ai-security-reference-architectures

AI Security Risks: https://www.robustintelligence.com/ai-security-taxonomy

From Cisco:

• AI-related blogs: https://blogs.cisco.com/tag/artificial-intelligence
• Security Portal: https://cisco.com/security
• AI Security Portal: https://aisecurity.cisco.com
• Trust Center: https://trust.cisco.com
• AI and Emerging Technologies Research: https://research.cisco.com
• Zero Trust Architecture Guide: https://www.cisco.com/c/en/us/solutions/collateral/enterprise/design-zone-security/zt-ag.html

From NIST:

• AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework/ai-risk-management-framework-resources
• AI Risk Management Framework Playbook: https://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook
• Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations: https://csrc.nist.gov/pubs/sp/800/161/r1/final
• Privacy Framework: https://www.nist.gov/privacy-framework
• IR 8286: Integrating Cybersecurity and Enterprise Risk Management (ERM): https://csrc.nist.gov/pubs/ir/8286/final
• Towards a Standard for Identifying and Managing Bias in Artificial Intelligence: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf
• Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations: https://csrc.nist.gov/pubs/ai/100/2/e2023/final
• Security and Privacy Controls for Information Systems and Organizations: https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final
• Zero Trust Architecture: https://csrc.nist.gov/pubs/sp/800/207/final

Acknowledgements

Authors:

• Omar Santos, Distinguished Engineer, Cisco Security & Trust
• Mustafa Kadioglu, Data Scientist, Cisco IT
• Sushma Mahadevaswamy, Security Engineer, Cisco Security & Trust Organization
• Akram Sheriff, Senior AI Lead Scientist, Outshift Gen AI Engineering Team
• Dr. Gaowen Liu, Research Engineering Technical leader, Efficient AI Research, Cisco

Editor: Diane Morris, Content Manager, Cisco Security & Trust

Revision History