忘却を教える：Amazon Novaによる選択的アンラーニング

Organizations deploying foundation models (FMs) often encounter a common challenge: model safeguards designed for content moderation can also prevent legitimate, business-critical use cases. A media company summarizing scripts with mature language, a cyber security firm simulating real-world threats, or a legal team processing sensitive evidence may all find that default content moderation controls deflect the very content they need to work with. For example, a security team asking the model to generate a sample phishing email for employee awareness training may receive a refusal, even though the intent is defensive.

Because the model learns these safeguards during post-training alignment, prompt engineering alone cannot overcome them. The model’s tendency to deflect is embedded in its parameters, requiring a targeted modification at the model level to selectively adjust this behavior. In this post, we introduce *Reverse Direct Preference Optimization (rDPO)*, the novel unlearning technique behind Amazon Nova Customizable Content Moderation Settings (CCMS), and show how it reduces over-deflection while preserving model quality. We also provide pointers for customers who want to apply these preference optimization techniques to their own experiments.

Amazon Nova Customizable Content Moderation Settings (CCMS) addresses this by letting approved customers selectively adjust safeguards across four responsible AI (RAI) pillars. These pillars encompass:

• Safety – Covering dangerous activities, weapons, and controlled substances.

• Sensitive content – Including profanity, nudity, and bullying.

• Fairness – Considerations around bias and culture.

• Security – Concerns involving malware and malicious content.

Amazon Nova enforces essential, non-configurable controls for responsible use of AI, such as controls to prevent harm to children and preserve privacy.

The science behind CCMS is *unlearning*, a technique for selectively removing learned behaviors from a model’s parameters without retraining from scratch. We train Low-Rank Adaptation (LoRA) adapters to reverse the model’s alignment to specific policies. The result is a custom model variant that generates content in customer-approved policy areas while remaining aligned everywhere else.

Solution

LoRA adapter training

CCMS works by providing customers with a LoRA adapter that has been trained to unlearn specific RAI policies from the core model. When a customer imports this adapter, they receive a custom model identified by a unique Amazon Resource Name (ARN). At inference time, the adapter steers the core model away from deflecting content in the customer’s approved policy areas. Additionally, Nova’s output moderation guardrails are configured for the customer’s approved policies when using that custom model ARN.

rDPO: A novel approach for unlearning

The key scientific challenge is performing this unlearning effectively: the model must stop deflecting content in targeted policy areas while preserving its general capabilities (instruction following, coding, math) and maintaining alignment in non-targeted areas. A direct fine-tuning approach risks degrading overall model quality.

One state-of-the-art approach is Negative Preference Optimization (NPO), which builds upon Direct Preference Optimization (DPO). DPO is a preference optimization approach that trains the model to rank the preferred response higher than a dispreferred one, relative to a reference model. It moves the model toward the preferred response and away from the dispreferred one. NPO achieves unlearning by removing positive samples from the optimization objective, effectively training the model to move away from its learned deflection behavior. However, NPO only teaches the model to forget without guiding it toward high-quality alternative responses, which can result in degraded output quality.

To address this, we developed rDPO, which reverses the preference pair in the DPO objective. Rather than simply making the model forget, rDPO simultaneously guides it toward generating high-quality responses in the unlearned policy areas. This dual objective not only produces better response quality but also improves training efficiency, requiring fewer optimization steps to converge.

The following shows the comparison of DPO, NPO, and rDPO loss functions:

In DPO loss, *yw*, *yl* are the preferred and dispreferred response, respectively, where *w* and *l* mean win and lose responses, respectively. The loss function optimizes the model to rank preferred response higher than the dispreferred one. In NPO loss, *yf* is the response to forget, the NPO loss trains the model to move away from *yf*. In rDPO, *yt* is the target response and *yf* is the forgetting response. The rDPO loss function teaches the model to move away from the forgetting response while moving close to the target response. We call our approach rDPO as the training objectives are the reverse of the DPO applied in the original model’s post-training stage.

The following figures show the training dynamics of NPO and rDPO and their performance on the evaluation dataset.

From the training dynamics, we can see that the training accuracy of rDPO converges to almost 1 at around step 30, and the training accuracy of NPO does not change too much. The training rewards of the chosen response (aka target response) of rDPO continues growing while that of the NPO continues dropping. This implies that rDPO moves the behavior of the model to the target response, while it’s hard for NPO to move the model away from the forgetting response and to the target response. This is likely because the base model has strong RAI alignment, making it difficult to move away from safe responses.

We can see similar trends on NPO and rDPO’s performance on the evaluation dataset. The training dynamics plots show that rDPO is much more efficient than NPO in terms of unlearning performance.

Customization pipeline

The following figure shows the model customization pipeline.

It starts with data preparation, where a set of prompts is curated to represent the target policy areas where we want to unlearn alignment behaviors. Given those prompts, we generate the forgetting and target responses. We train a model with rDPO and the generated data. Once the training loop finishes, the LoRA adapter is exported and ready for serving. We choose LoRA fine-tuning because of its training efficiency and lower inference cost compared to full-rank fine-tuning.

Results

We evaluate the RAI customized model on two dimensions: (1) reduction in over-deflection on sensitive requests, and (2) preservation of general model capabilities.

Deflection rate measures the percentage of prompts that the model refuses to answer. For the baseline model, a high deflection rate on sensitive content indicates that legitimate requests are being blocked. A lower deflection rate after customization means the model is now able to process those requests successfully. The following table shows deflection rates across five evaluation categories:

Model

Red Team Prompts

Fairness

Safety

Security

Sensitive Content

Baseline

98.10%

51.84%

86.51%

91.61%

79.02%

RAI customized model

47%

23.83%

32.77%

45.73%

33.58%

The results show substantial reductions in deflection rate across all policy categories. Safety deflections dropped from 86.51 percent to 32.77 percent (a 53.74 percentage point reduction). The customized model now processes the majority of safety-related requests that the baseline would have refused. Similar improvements are seen across Fairness (28 pp reduction), Security (46 pp), and Sensitive Content (45 pp). Importantly, these reductions are achieved through the rDPO-trained LoRA adapter without any modification to the base model weights.

A critical requirement for any unlearning approach is that it must not degrade the model’s general capabilities. To verify this, we evaluate the customized model on three utility benchmarks: instruction following, mathematical reasoning (Math Mini), and code generation (MBXP Python):

Model

Instruction Following

Math Mini

MBXP Python

Baseline

94.12%

86.40%

74.80%

RAI customized model

92.57%

85.20%

73%

The customized model retains near-baseline performance across all utility benchmarks, with drops of only 1.55 percentage points on instruction following, 1.20 pp on math, and 1.80 pp on code generation. These minimal degradations confirm that the rDPO approach successfully targets only the RAI alignment parameters while leaving the model’s core capabilities intact, a key advantage of the LoRA-based approach over full model fine-tuning.

Getting started

The preference optimization techniques described in this post, including DPO, NPO, and rDPO, are accessible to customers who want to explore unlearning and alignment research on their own. Amazon SageMaker AI supports DPO training with both full-rank and LoRA approaches for Amazon Nova and over 20 open-weight models, and Amazon SageMaker HyperPod provides ready-made DPO recipes that can serve as a starting point for custom unlearning experiments. See Customize Amazon Nova in Amazon SageMaker AI using Direct Preference Optimization for a step-by-step walkthrough.

For customers who want the benefits of unlearning without building it themselves, CCMS provides pre-trained LoRA adapters for Amazon Nova ready to deploy. Adapters are shared through AWS Resource Access Manager (AWS RAM). After you accept and copy an adapter into your account, it appears as a custom model in Amazon Bedrock with a unique ARN. You then create an on-demand inference custom model deployment and use the custom model deployment ARN in the standard Converse API. No other code changes are required:

aws bedrock-runtime converse \
  --model-id "arn:aws:bedrock:us-east-1::custom-model-deployment/" \
  --region us-east-1 \
  --messages '[{"role": "user", "content": [{"text": "Your prompt here"}]}]'

Note: Deploying a custom model incurs standard Amazon Bedrock inference charges. See Amazon Bedrock pricing for details.

The customized model behaves identically to the base Nova model for non-configured policy areas. For additional application-level safeguards, you can pair CCMS with Amazon Bedrock Guardrails to implement topic-specific filtering, hallucination detection, or custom content policies on top of the configured model.

To request access and discuss which policy pillars to configure for your use case, reach out to your AWS account team or create a support case from the AWS Support Center Console.

Conclusion

In this post, we introduced Reverse DPO (rDPO), a preference-based unlearning technique. It addresses a key limitation of existing approaches like NPO: the inability to simultaneously guide the model toward high-quality responses while unlearning targeted behaviors. By reversing the preference pair in the DPO objective, rDPO achieves both goals in a single training signal, converging faster and producing better output quality than NPO. Requiring fewer training iterations, rDPO also reduces the time and cost of producing customized adapters for customers.

Our evaluation on Amazon Nova 2 Lite demonstrates that rDPO-trained LoRA adapters reduce deflection rates by up to 54 percentage points across RAI policy categories while degrading utility benchmarks by less than 2 percentage points. Importantly, these customizations operate within the bounds of Amazon Nova universal, non-configurable protections, ensuring that core safeguards remain intact regardless of adapter configuration. The LoRA-based approach is particularly well-suited to this problem: it isolates unlearning to a small set of adapter parameters, preserves the base model’s integrity, and enables modular deployment where different policy configurations can be served from the same foundation model.

These techniques power Amazon Nova CCMS, which provides customers with pre-trained adapters ready to deploy using Amazon Bedrock. For researchers and practitioners interested in further exploring preference-based unlearning, Amazon SageMaker AI offers DPO training recipes that can serve as a foundation for custom experimentation. For further instructions on customizing Nova in Amazon SageMaker AI, see our documentation and blogs on Nova customization and advanced fine-tuning methods on Amazon SageMaker AI.

Acknowledgement

The contributors to this project include Ekraam Sabir, Weitong Ruan, Payal Motwani, Rahul Gupta, Claire O’Brien Rajkumar, Dhwanil Desai, and Nikhil Sanil.

About the authors

Qian Hu

Qian is a Senior Applied Scientist in the Amazon AGI Foundations RAI team, where he focuses on Responsible AI including guardrail models, RAI customization, post-training alignment.

Veda Raman

Veda is a Sr Solutions Architect for Generative AI for Amazon Nova and Agentic AI at AWS. She helps customers design and build Agentic AI solutions using Amazon Nova models and Amazon Bedrock AgentCore. She previously worked with customers building ML solutions using Amazon SageMaker and also as a serverless solutions architect at AWS.

Dan Sinnreich

Dan is a Sr. Product Manager at Amazon, where he builds AI safety controls for Amazon Nova models. Previously, he built no-code/low-code machine learning capabilities for Amazon SageMaker, making ML accessible to a broader range of practitioners. Outside of work, he can be found playing hockey and reading science fiction.