State-Aware Fallback Imitation

Teaching Vision-Language-Action
Models to Stop Safely

Zheyuan Liu1Chuhan Wang1Jinduo Guo1Yue Huang1Xiangchi Yuan2Ningshan Ma3Meng Jiang1

1 University of Notre Dame2 Georgia Tech3 MIT

Behavior after takeover

Same starting state. Different outcomes.

Prohibited task Place the wine bottle on the rack

Mid-task takeover

Original

Places the bottle

NPO-RT

Drops the bottle

SAFI

Ours
Keeps the bottle held
Takeover 0.25× speed

The problem

Stopping a task takes more than non-completion.

A policy can suppress a prohibited task from its initial state yet continue it when execution is already underway. Even when the task remains unfinished, the robot may drop an object it is holding.

SAFI addresses both challenges by teaching an explicit fallback across demonstration states. For tasks where stationary holding is appropriate, the policy learns to stop the prohibited behavior and maintain a holding response, alongside continued imitation of its permitted skills.

The physical response matters even when the task is unfinished.

The approach

Learn how to stop. Preserve what to keep.

SAFI first constructs fallback targets, then learns them together with the original actions of retained tasks. This gives the policy supervision for both prohibited and permitted behavior.

Fallback target construction

Teach a response across execution states.

We pair prohibited-task demonstrations with stationary arm targets and consistent gripper commands, preserving their observations and instructions. The resulting examples teach holding at intermediate as well as initial states.

Joint policy adaptation

Keep the robot’s permitted skills.

The policy learns the fallback targets alongside the original actions on retained tasks. Joint fallback and retained-task imitation balances stopping the designated behavior with preserving the other skills.

Experimental results

Suppress the target. Control the response.

We evaluate OpenVLA-OFT policies on selected tasks from LIBERO and a separate BadLIBERO backdoor setting. On the evaluated LIBERO tasks, SAFI lowers target success after takeover to 1.3%, with no recorded physical incidents or action saturation.

All rates are percentages. Lower is better for these three metrics.

Takeover outcomes on the evaluated LIBERO tasks and BadLIBERO.
MethodLIBERO (Avg.)BadLIBERO
Target success
rate (%)
Physical incident
rate (%)
Action saturation
rate (%)
Target success
rate (%)
Physical incident
rate (%)
Action saturation
rate (%)
Original policy99.712.10.087.537.50.0
GA12.216.3100.00.040.0100.0
GradDiff11.727.599.40.030.0100.0
NPO-RT31.919.946.90.022.5100.0
RMU50.614.40.00.00.00.0
Refusal FT11.64.10.00.00.02.5
VLA-Forget (OFT)30.920.353.635.035.0100.0
SAFI (ours)1.30.00.00.00.00.0

Swipe the table to see all metrics.

Metric definitions and comparison scope

LIBERO averages the evaluated Goal, Spatial and Object settings equally. The BadLIBERO reference is the implanted policy. Bold marks the lowest rates among adapted policies.

Target Success Rate measures whether the prohibited target is attained after takeover. In BadLIBERO, the target is the attack objective. Physical Incident Rate counts defined events such as object drops, including incidents that accompany target attainment. Action Saturation Rate measures episodes meeting the setting’s criterion for commands at the controller’s clipping bounds.

The evaluated LIBERO tasks are ones the original policy can execute and for which scripted holding passes the applicability check. Baselines use the adaptations described in the study. The VLA-Forget results describe our deployed OFT adaptation, whose action outputs differ after checkpoint reloading.

Evaluation protocol · Result sources and interpretation

Stopping prohibited tasks while retaining other skills.

The horizontal axis shows target suppression after takeover, and the vertical axis shows success on permitted tasks. SAFI’s later checkpoints occupy the upper-right region, combining high suppression with high retained success.

Explore SAFI

Train, evaluate, and build on the method.

The public repository includes step-by-step training and evaluation workflows for LIBERO and BadLIBERO.

Get the code