Evaluating Vision-Language-Action Models in Robotic Manipulation

Performance, Implementation, and Comparison with Deterministic Systems

A Master's Thesis by Jake Kemple
Committee: Min Chen (Chair), Clark Olson, Yang Peng
M.S. in Computer Science & Software Engineering, University of Washington, 2025
Thesis Paper Code Defense Slides UW ResearchWorks

Zero-shot OpenVLA-7B completing the pick-and-place task on the WidowX 250 across several trials.

Robotic manipulation systems typically rely on deterministic policies for perception, decision-making, and task planning. These reach millimeter-level precision but require extensive specialized development and do not generalize easily to new tasks. Vision-Language-Action (VLA) foundation models promise to reduce that effort through learned multimodal reasoning, but their real-world practicality and development cost remain largely unknown.

This thesis presents a real-world comparison of a strong open-source VLA model, OpenVLA-7B, against a tuned deterministic control system. Both run on identical hardware (a WidowX 250 arm, an Intel RealSense D415 camera, and an NVIDIA Jetson AGX Orin under ROS 2) and repeatedly execute a pick-and-place task under randomized conditions. Performance is measured with object-centric metrics of accuracy, repeatability, and cycle time adapted from ISO 9283, alongside a comparison of the setup effort each system required.

Placement Accuracy
3.24 vs 54.4 mm
error to target
Cycle Time
14.1 vs 64.2 s
mean per trial
Repeatability Radius
13.5 vs 95.3 mm
99% inclusion
Task Success
10/10 vs 9/10
completed trials

Deterministic system shown in teal, OpenVLA-7B system in amber.

Contributions

  1. An empirical comparison of performance and setup complexity between a VLA-based control policy and conventional deterministic control logic.
  2. Documentation of the hardware, software, and configuration challenges encountered while implementing the VLA system on real hardware.
  3. Qualitative insights from real-world deployment, with practical guidance for engineers considering VLA-based control.

Vision-Language-Action Models

Progression from neural networks to LLMs to VLMs to VLAs

A VLA model extends the lineage of modern neural networks. Large Language Models added attention over text, Vision-Language Models added image input, and VLA models replace the text output with robot action deltas, mapping a camera image and a language instruction directly to motion.

This research uses OpenVLA-7B, an open-source VLA. It combines a fused DINOv2 and SigLIP visual encoder with a Llama-2 7B language backbone, and decodes seven action tokens per step: position (Δx, Δy, Δz), rotation (Δroll, Δpitch, Δyaw), and a gripper state. It was pretrained on the BridgeData V2 dataset, which includes the WidowX 250 arm used here, making zero-shot deployment feasible without fine-tuning.

OpenVLA model architecture

OpenVLA architecture: a fused visual encoder and a Llama-2 backbone predict tokenized actions that decode into continuous robot commands. Figure from the OpenVLA paper.

System Design

Both systems share one hardware platform and the same low-level Interbotix arm control. They differ only in the high-level policy that decides where the arm moves. The deterministic pipeline reads depth point clouds and runs scripted Cartesian moves; the VLA pipeline passes RGB frames and a prompt to OpenVLA and converts its predicted action deltas into motion.

Overall system architecture across three policy implementations

The full architecture. Three interchangeable control policies drive the same WidowX arm through shared Interbotix control.

Hardware & Software

WidowX 250
6-DoF Interbotix X-Series arm. Included in the BridgeData V2 training set, enabling zero-shot use.
Intel RealSense D415
RGB-D camera. RGB feeds the VLA; depth point clouds feed the deterministic pipeline.
Jetson AGX Orin 64GB
On-board edge inference for the 7B model, with a 1TB NVMe SSD for model weights and caches.
ROS 2 Humble
Publish-subscribe middleware connecting perception, inference, and arm control on JetPack 6.2.
OpenVLA-7B
The vision-language-action policy, run zero-shot from the HuggingFace checkpoint.
Interbotix Stack
Shared SDK for inverse kinematics, trajectory generation, perception, and servo control.
The workspace: WidowX arm, RealSense camera on a stand, and 1mm graph paper

The workspace: arm, camera on an adjustable stand, and the 1 mm graph paper used to measure placement by hand.

Control Policies

Reaching a working VLA system took two attempts, which is itself a finding.

Baseline

Deterministic pipeline

Color segmentation and point-cloud clustering localize the block, AR-tag calibration maps camera to robot, and a scripted state machine grasps and places it.

Initial attempt

Custom VLA nodes

Three custom ROS 2 nodes fed OpenVLA's deltas directly to the Interbotix SDK. The loop ran end to end but produced no meaningful task progress, showing OpenVLA is not plug-and-play.

Used for evaluation

bridge_data_robot

Adapting and ROS 2-patching the official Bridge V2 evaluation stack supplied the missing control layer (action scaling, workspace bounds, fixed initial pose), after which the model performed the task.

A central implementation finding: the model is only one component of the system. OpenVLA's raw action deltas were not usable until a control layer handled action rescaling, per-step clipping (±5 cm and ±0.25 rad), a downward-pitched initial pose, and inference-loop timing. The robotics integration around the model mattered as much as the model itself.
bridge_data_robot VLA pipeline with pre- and post-processing layers

The working VLA pipeline, with pre- and post-processing wrapping the model.

Deterministic control logic flow

The deterministic policy: a short, explicit script (localize, grasp, place).

Experiment Setup and Trials

The task is a standard pick-and-place: lift a 1-inch wooden block and place it on a target dot. Each trial begins with the block at a random position and orientation within a 28 by 20 cm zone. The deterministic system receives fixed target coordinates; the VLA system receives an image and the prompt "Put block onto paper dot."

Each system ran 10 trials across six block colors. Final placement was measured by hand on 1 mm graph paper, giving roughly 1 to 2 mm of measurement uncertainty. Because OpenVLA has no built-in notion of task completion, the VLA loop was capped at 80 steps.

WidowX gripper grasping a block over graph paper

The gripper closing on a block, AR tag used for calibration.

Five-frame RealSense camera view of the VLA picking up and moving a block

A VLA rollout from the RealSense camera, the same camera framing as the BridgeData V2 training set.

Metrics

The systems are scored with object-centric metrics adapted from ISO 9283, which evaluate where the block ends up rather than where the robot's wrist goes:

  • Accuracy: distance from the average final block position to the intended target.
  • Repeatability: spread of final positions across trials, reported as axis standard deviation and a 99% inclusion radius.
  • Cycle time: ROS 2-timestamped seconds from the first command to completed placement.

Results and Analysis

Across every quantitative metric the deterministic system outperformed the VLA system, often by a wide margin. The VLA system completed 9 of 10 tasks but was far less accurate, less repeatable, and several times slower.

Metric Deterministic VLA (OpenVLA-7B)
Task success10 / 109 / 10
Accuracy, error to target (mm)3.2454.37
Repeatability radius, 99% (mm)13.4895.27
Axis repeatability σx, σy (mm)0.42, 4.4716.14, 27.35
Bias vector (mm)(1.2, −3.0)(−45.1, 30.3)
Mean cycle time (s)14.0964.22

Aggregated over 10 trials per system. Positions measured by hand on 1 mm graph paper (1 to 2 mm uncertainty).

Trial Comparisons

The same task under each policy. Deterministic clips play in real time; VLA clips are sped up 5 times.

Deterministic
Green block, real time
OpenVLA-7B
Green block, sped up 5x
Deterministic
Yellow block, real time
OpenVLA-7B
Yellow block, sped up 5x, with a re-grasp after a failed attempt

Representative Trials

Deterministic, success
Frames from a successful deterministic trial

A direct pick-and-place.

VLA, recovery
Frames from a VLA trial with a retry

A failed grasp, then a retry (Trial 7).

VLA, failure
Frames from a failed VLA trial

The arm proceeds as if holding the block and never recovers (Trial 8).

VLA Rollouts (Camera View)

The RGB feed OpenVLA receives, across a sweep of zero-shot trials.

Zero-shot OpenVLA-7B rollouts on the WidowX 250, captured from the RealSense D415.

Qualitative Observations

  • Prompt sensitivity. Wording changed behavior substantially. "Put block onto paper dot" succeeded most often, while "Pick up block and place on center of white paper" produced hesitation, poor grasps, or task misreads.
  • Recovery. The VLA sometimes detected a failed grasp and retried, eventually succeeding. This was inconsistent; in other trials it falsely assumed success and continued until the step limit.
  • System-level fragility. Performance was sensitive to block color, calibration, the arm's starting pose, and inference timing. Several failures stemmed from the full stack, not the model alone.

Setup and Installation Complexity

Performance is one half of the comparison. The other is what each system required to run. The deterministic stack shipped with documentation and demonstration scripts. The VLA stack was a research codebase that had to be adapted to modern edge hardware.

Deterministic setup
  • Organized vendor documentation and ready-made demo scripts.
  • No GPU required; runs on modest compute.
  • Predictable once calibrated.
  • Cost: manual recalibration and rescripting for any change in task or layout.
VLA setup
  • No framework for the inference and action loop; built and tuned from scratch.
  • The 7B model needs substantial GPU memory just for inference.
  • Reference code was ROS 1 with Docker, neither viable on JetPack 6.
  • No notion of task completion, requiring a manual step cap.

Integration challenges

ROS 1 to ROS 2
The bridge_data_robot reference stack assumed ROS 1 conventions; launch files and interfaces were rewritten.
ARM64 and JetPack 6
Interbotix install scripts targeted AMD64 or Raspberry Pi, so the Orin needed a custom install path.
Docker and GPU
GPU passthrough in Docker on JetPack 6 was unworkable, and VMs are unsupported by Interbotix, leaving a bare-metal install.
Inference latency
A single un-quantized OpenVLA forward pass averages about 840 ms on the Orin, far from the 20 to 30 Hz of conventional control.
Calibration sensitivity
Millimeter camera-arm miscalibration cascades into centimeter task error, so small bumps required full recalibration.
Dependency management
Version conflicts across CUDA, PyTorch, Transformers, and ROS required careful version pinning.

Conclusion

Quantitatively, the deterministic system outperformed the VLA system in accuracy, repeatability, and execution speed. Qualitatively, the VLA system showed advantages in adaptability and language-driven flexibility that the deterministic system lacks.

For structured tasks that demand precise, consistent, and fast manipulation, deterministic control remains the better choice, and the current cost of deploying a VLA generally outweighs its benefits. In dynamic or exploratory settings where adaptability and natural-language interaction are valuable, VLA models show clear promise. Reaching production-grade reliability will require further fine-tuning, better integration tooling, and continued model improvements.

Implications to Hypotheses

Not supported

H1: Comparable performance

The VLA system trailed deterministic control on accuracy, repeatability, and speed.

Partially supported

H2: Resilience and recovery

It recovered from some failures and adapted to prompts, but inconsistently, with new failure modes.

Supported

H3: Integration challenges

Latency, calibration sensitivity, GPU requirements, and ROS and hardware incompatibilities were all significant.

Insights and Recommendations

Recommendations

  • Budget for the integration layer, not just the model. Action scaling, workspace bounds, and a fixed initial pose were decisive for success.
  • Treat prompts as configuration and keep a tested set, since small wording changes alter behavior.
  • Tune the inference loop. Image capture, inference, and actuation rates must be coordinated.
  • Plan for trial and error in calibration, timing, and hardware compatibility, and size compute for inference, not only training.

Future Work

  • Harder tasks: clutter, irregular objects, and multi-step or dynamic scenarios.
  • Fine-tuning and newer models, including optimized variants such as OpenVLA-OFT.
  • Modular integration frameworks so VLA policies can be staged into existing robot stacks with less custom code.
  • Better introspection (attention maps, confidence scores, embeddings) to debug system-level failures.

About the Author

Jake Kemple

M.S. in Computer Science & Software Engineering, University of Washington, 2025

Email GitHub LinkedIn

BibTeX

@mastersthesis{kemple2025vla,
    author  = {Jake Kemple},
    title   = {Evaluating Vision-Language-Action Models in Robotic Manipulation:
               Performance, Implementation, and Comparison with Deterministic Systems},
    school  = {University of Washington},
    year    = {2025},
    type    = {Master's thesis},
    url     = {https://digital.lib.washington.edu/researchworks/items/9ea676ab-f431-4c21-bc80-a1ac280b84f4},
}