A robotic fingertip spinning a pen at high speed, with blue motion Trials

Reliable and Diverse Robotic Pen Spinning via Guided Model-to-Policy Refinement

Robotic pen spinning

Main Real World Results (featured by high-speed, fine-grained movements, sophisiticated gaiting, and realiability)

Diverse Pen Spinning Skills

Thumb-Around

01 Reliable one-episode success (49 in 50)

98% (49 in 50) success rate

04 Diverse Thumb-Around Spinning

Diverse Thumb-Around Spinning Clips

Sonic (down)

01 Reliable one-peisode successes (50 in 50)

100% (50 in 50) success rate

04 Sonic Spinning

Sonic Spinning

Pass-V-R

01 Reliable one-episode success

100% (50 in 50) success rate

02 Sustained rotation (averaged 4.025 episodes across 20 evaluations)

Avg. 4.025 episodes, Total 80.5 episodes, 20 Evaluation Trials

01 Sustained rotation (all in the original 1x speed)

Four examples

01 Sustained rotation (all in the original 1x speed)

Four examples

Pass-V-NR

01 Reliable one-episode success (50 in 50)

100% (50 in 50) success rate

02 Sustained rotation (4 rotation episodes)

4 rotation episodes

Pass-H-NR

01 Reliable one-episode success (45 in 50)

90% (45 in 50) success rate

02 Sustained rotation (2 rotation episodes)

2 rotation episodes

Pass-H-R

01 Reliable one-episode success (48 in 50)

96% (48 in 50) success rate

02 Sustained rotation (3 rotation episodes)

3 rotation episodes

Triangle Pass (palm-up)

01 Reliable one-episode successes (50 in 50)

100% (50 in 50) success rate

02 Sustained Rotation (avg. 39.9 episodes across 5 Trials)

Avg. 39.9 episodes

03 Reliable one-episode successes (100 in 100)

100% (100 in 100) success rate

04 Sustained Rotation (25+ mins, 91 episodes x 2pi)

25+ mins, 91 episodes x 2pi

Triangle Pass (palm-down)

01 Reliable one-episode successes (50 in 50)

100% (50 in 50) success rate

02 Sustained Rotation (avg. 40.1 episodes across 5 Trials)

Avg. 40.1 episodes (20 episodes (4.5mins) - 43 episodes (~13mins) - 27.5 episodes (8mins 26s) - 77 episodes (23mins 38s) - 32.5 episodes (10mins))

Triangle Pass (palm-down, reversed)

01 Reliable one-episode successes (50 in 50)

100% (50 in 50) success rate

02 Sustained Rotation (avg. 10.7 episodes across 5 Trials)

Avg. 10.7 episodes

Four-Finger-Pass (palm-down)

01 Reliable one-episode successes (100 in 100) and Sustained Rotation (30+ mins, 134 x 2pi in total)

100% (100 in 100) success rate and 30+ mins, 134 x 2pi in total

02 Sustained Rotation (15+ mins)

15+ mins

Four-Finger-Pass (palm-down, LEAP)

02 Reliable one-episode successes (48 in 50)

96% (48 in 50) success rate

04 Sustained Rotation (2 episodes)

2 episodes

01 Reliable one-episode successes (50 in 50)

100% (50 in 50) success rate

03 Sustained Rotation

sustained rotation

Fun Results

Fun Results

01 Fun Result (1) - Pass-H-NR

Pass-H-NR Variant

02 Fun Result (2) - Policy resulting from an intermediate reward searched by GPT

Fun Result 2

Zero-shot Generalization to 1) different object materials, 2) changed dynamics, 3) various object shapes.

Zero-shot Generalization

01 Pass-V-R: Cylinder -> Hexagonal Prism, Cctagonal Prism (continuous rotation & multiple successes)

Changes in surface flatness and contact patch size.

02 Pass-V-R: Rigid -> Compliant Object (continuous rotation & multiple successe)

Increased object compliance and altered contact response.

03 Pass-V-NR: Capsule -> Octagonal prism

Changes in surface flatness and contact patch size.

04 Pass-V-NR: Rigid -> Compliant Object (continuous rotation & multiple successe)

Increased object compliance and altered contact response.

05 Pass-H-NR: Rigid Capsule -> Hexagonal Prism, Cctagonal Prism, and Compliant Object

Changes in surface flatness, contact patch size, and contact response.

06 Pass-H-R: Cylinder -> Cctagonal Prism

Changes in surface flatness and contact patch size.

07 Pass-H-R: Cylinder -> Hexagonal Prism

Changes in surface flatness and contact patch size.

08 Pass-H-R: Rigid -> Compliant Object

Increased object compliance and altered contact response.

09 Triangle Pass (palm-up): T shape, L shape, long board

Shifted center of mass and altered gravitational torques.

10 Triangle Pass (palm-down): T shape, L shape

Shifted center of mass and altered gravitational torques.

11 A low-friction object

Reduced contact friction and increased susceptibility to slipping.

12 Remove the tape wrapped around the fingers (success rate: ~60%)

Low-friction scenario.

13 Triangle Pass (palm-down): Heavy Object

Substantially increased object weight.

14 Four-Finger-Pass (LEAP): T shape, L shape, Elliptical Cylinder, C-Elliptical Cylinder

Shifted center of mass, altered gravitational torques, and varying surface curvature.

Adapation to task variants with 200-500 additional real world rollouts.

Adaption to Task Variants

01 Pass-V-R: Object with a larger diamerter (1.4 cm -> 1.6 cm, from around 50% success rate to 100% success rate)

Adapation to object with a larger diamerter

02 Thumb-Around: Compliant Object (from less than 10% success rate to 96% success rate (48/50))

Increased object compliance and altered contact response.

03 Four-Finger-Pass (LEAP): Elliptical Cylinder (from around 40% to 100% success rate)

Varied surface curvature.

04 Triangle Pass (palm-down): Rectangular Plates (15cm x 2cm x 5cm, 16mins, from around 45% success rate to 100% success rate and sustained rotation; 15cm x 2cm x 10cm; 15cm x 2cm x 7cm )

Changes in object mass, rotational inertia, and contact geometry.

05 Triangle Pass (palm-down): Cuboid -> Cylinder (from around 25% to 100% success rate)

Changes in surface flatness and contact patch size.

06 Triangle Pass (palm-down): Cuboid -> Octagonal Prism (from around 50% to 100% success rate)

Changes in surface flatness and contact patch size.

07 Triangle Pass (palm-up): A thinner ``pen''' (2.0 cm -> 1.5 cm, from less than 10% success rate to ~100% success rate)

Thick pen (2.0 cm) to thin pen (1.5cm)

08 Triangle Pass (palm-up): Cuboid -> Octagonal Prism

Changes in surface flatness and contact patch size.

07 Triangle Pass (palm-up): Cuboid -> Cylinder (from around 30% to 100% success rate)

Changes in surface flatness and contact patch size.

Robustness to Human Disturbances

Robustness to Human Disturbances

01 1) Push the object out of the balanced pose, 2) Drag the object to an extreme initial position

Triangle Pass (palm-up)

02 Consecutive successes with disturbances (Triangle Pass (palm-up))

Triangle Pass (palm-up)

03 1) Push the object, 2) Drag the object out of the balancing pose

Pass-V-R

04 Consecutive successes with disturbances (Pass-V-R)

Pass-V-R

05 Push the object off the balance x 3

Triangle Pass (palm-down)

06 Consecutive successes with disturbances (Triangle Pass (palm-down))

Triangle Pass (palm-down)

Initial position distribution, rollout distribution

Analysis on Cross-Rollout Variations

Triangle Pass (palm up)

01 Consecutive success one-episode rollout (59/59)

59 consecutive successes · real-world footage at 10× speed.

02Rollout distribution

Drag to orbit · scroll to zoom
59 recorded trajectories, overlaid in 3D. Pause, scrub time, or change playback speed.
Open standalone viewer ↗

03Initial position distribution

Rotate · zoom · select a point
One point per rollout, at frame 0. Coordinates in mm relative to one shared centroid.
Full report & XY / XZ / YZ views ↗

Pass-V-R

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-V-R

02Rollout distribution

Drag to orbit · scroll to zoom
93 recorded rollouts with measured and target hand poses overlaid in 3D. Pause, scrub time, or change playback speed.
Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point
93 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.
Full report & XY / XZ / YZ views ↗

Triangle Pass (palm-down, reversed)

01 Successful one episode rollouts (the video is play in 10x speed)

Triangle Pass (palm-down, reversed)

02Rollout distribution

Drag to orbit · scroll to zoom
33 successful rollouts with measured and target hand poses. Unreliable object-tracking intervals are hidden.
Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point
33 frame-zero poses · XYZ / Quaternion. The diamond marks one pose estimated across a brief tracking gap.
Full report & XY / XZ / YZ views ↗

Triangle Pass (palm-down)

01 Successful one episode rollouts (the video is play in 10x speed)

Triangle Pass (palm-down)

02Rollout distribution

Drag to orbit · scroll to zoom
41 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.
Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point
41 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.
Full report & XY / XZ / YZ views ↗

Pass-V-NR

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-V-NR

02Rollout distribution

Drag to orbit · scroll to zoom
50 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.
Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point
50 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.
Full report & XY / XZ / YZ views ↗

Pass-H-R

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-H-R

02Rollout distribution

Drag to orbit · scroll to zoom
59 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.
Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point
59 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.
Full report & XY / XZ / YZ views ↗

Pass-v2

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-H-NR

02Rollout distribution

Drag to orbit · scroll to zoom
52 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.
Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point
52 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.
Full report & XY / XZ / YZ views ↗

Full takeover and partial online takeover

Trials on Human Takeover for Failure Recovery

01 Pass-V-NR, full takeover

full takeover

02 Pass-V-NR, partial online takeover

Partial online takeover

01 Pass-H-NR, full takeover

full takeover

02 Pass-H-NR, partial online takeover

Partial online takeover

01 Triangle pass (palm-up), full takeover

full takeover

02 Triangle pass (palm-up), partial online takeover

Partial online takeover