Mastering “Pen” Spinning in Diverse Styles

Main Real World Results (featured by high-speed, fine-grained movements, sophisiticated gaiting, and realiability)

Diverse “Pen” Spinning Skills

Pass-v0

              
                  01
                  Reliable one-episode success
                
100% (50 in 50) success rate

                  02
                  Sustained rotation (averaged 4.025 episodes across 20 evaluations)
                
Avg. 4.025 episodes, Total 80.5 episodes, #Evaluations 20

              
                  01
                  Sustained rotation (all in the original 1x speed)
                
Four examples

                    01
                    Sustained rotation (all in the original 1x speed)
                  
Four examples

Pass-v1

              
                  01
                  Reliable one-episode success (48 in 50)
                
96% (48 in 50) success rate

                  02
                  Sustained rotation (4 rotation episodes)
                
4 rotation episodes

Pass-v2

              
                  01
                  Reliable one-episode success (45 in 50)
                
90% (45 in 50) success rate

                  02
                  Sustained rotation (2 rotation episodes)
                
2 rotation episodes

Pass-v3

              
                  01
                  Reliable one-episode success (48 in 50)
                
96% (48 in 50) success rate

                  02
                  Sustained rotation (3 rotation episodes)
                
3 rotation episodes

Thumb-Around

              
                  01
                  Reliable one-episode success (49 in 50)
                
98% (49 in 50) success rate

                  04
                  Diverse Thumb-Around Spinning
                
Diverse Thumb-Around Spinning Clips

Sonic (down)

              
                  01
                  Reliable one-peisode successes (50 in 50)
                
100% (50 in 50) success rate

                  04
                  Sonic Spinning
                
Sonic Spinning

Triangle Pass (palm-up)

              
                  01
                  Reliable one-episode successes (50 in 50)
                
100% (50 in 50) success rate

                  02
                  Sustained Rotation (avg. 39.9 episodes across 5 trails)
                
Avg. 39.9 episodes

              
                  01
                  Reliable one-episode successes (50 in 50)
                
100% (50 in 50) success rate

                  02
                  Sustained Rotation (25+ mins, 91 episodes x 2pi)
                
25+ mins, 91 episodes x 2pi

              
                  01
                  Reliable one-episode successes (100 in 100)
                
100% (100 in 100) success rate

                  02
                  Sustained Rotation (6 mins)
                
6 mins

Triangle Pass (palm-down)

              
                  01
                  Reliable one-episode successes (50 in 50)
                
100% (50 in 50) success rate

                  02
                  Sustained Rotation (avg. 40.1 episodes across 5 trails)
                
Avg. 40.1 episodes (20 episodes (4.5mins) - 43 episodes (~13mins) - 27.5 episodes (8mins 26s) - 77 episodes (23mins 38s) - 32.5 episodes (10mins))

Triangle Pass (palm-down, reversed)

              
                  01
                  Reliable one-episode successes (50 in 50)
                
100% (50 in 50) success rate

                  02
                  
                  Sustained Rotation (avg. 10.7 episodes across 5 trails)
                
Avg. 10.7 episodes

Four-Finger-Pass (palm-down)

              
                  01
                  Reliable one-episode successes (100 in 100) and Sustained Rotation (30+ mins, 134 x 2pi in total)
                
100% (100 in 100) success rate and 30+ mins, 134 x 2pi in total

                  02
                  Sustained Rotation (15+ mins)
                
15+ mins

Four-Finger-Pass (palm-down, LEAP)

              
                  01
                  Reliable one-episode successes (50 in 50)
                
100% (50 in 50) success rate

                  02
                  Reliable one-episode successes (48 in 50)
                
96% (48 in 50) success rate

              
                  03
                  Sustained Rotation
                
sustained rotation

                  04
                  Sustained Rotation (2 episodes)
                
2 episodes

Zero-shot Adaptation to 1) different object materials, 2) changed dynamics, 3) various object shapes.

Zero-shot Adaptation

              
                  01
                  Adapation to new geometry (cylinder -> hexagonal prism, octagonal prism; continuous rotation & multiple successes)
                
cylinder -> hexagonal prism, octagonal prism

                  02
                  Adapation to unseen materials (rigid -> soft; continuous rotation & multiple successe)
                
rigid -> soft

              
                  03
                  Adaptation to diverse shapes (T shape, L shape, ellipse, c-ellipse)
                
T shape, L shape, ellipse, c-ellipse

                  04
                  Remove the tape wrapped around the fingers (success rate: ~60%)
                
Low-friction scenario

              
                  05
                  Adaptation to new object shapes (T shape, L shape, long board)
                
T shape, L shape, long board

                  06
                  Adaptation to OOD object mass
                
Heavy object

              
                  06
                  Pass-v1 (cylinder -> octagonal prism)
                
cylinder -> octagonal prism

                  05
                  Adaptation to new object material (rigid -> soft)
                
rigid -> soft

              
                  05
                  Adaptation to new object geometry (T shape, L shape)
                
T shape, L shape

                  06
                  Adaptation to new object geometry (cylinder -> hexagonal prism)
                
cylinder -> hexagonal prism

              
                  09
                  Adaptation to new object geometry (cylinder -> octagonal prism)
                
cylinder -> octagonal prism

                  10
                  Adaptation to new object material (rigid -> soft)
                
rigid -> soft

              
                  09
                  Adaptation to new object geometry (cylinder -> hexagonal prism and octagonal prism) and new object material (rigid -> soft)
                
hexagonal prism, octagonal prism, soft material

                  01
                  A low-friction object (from lees than 50% success rate to 94% success rate (47/50))
                
A low-friction object

Some interesting results from our experiments.

Few-shot Adaptation

              
                  06
                  Adaptation to new object geometry (15cm x 2cm x 5cm, 16mins, from around 45% success rate to 100% success rate and sustained rotation; 15cm x 2cm x 10cm; 15cm x 2cm x 7cm )
                
Adaptation to new object geometry

                  02
                  A thinner ``pen''' (2.0 cm -> 1.4 cm, from less than 10% success rate to ~100% success rate)
                
Thick pen (2.0 cm) to thin pen (1.4cm)

              
                  03
                  Cuboid -> Octagonal Prism
                
Cuboid -> Octagonal Prism

                  04
                  Thumb-around (unssen materials, rigid -> soft; from less than 10% success rate to 96% success rate (48/50))
                
Adapation to unseen materials

              
                  05
                  New object geometry (from around 40% to 100% success rate)
                
New object geometry 

                  06
                  Object with a larger diamerter (1.4 cm -> 1.6 cm, from around 50% success rate to 100% success rate)
                
Adapation to object with a larger diamerter

              
                  05
                  New object geometry (cubodi -> cylinder, from around 30% to 100% success rate)
                
New object geometry 

                  05
                  Adaptation to new object shapes (cuboid -> octagonal prism, from around 50% to 100% success rate)
                
Adaptation to new object shapes (cuboid -> octagonal prism)

Robustness to Human Disturbances

              
                  01
                  1) Push the object out of the balanced pose, 2) Drag the object to an extreme initial position
                
Three-finger-pass (palm-up)

                  02
                  Consecutive successes with disturbances (three-finger-pass, palm-up)
                
Three-finger-pass (palm-up)

              
                  03
                  1) Push the object, 2) Drag the object out of the balancing pose
                
pass-v0

                  04
                  Consecutive successes with disturbances (pass-v0)
                
pass-v0

              
                  05
                  Push the object off the balance x 3
                
three-finger-pass, palm-down

                  06
                  Consecutive successes with disturbances (three-finger-pass, palm-down)
                
three-finger-pass, palm-down

Full takeover and partial online takeover

Trails on Human Takeover for Failure Recovery

              
                  01
                  Pass-v1, full takeover
                
full takeover

                  02
                  Pass-v1, partial online takeover
                
Partial online takeover

              
                  01
                  Pass-v2, full takeover
                
full takeover

                  02
                  Pass-v2, partial online takeover
                
Partial online takeover

              
                  01
                  Triangle pass (palm-up), full takeover
                
full takeover

                  02
                  Triangle pass (palm-up), partial online takeover
                
Partial online takeover

Initial position distribution, rollout distribution

Analysis on Cross-Rollout Variations

Three-finger-pass (palm up)

01 Consecutive success one-episode rollout (59/59)

59 consecutive successes · real-world footage at 10× speed.

02Rollout distribution

Drag to orbit · scroll to zoom

59 recorded trajectories, overlaid in 3D. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position distribution

Rotate · zoom · select a point

One point per rollout, at frame 0. Coordinates in mm relative to one shared centroid.

Full report & XY / XZ / YZ views ↗

Pass-v0

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-v0

02Rollout distribution

Drag to orbit · scroll to zoom

93 recorded rollouts with measured and target hand poses overlaid in 3D. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

93 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.

Full report & XY / XZ / YZ views ↗

Three-finger-pass (palm-down, reversed)

01 Successful one episode rollouts (the video is play in 10x speed)

Three-finger-pass (palm-down, reversed)

02Rollout distribution

Drag to orbit · scroll to zoom

33 successful rollouts with measured and target hand poses. Unreliable object-tracking intervals are hidden.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

33 frame-zero poses · XYZ / Quaternion. The diamond marks one pose estimated across a brief tracking gap.

Full report & XY / XZ / YZ views ↗

Three-finger-pass (palm-down)

01 Successful one episode rollouts (the video is play in 10x speed)

Three-finger-pass (palm-down)

02Rollout distribution

Drag to orbit · scroll to zoom

41 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

41 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.

Full report & XY / XZ / YZ views ↗

Pass-v1

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-v1

02Rollout distribution

Drag to orbit · scroll to zoom

50 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

50 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.

Full report & XY / XZ / YZ views ↗

Pass-v3

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-v3

02Rollout distribution

Drag to orbit · scroll to zoom

59 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

59 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.

Full report & XY / XZ / YZ views ↗

Fun Results

              
                  01
                  Fun Result (1) - Pass-v2
                
Fun Result (1) - Pass-v2

                  02
                  Fun Result (2) - Policy resulting from an intermediate reward searched by GPT
                
Fun Result (2)

The idea

TODO

The method

Mastering “Pen” Spinning in Diverse Styles

Diverse “Pen” Spinning Skills

01 Reliable one-episode success

02 Sustained rotation (averaged 4.025 episodes across 20 evaluations)

01 Sustained rotation (all in the original 1x speed)

01 Sustained rotation (all in the original 1x speed)

01 Reliable one-episode success (48 in 50)

02 Sustained rotation (4 rotation episodes)

01 Reliable one-episode success (45 in 50)

02 Sustained rotation (2 rotation episodes)

01 Reliable one-episode success (48 in 50)

02 Sustained rotation (3 rotation episodes)

01 Reliable one-episode success (49 in 50)

04 Diverse Thumb-Around Spinning

01 Reliable one-peisode successes (50 in 50)

04 Sonic Spinning

01 Reliable one-episode successes (50 in 50)

02 Sustained Rotation (avg. 39.9 episodes across 5 trails)

01 Reliable one-episode successes (50 in 50)

02 Sustained Rotation (25+ mins, 91 episodes x 2pi)

01 Reliable one-episode successes (100 in 100)

02 Sustained Rotation (6 mins)

01 Reliable one-episode successes (50 in 50)

02 Sustained Rotation (avg. 40.1 episodes across 5 trails)

01 Reliable one-episode successes (50 in 50)

02 Sustained Rotation (avg. 10.7 episodes across 5 trails)

01 Reliable one-episode successes (100 in 100) and Sustained Rotation (30+ mins, 134 x 2pi in total)

02 Sustained Rotation (15+ mins)

01 Reliable one-episode successes (50 in 50)

02 Reliable one-episode successes (48 in 50)

03 Sustained Rotation

04 Sustained Rotation (2 episodes)

Zero-shot Adaptation

01 Adapation to new geometry (cylinder -> hexagonal prism, octagonal prism; continuous rotation & multiple successes)

02 Adapation to unseen materials (rigid -> soft; continuous rotation & multiple successe)

03 Adaptation to diverse shapes (T shape, L shape, ellipse, c-ellipse)

04 Remove the tape wrapped around the fingers (success rate: ~60%)

05 Adaptation to new object shapes (T shape, L shape, long board)

06 Adaptation to OOD object mass

06 Pass-v1 (cylinder -> octagonal prism)

05 Adaptation to new object material (rigid -> soft)

05 Adaptation to new object geometry (T shape, L shape)

06 Adaptation to new object geometry (cylinder -> hexagonal prism)

09 Adaptation to new object geometry (cylinder -> octagonal prism)

10 Adaptation to new object material (rigid -> soft)

09 Adaptation to new object geometry (cylinder -> hexagonal prism and octagonal prism) and new object material (rigid -> soft)

01 A low-friction object (from lees than 50% success rate to 94% success rate (47/50))

Few-shot Adaptation

06 Adaptation to new object geometry (15cm x 2cm x 5cm, 16mins, from around 45% success rate to 100% success rate and sustained rotation; 15cm x 2cm x 10cm; 15cm x 2cm x 7cm )

02 A thinner ``pen''' (2.0 cm -> 1.4 cm, from less than 10% success rate to ~100% success rate)

03 Cuboid -> Octagonal Prism

04 Thumb-around (unssen materials, rigid -> soft; from less than 10% success rate to 96% success rate (48/50))

05 New object geometry (from around 40% to 100% success rate)

06 Object with a larger diamerter (1.4 cm -> 1.6 cm, from around 50% success rate to 100% success rate)

05 New object geometry (cubodi -> cylinder, from around 30% to 100% success rate)

05 Adaptation to new object shapes (cuboid -> octagonal prism, from around 50% to 100% success rate)

Robustness to Human Disturbances

01 1) Push the object out of the balanced pose, 2) Drag the object to an extreme initial position

02 Consecutive successes with disturbances (three-finger-pass, palm-up)

03 1) Push the object, 2) Drag the object out of the balancing pose

04 Consecutive successes with disturbances (pass-v0)

05 Push the object off the balance x 3

06 Consecutive successes with disturbances (three-finger-pass, palm-down)

Trails on Human Takeover for Failure Recovery

01 Pass-v1, full takeover

02 Pass-v1, partial online takeover

01 Pass-v2, full takeover

02 Pass-v2, partial online takeover

01 Triangle pass (palm-up), full takeover

02 Triangle pass (palm-up), partial online takeover

Analysis on Cross-Rollout Variations

Three-finger-pass (palm up)

01 Consecutive success one-episode rollout (59/59)

02Rollout distribution

03Initial position distribution

Pass-v0

01 Successful one episode rollouts (the video is play in 10x speed)

02Rollout distribution

03Initial position & orientation

Three-finger-pass (palm-down, reversed)