Mastering “Pen” Spinning in Diverse Styles

Main Real World Results (featured by high-speed, fine-grained movements, sophisiticated gaiting, and realiability)

Diverse “Pen” Spinning Skills

Pass-v0

              
                  01
                  Pass-v0 consecutive one-episode success
                
Pass-v0 task with 50 in 50 success

                  02
                  Pass-v0 continuous rotation (10.5 rotation episodes)
                
Pass-v0 continuous rotation (10.5 rotation episodes)

              
                  01
                  Pass-v0 continuous success (all in the original 1x speed)
                
Pass-v0 continuous success

                    01
                    Pass-v0 continuous success (all in the original 1x speed)
                  
Pass-v0 continuous success

Pass-v1

              
                  01
                  Reliable one-episode success (48 in 50)
                
96% success rate

                  02
                  Pass-v1 continuous rotation (4 rotation episodes)
                
Pass-v1 continuous rotation (4 rotation episodes)

Pass-v2

              
                  01
                  Pass-v2 continuous success
                
Pass-v2 task with 45 in 50 success

                  02
                  Pass-v2 continuous rotation (2 rotation episodes)
                
Pass-v2 continuous rotation (2 rotation episodes)

Pass-v3

              
                  01
                  Pass-v3 reliable one-episode success (48 in 50)
                
96% success rate

                  02
                  Pass-v3 continuous rotation (3 rotation episodes)
                
Pass-v3 continuous rotation (3 rotation episodes)

Thumb-Around

              
                  01
                  Thumb-Around (rigid object) task continuous success
                
Thumb-Around (rigid object) task with 49 in 50 success

                  04
                  Diverse Thumb-Around Rotations
                
Diverse Thumb-Around Rotations

Sonic (down)

              
                  01
                  Reliable one-peisode successes (50 in 50)
                
100% success rate

                  04
                  Sonic Rotations
                
Sonic Rotations

                  01
                  Simple Sonic-1 task with 50 in 50 success
                
Simple Sonic-1 task with 50 in 50 success

                  01
                  Simple Sonic-1 task with 50 in 50 success
                
Simple Sonic-1 task with 50 in 50 success

Three-finger-pass (palm up)

              
                  01
                  Three-finger-pass (palm up) continuous success
                
Three-finger-pass (palm up) task with 50 in 50 success

                  02
                  Continuous rotation (avg. 39.9 episodes across 5 trails)
                
Avg. 39.9 episodes

              
                  01
                  Three-finger-pass (palm up) continuous success
                
Three-finger-pass (palm up) task with 50 in 50 success

                  02
                  Three-finger-pass (palm up) continuous rotation (25+ mins, 91 episodes x 2pi)
                
Three-finger-pass (palm up) continuous rotation (25+ mins, 91 episodes x 2pi)

              
                  01
                  Three-finger-pass (palm up) continuous success
                
Three-finger-pass (palm up) task with 100 in 100 success

                  02
                  Three-finger-pass (palm up) continuous rotation (6 mins)
                
Three-finger-pass (palm up) continuous rotation (6 mins)

Three-finger-pass (palm down)

              
                  01
                  Three-finger-pass (palm down) continuous success
                
Three-finger-pass (palm down) task with 50 in 50 success

                  02
                  Continuous rotation (avg. 40.1 episodes across 5 trails)
                
Avg. 40.1 episodes (20 episodes (4.5mins) - 43 episodes (~13mins) - 27.5 episodes (8mins 26s) - 77 episodes (23mins 38s) - 32.5 episodes (10mins))

Three-finger-pass-v2 (palm down)

              
                  01
                  Three-finger-pass-v2 (palm down) continuous success
                
Three-finger-pass-v2 (palm down) task with 50 in 50 success

                  02
                  
                  Continuous rotation (avg. 10.7 episodes across 5 trails)
                
Avg. 10.7 episodes

Four-finger-pass (palm down)

              
                  01
                  Four-finger-pass (palm down) continuous rotation (30+ mins, 134 x 2pi in total)
                
Four-finger-pass (palm down) task continuous rotation (30+mins, 134 x 2pi in total)

                  02
                  Four-finger-pass (palm down) continuous rotation (15+ mins, 134 x 2pi in total)
                
Four-finger-pass (palm down) continuous rotation (15+ mins, 134 x 2pi in total)

Four-finger-pass (palm down)

              
                  01
                  Four-finger-pass (palm down, leap) task continuous success (50 in 50)
                
Four-finger-pass (palm down, leap) task continuous success (50 in 50)

                  02
                  Four-finger-pass (palm down, leap) task continuous success (48 in 50)
                
Four-finger-pass (palm down, leap) task continuous success (48 in 50)

              
                  03
                  Four-finger-pass (palm down, leap) task continuous rotation
                
Four-finger-pass (palm down, leap) task continuous rotation

                  04
                  Four-finger-pass (palm down, leap) task continuous rotation
                
Four-finger-pass (palm down, leap) task continuous rotation

Zero-shot Adaptation to 1) different object materials, 2) changed dynamics, 3) various object shapes.

Zero-shot Adaptation

              
                  01
                  Adapation to new geometry (cylinder -> hexagonal prism, octagonal prism; continuous rotation & multiple successes)
                
Adapation to new geometry (cylinder -> hexagonal prism, octagonal prism, continuous rotation & multiple successes)

                  02
                  Adapation to unseen materials (rigid -> soft; continuous rotation & multiple successe)
                
Adapation to unseen materials (rigid -> soft; continuous rotation & multiple successe)

              
                  03
                  Adaptation to diverse shapes (T shape, L shape, ellipse, c-ellipse)
                
Adaptation to diverse shapes (T shape, L shape, ellipse, c-ellipse)

                  04
                  Remove the tape wrapped around the fingers (success rate: ~60%)
                
Remove the tape wrapped around the fingers

              
                  05
                  Adaptation to new object shapes (T shape, L shape, long board)
                
Adaptation to new object shapes (T shape, L shape, long board)

                  06
                  Adaptation to OOD object mass
                
Adaptation to OOD object mass

              
                  06
                  Pass-v1 (cylinder -> octagonal prism)
                
Adaptation to new object geometry (cylinder -> octagonal prism)

                  05
                  Adaptation to new object material (rigid -> soft)
                
Adaptation to new object material (rigid -> soft)

              
                  05
                  Adaptation to new object geometry (T shape, L shape)
                
T shape, L shape

                  06
                  Adaptation to new object geometry (cylinder -> hexagonal prism)
                
cylinder -> hexagonal prism

              
                  09
                  Adaptation to new object geometry (cylinder -> octagonal prism)
                
cylinder -> octagonal prism

                  10
                  Adaptation to new object material (rigid -> soft)
                
rigid -> soft

              
                  09
                  Adaptation to new object geometry (cylinder -> hexagonal prism and octagonal prism) and new object material (rigid -> soft)
                
hexagonal prism, octagonal prism, soft material

Some interesting results from our experiments.

Few-shot Adaptation

              
                  01
                  A low-friction object (from lees than 50% success rate to 94% success rate (47/50))
                
A low-friction object

                  02
                  A thinner ``pen''' (2.0 cm -> 1.4 cm, from less than 10% success rate to ~100% success rate)
                
Thick pen (2.0 cm) to thin pen (1.4cm)

              
                  03
                  Cuboid -> Octagonal Prism
                
Cuboid -> Octagonal Prism

                  04
                  Thumb-around (unssen materials, rigid -> soft; from less than 10% success rate to 96% success rate (48/50))
                
Adapation to unseen materials

              
                  05
                  New object geometry (from around 40% to 100% success rate)
                
New object geometry 

                  06
                  Object with a larger diamerter (1.4 cm -> 1.6 cm, from around 50% success rate to 100% success rate)
                
Adapation to object with a larger diamerter

              
                  05
                  New object geometry (cubodi -> cylinder, from around 30% to 100% success rate)
                
New object geometry 

                  05
                  Adaptation to new object shapes (cuboid -> octagonal prism, from around 50% to 100% success rate)
                
Adaptation to new object shapes (cuboid -> octagonal prism)

              
                  06
                  Adaptation to new object geometry (15cm x 2cm x 5cm, 16mins, from around 45% success rate to 100% success rate and sustained rotation; 15cm x 2cm x 10cm; 15cm x 2cm x 7cm )
                
Adaptation to new object geometry

Robustness to Human Disturbances

              
                  01
                  1) Push the object out of the balanced pose, 2) Drag the object to an extreme initial position
                
Three-finger-pass (palm-up)

                  02
                  Consecutive successes with disturbances (three-finger-pass, palm-up)
                
Three-finger-pass (palm-up)

              
                  03
                  1) Push the object, 2) Drag the object out of the balancing pose
                
pass-v0

                  04
                  Consecutive successes with disturbances (pass-v0)
                
pass-v0

              
                  05
                  Push the object off the balance x 3
                
three-finger-pass, palm-down

                  06
                  Consecutive successes with disturbances (three-finger-pass, palm-down)
                
three-finger-pass, palm-down

Initial position distribution, rollout distribution

Robustness to initial position distribution, transition stochasticity, and hardware instability

Three-finger-pass (palm up)

01 Consecutive success one-episode rollout (59/59)

59 consecutive successes · real-world footage at 10× speed.

02Rollout distribution

Drag to orbit · scroll to zoom

59 recorded trajectories, overlaid in 3D. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position distribution

Rotate · zoom · select a point

One point per rollout, at frame 0. Coordinates in mm relative to one shared centroid.

Full report & XY / XZ / YZ views ↗

Pass-v0

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-v0

02Rollout distribution

Drag to orbit · scroll to zoom

93 recorded rollouts with measured and target hand poses overlaid in 3D. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

93 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.

Full report & XY / XZ / YZ views ↗

Three-finger-pass (palm-down, reversed)

01 Successful one episode rollouts (the video is play in 10x speed)

Three-finger-pass (palm-down, reversed)

02Rollout distribution

Drag to orbit · scroll to zoom

33 successful rollouts with measured and target hand poses. Unreliable object-tracking intervals are hidden.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

33 frame-zero poses · XYZ / Quaternion. The diamond marks one pose estimated across a brief tracking gap.

Full report & XY / XZ / YZ views ↗

Three-finger-pass (palm-down)

01 Successful one episode rollouts (the video is play in 10x speed)

Three-finger-pass (palm-down)

02Rollout distribution

Drag to orbit · scroll to zoom

41 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

41 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.

Full report & XY / XZ / YZ views ↗

Pass-v1

01 Successful one episode rollouts (the video is play in 10x speed)

Pass-v1

02Rollout distribution

Drag to orbit · scroll to zoom

50 successful rollouts with measured and target hand poses. Pause, scrub time, or change playback speed.

Open standalone viewer ↗

03Initial position & orientation

Rotate · zoom · select a point

50 initial poses, one per rollout at frame 0. Switch between XYZ positions and quaternion orientations.

Full report & XY / XZ / YZ views ↗

Fun Results

              
                  01
                  Fun Result (1) - Pass-v2
                
Fun Result (1) - Pass-v2

                  02
                  Fun Result (2) - Policy resulting from an intermediate reward searched by GPT
                
Fun Result (2)

The idea

TODO

The method

Mastering “Pen” Spinning in Diverse Styles

Diverse “Pen” Spinning Skills

01 Pass-v0 consecutive one-episode success

02 Pass-v0 continuous rotation (10.5 rotation episodes)

01 Pass-v0 continuous success (all in the original 1x speed)

01 Pass-v0 continuous success (all in the original 1x speed)

01 Reliable one-episode success (48 in 50)

02 Pass-v1 continuous rotation (4 rotation episodes)

01 Pass-v2 continuous success

02 Pass-v2 continuous rotation (2 rotation episodes)

01 Pass-v3 reliable one-episode success (48 in 50)

02 Pass-v3 continuous rotation (3 rotation episodes)

01 Thumb-Around (rigid object) task continuous success

04 Diverse Thumb-Around Rotations

01 Reliable one-peisode successes (50 in 50)

04 Sonic Rotations

01 Simple Sonic-1 task with 50 in 50 success

01 Simple Sonic-1 task with 50 in 50 success

01 Three-finger-pass (palm up) continuous success

02 Continuous rotation (avg. 39.9 episodes across 5 trails)

01 Three-finger-pass (palm up) continuous success

02 Three-finger-pass (palm up) continuous rotation (25+ mins, 91 episodes x 2pi)

01 Three-finger-pass (palm up) continuous success

02 Three-finger-pass (palm up) continuous rotation (6 mins)

01 Three-finger-pass (palm down) continuous success

02 Continuous rotation (avg. 40.1 episodes across 5 trails)

01 Three-finger-pass-v2 (palm down) continuous success

02 Continuous rotation (avg. 10.7 episodes across 5 trails)

01 Four-finger-pass (palm down) continuous rotation (30+ mins, 134 x 2pi in total)

02 Four-finger-pass (palm down) continuous rotation (15+ mins, 134 x 2pi in total)

01 Four-finger-pass (palm down, leap) task continuous success (50 in 50)

02 Four-finger-pass (palm down, leap) task continuous success (48 in 50)

03 Four-finger-pass (palm down, leap) task continuous rotation

04 Four-finger-pass (palm down, leap) task continuous rotation

Zero-shot Adaptation

01 Adapation to new geometry (cylinder -> hexagonal prism, octagonal prism; continuous rotation & multiple successes)

02 Adapation to unseen materials (rigid -> soft; continuous rotation & multiple successe)

03 Adaptation to diverse shapes (T shape, L shape, ellipse, c-ellipse)

04 Remove the tape wrapped around the fingers (success rate: ~60%)

05 Adaptation to new object shapes (T shape, L shape, long board)

06 Adaptation to OOD object mass

06 Pass-v1 (cylinder -> octagonal prism)

05 Adaptation to new object material (rigid -> soft)

05 Adaptation to new object geometry (T shape, L shape)

06 Adaptation to new object geometry (cylinder -> hexagonal prism)

09 Adaptation to new object geometry (cylinder -> octagonal prism)

10 Adaptation to new object material (rigid -> soft)

09 Adaptation to new object geometry (cylinder -> hexagonal prism and octagonal prism) and new object material (rigid -> soft)

Few-shot Adaptation

01 A low-friction object (from lees than 50% success rate to 94% success rate (47/50))

02 A thinner ``pen''' (2.0 cm -> 1.4 cm, from less than 10% success rate to ~100% success rate)

03 Cuboid -> Octagonal Prism

04 Thumb-around (unssen materials, rigid -> soft; from less than 10% success rate to 96% success rate (48/50))

05 New object geometry (from around 40% to 100% success rate)

06 Object with a larger diamerter (1.4 cm -> 1.6 cm, from around 50% success rate to 100% success rate)

05 New object geometry (cubodi -> cylinder, from around 30% to 100% success rate)

05 Adaptation to new object shapes (cuboid -> octagonal prism, from around 50% to 100% success rate)

06 Adaptation to new object geometry (15cm x 2cm x 5cm, 16mins, from around 45% success rate to 100% success rate and sustained rotation; 15cm x 2cm x 10cm; 15cm x 2cm x 7cm )

Robustness to Human Disturbances

01 1) Push the object out of the balanced pose, 2) Drag the object to an extreme initial position

02 Consecutive successes with disturbances (three-finger-pass, palm-up)

03 1) Push the object, 2) Drag the object out of the balancing pose

04 Consecutive successes with disturbances (pass-v0)

05 Push the object off the balance x 3

06 Consecutive successes with disturbances (three-finger-pass, palm-down)

Robustness to initial position distribution, transition stochasticity, and hardware instability

Three-finger-pass (palm up)

01 Consecutive success one-episode rollout (59/59)

02Rollout distribution

03Initial position distribution

Pass-v0

01 Successful one episode rollouts (the video is play in 10x speed)

02Rollout distribution

03Initial position & orientation

Three-finger-pass (palm-down, reversed)

01 Successful one episode rollouts (the video is play in 10x speed)

02Rollout distribution

03Initial position & orientation

Three-finger-pass (palm-down)

01 Successful one episode rollouts (the video is play in 10x speed)