ShadowBench

In 1859 Henry Bursill published a book of hand shadows, in which each engraved plate shows the hands and the animal their shadow makes on the wall, and he assured his readers that he had drawn the position of the fingers with such care “that the most difficult subject may be accomplished after a few minutes”. ShadowBench gives frontier models the same task. A model sees one of these plates, controls two articulated 3D hands that stand between a lamp and a screen, and poses them through tool calls until their shadow matches the animal.

ShadowBench is on Hugging Face: browse it in ShadowBench Explorer, or load it from the ShadowBench dataset.

None of the six models we evaluated replicated a single plate. The best of them, Claude Opus 5.5 and Claude Sonnet 5.5 at maximum reasoning effort, average a little under half of a perfect score, which is less than a featureless blob of the right size earns when it is simply laid over each animal. More reasoning effort helps the Claude models considerably and the GPT models only slightly, and its price differs greatly from one model to another.

Mean score against the cost of one episode, at medium, high and maximum reasoning effort

The task in brief

There is no suitable dataset of hand shadows, since the one academic collection, HaSPeR, consists of frames from performers' videos labelled for recognition and records neither clean silhouettes nor the arrangement of the hands, so the targets come from Bursill's book and from two collections published around 1900 by William Hilliar and Félicien Trewey. In each plate the drawn hands hide part of their own shadow, so we traced the shadow, completed what the hands cover, and certified twenty of the results as targets. In an episode the model faces one target with two hands, each with the joints of a human hand, raised between a point lamp and a screen (Figure 2). It must first write a plan that says what the silhouette depicts and which parts of the hands will form which features, and it then bends the fingers and moves and turns the hands one tool call at a time, while the environment refuses any pose that a hand could not make. The model sees its shadow only when it asks to look, may consult the original page of the book at a small cost to its score, and has a hundred calls in all. The score compares the shadow with the target by area and by outline, so a blob in the right place earns part of it, and the ears, beaks and legs earn the rest.

One episode, from the book to the model's view of its own shadow

The models recognise the animals, recall the arrangements the books teach and describe accurately what is wrong with their shadows. What they lack is the ability to foresee what a change to their hands will do to the shadow, and so they assemble the animal from two separate shadows, with one hand placed far behind the other, where a person would hold the hands together. The second part of this post shows these results in detail, and the third describes the environment.

We evaluated six models, Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6.1 Sol, GPT-6.1 Terra and GPT-6 Luna, each at medium, high and maximum reasoning effort, with one episode on every plate, which makes 360 episodes, and no model solved a plate.

The models build the animal as a collage, with the hands far apart

In the plates that use both hands, the hands touch. They are clasped for the rabbit, laid one on the other for the elephant and crossed for the deer, so that together they form a single object held at one distance from the lamp, and the outline of that object is the animal. The models almost never arrange their hands in this way. Of the 295 final poses in which both hands cast a shadow, only seven hold the hands together. In the others the hands are typically some eighteen centimetres apart, and they are separated mostly along the light, with one hand about a quarter of a metre nearer the lamp than the other, so that their shadows overlap on the screen although the hands never meet. The hand nearer the lamp casts the larger shadow and usually supplies the bulk of the animal, a head or a body, while the other adds the details. Seen from the lamp, as in the second column of Figure 3, the hands appear joined, and seen from the side they turn out to be two separate objects.

How the books hold the hands, and how the models did

This is the arrangement one arrives at by thinking about the screen instead of the hands. Each hand's shadow is placed where it should fall, and the hand's distance from the lamp serves as a zoom, much as one would lay two paper cut-outs on a picture, and nothing in this method requires knowing how the two hands would fit against each other. Holding the hands together, as a person does, requires precisely that knowledge: where each finger lies in depth, which part of one hand rests against the other, and which part will hide which from the light. The models that tried to bring their hands together ran into the rule that skin cannot pass through skin, which refused such moves nearly eight hundred times, and they almost never looked at their hands from any side but the front, where depth cannot be seen. At maximum effort the hands end up even further apart than at medium, so the extra reasoning goes into placing the separate shadows more carefully.

The same habit explains how the models use their fingers. Their plans name the fingers the books use, and the Claude models usually raise the index and middle fingers as a rabbit's ears, while the GPT models often raise only the index finger and make the second ear from the thumb. In either case a finger is chosen for the place its shadow should occupy on the screen, and since the model cannot foresee where the shadow of a bent finger will fall, it finds that place by trial, bending, looking, bending again and often undoing what it did. This is the fumbling that one sees in the replays, and it continues at maximum effort because more reasoning does not supply the foresight that is missing.

Every shadow covers the right area and misses the animal's features

Figure 4 shows every final shadow at once, with the hardest plates on the left. The pattern is the same across models and plates: a dark mass of the right size sits in the right place, while the ears, legs, horns and beaks that make an animal recognisable are missing, merged into the mass or pointing the wrong way. The easiest plates are those built around one long shape that a forearm can supply, such as the neck of Hilliar's giraffe, and the best single episode of the evaluation, Claude Opus 5.5 on that giraffe, scored 68.

Every final shadow at maximum effort, laid over its target

More reasoning effort helps the Claude models far more than the GPT models

From medium to maximum effort, Claude Sonnet 5.5 gains about thirty points, Claude Haiku 5.5 about thirteen and Claude Opus 5.5 about nine, while the three GPT models gain about three each. For the Claude models most of the change lies in how much of the episode they use. At medium effort Claude Sonnet 5.5 submitted without moving either hand in thirteen of its twenty episodes, usually right after writing its plan, and its closing messages say so plainly, while at maximum effort it works until the call budget is nearly spent and ties for first place. The GPT models use almost the whole budget at every effort, so more reasoning has less to add.

Claude Sonnet 5.5's final shadows on the same plates, at medium and at maximum effort

The models know the animal and the arrangement the books teach

The plans the models write show how much they know about the task. About two thirds of those written before any look at the book name the right animal, and most of the misses are understandable ones, such as the head of a camel taken for a bird's head on a long neck. The plans assign parts of the hands to features in the way the books do, with two raised fingers for a rabbit's ears and an upright forearm for a swan's neck, and they go into some detail about angles and overlaps. Nearly nine in ten episodes also consult the book page, almost always within the first few calls and before moving a hand, so most models begin to build with the author's drawing of the hands in view. The episodes that consulted it nevertheless score no better than those that did not.

What the models planned to build, and what they built

The models are also good critics of their own shadows. Claude Opus 5.5 ended its best episode by observing that “the horn is missing, the snout sits too high and the eye hole is in the wrong place”, which is an exact account of what Figure 4 shows, and Claude Sonnet 5.5 closed its elephant by noting that the pose “misses the hanging trunk”. Knowing what the shadow should be, and seeing what is wrong with it, are both within their reach.

A move is almost as likely to make the shadow worse as better

Every model that keeps working improves its shadow from what it sees, and the scores rise quickly over the first thirty or so calls, while the hands are brought to roughly the right place and size. After that they rise slowly and unevenly, because the individual moves are poorly aimed. Of the seventeen thousand hand moves that the environment accepted, about four in ten made the shadow better, three in ten made it worse and the rest changed nothing the score could detect, and these proportions are nearly the same at maximum effort as at medium. Most of the moves that left the score unchanged did alter the shadow, but what they gained in one place they lost in another. With so many steps backwards a good pose is easily lost, and in more than a third of all episodes the submitted shadow scores clearly below the best one the same episode had reached, as when GPT-6 Luna lost its camel's head a few calls after finding it.

The score through an episode, and what each move did to the shadow

A blind search that is told the score does worse than the models

To measure what the models' knowledge of hands is worth, we ran CMA-ES, a standard evolutionary optimizer, over the same fifty-eight numbers that describe a pose, under the same anatomical rules. It starts from the resting pose, knows nothing about hands or animals, and is told the score of every pose it tries, some six thousand of them for each plate. It averages 36, below every model at maximum effort except GPT-6 Luna, and its best poses, in the bottom row of Figure 4, are tangles of fingers that resemble nothing. The models' knowledge of which finger becomes which feature is therefore worth a good deal, although it falls far short of what the task requires.

Why models that understand 3D cannot make a shadow puppet

Frontier models are described, with reason, as having a good grasp of three-dimensional space. They answer questions about depth, shape and arrangement in images, and in our own SplatBench the best of them rebuilt objects from photographs by placing hundreds of 3D primitives. ShadowBench shows the same competence: the models recognise an animal from its silhouette, read the drawing of the hands on a book plate, choose the classic arrangement of fingers, describe exactly what is wrong with their shadows and handle the tools fluently, which is unsurprising, since the controls are those of a hand, each joint bending as a hand's joint bends. Whatever stops them lies beyond the interface.

What a shadow puppet requires, and what the models lack, is the ability to predict how an arrangement of one's own body will look from another point of view, in this case from a lamp. A person carries such a model of their hands, built over a lifetime of moving them and seeing what happens, and kept current at every moment by the sense of where each finger is, which is why a reader of Bursill's book can copy one of his drawings in a few minutes. Language models learned about three dimensions by looking at the world in images and reading about it, and their training contains almost nothing that pairs an arrangement of a body with the appearance it produces from a given viewpoint. They can recognise and describe a hand in any pose, but they cannot imagine the shadow of a pose before they make it.

The results show the consequences one would expect from this. Because the models cannot predict the effect of a move, a move is nearly as likely to make the shadow worse as better. Because they cannot reason about how two hands fit together in space, they avoid the question and compose the animal on the screen, with the hands far apart. And because more reasoning works with the knowledge a model already has, it makes the trial and error more patient and more careful without making the trials better aimed. The same gap explains how robotics systems that control real hands are built, with a language model that plans paired with a separate policy trained on recorded actions and their outcomes; specialised methods such as Hand-Shadow Poser (Xu et al., SIGGRAPH 2025) likewise solve the inverse problem of hand shadows with machinery built for it. ShadowBench asks a single general model to plan and to act at once, and as an environment with a verifiable reward it is also a place where a model could acquire the missing relation from experience, through many poses and the shadows they cast.

Replays of all 360 episodes

Every episode can be replayed call by call, with the stage in 3D, the screen as the model saw it, the plan in force and the book page once the model had consulted it. The explorer below opens on the best episode of the evaluation, Claude Opus 5.5's giraffe, in which the neck is made from the right forearm and most of the calls go into the head; the marked calls are its three largest improvements. Choose a model, an effort and a plate to see any other episode, or select any shadow in Figure 4.

Replay any of the 360 episodes

The targets come from three books of hand shadows

We needed silhouettes that hands are known to make, drawn by people who made them, and the best source we found is the parlour books of the nineteenth century, which draw each shadow on the wall beside the hands that cast it. Fifteen plates come from Bursill's Hand Shadows to be Thrown upon the Wall (1859), four from Hilliar's Novel Hand Shadows (about 1900) and one from Trewey's The Art of Shadowgraphy (about 1900), all of them in the public domain. In every plate the drawn hands cover part of their own shadow, so we traced the visible shadow from each scan, completed the hidden parts, and then a person corrected each drawing in a vector editor and certified it. Ten plates were left out because they repeat a subject or cannot be completed with confidence. The targets are an illustrator's drawings of shadows, so whether real hands, or our model of them, can reproduce each one exactly is not known, and we read the best scores as lower bounds on what the hands can do.

The book plates and the targets made from them

The stage: a point lamp, a screen and two hands

Most of us made our first hand shadows as children, with a lamp, a wall and our own hands: two raised fingers for the ears of a rabbit, a flat hand whose little finger drops to open a dog's mouth, two hands hooked together at the thumbs for a bird in flight. The stage of ShadowBench is that scene reduced to its essentials. A single lamp shines on a screen, and two hands stand between them, raised at the start with their palms toward the screen.

The shadows behave as they did on the bedroom wall. A hand brought toward the lamp throws a larger shadow and a hand brought toward the screen a smaller one, and because the light spreads out from one point, a hand that enlarges its shadow in this way also pushes it outward from the middle of the screen, so the size of the animal and its place on the screen have to be found together. The one difference is that the lamp is a true point of light, so the shadow keeps a sharp edge at any distance, where a household bulb would blur it as the hand drew near.

The hands are the hand-and-forearm model from Ultraleap's Unity plugin (Apache License 2.0), and the same skin is drawn on the stage, casts the shadow and is checked for collisions. They can do what a child's hands do at the lamp: each finger bends at its knuckles and spreads away from its neighbours, the thumb swings across the palm, the wrist bends forward, back and to either side, the forearm rolls, and each hand can be moved and turned as a whole. The exact dimensions of the stage and the range of every joint are given in the specification below.

What differs is the way the hands are moved. A child moves a finger and watches the wall at the same moment, while the model must write each change as an angle, a distance or a turn, and it sees the shadow it has made only when it asks to look.

The stage, with GPT-6.1 Sol's swan

The environment refuses poses that a hand cannot make

Every joint has a conservative range, a finger cannot spread as far sideways when it is bent as when it is straight, and the skin may pass neither through itself nor through the other hand. The motion from one pose to the next is checked in small steps, so a finger cannot jump through another on its way to a legal position, and one call may change a joint by at most 45 degrees, move a hand by at most 15 centimetres or turn it by at most 45 degrees. A call that breaks a rule is refused: it uses up one of the model's calls, returns a short explanation and leaves the hands where they were.

Moves the environment refused during the evaluation

Tools, budgets and what the model sees

An episode begins with one image of the target and of the hands in their starting pose. Before it may move a hand the model must call plan, giving what the silhouette depicts and which parts of the hands will form which features; the plan can be revised at any time and is kept with the episode, but it is never scored. The tools set_fingers, move_hand and turn_hand change the pose, undo reverses the last change, and describe returns the joint angles and their ranges as numbers. look returns the four panels in Figure 2: the target, the shadow, their difference and the hands, seen from the front, the back, the side, above or at an angle. peek_book returns the original page of the book, with the author's drawing of the hands, and each peek costs 6% of the final score. The episode ends when the model calls submit or after 100 calls, of which at most 32 may be look, and the model never sees its score.

The score compares the shadow with the target by area and by outline

Most of the score comes from the overlap of the shadow and the target divided by their union, which rewards covering the right area. The rest measures whether the edge of the shadow lies where the edge of the target lies, within one, two and four pixels, which is the part that ears, beaks, snouts and legs earn. The holes in a target, such as an eye, add a small term, the posture of the wrists a smaller one, and each peek multiplies the result by 0.94:

where $h$ is the share of the target's holes that the shadow leaves open, $w$ is how close the worse wrist is to neutral according to the wrist-position step of the RULA ergonomic assessment, and $p$ is the number of peeks. A plate counts as solved only when the overlap, the outline and the holes each reach 95%.

What scores look like on one plate

Environment specification

Element	Specification
Stage	Point lamp 2.5 m from the screen; screen 0.8 × 0.6 m, scored at 512 × 384 pixels
Hands	Ultraleap hand and forearm, 23 controls per hand plus position and orientation (58 numbers)
Tools	plan, look, describe, set_fingers, move_hand, turn_hand, undo, peek_book, submit
Budget	100 calls per episode, at most 32 of them look; at most 3 peeks
Limits per call	45° per joint, 15 cm of translation, 45° of rotation
Hard rules	Joint ranges, spread that narrows with flexion, no skin through skin, motion checked in small steps
Score	Overlap (55%) and outline (45%), times hole and wrist terms, times 0.94 per peek
Solved	Overlap, outline and holes each at least 95%
Targets	20 certified plates from Bursill (1859), Hilliar (c. 1900) and Trewey (c. 1900)

The environment runs as an OpenEnv server, and every call is logged with the pose before and after it, whether it was accepted, the reason for any refusal and the hidden score. These logs make the replays on this page possible, and the score after each call can serve as a dense reward in training.

Limits

Each model was run once per plate and effort, so small differences between configurations, such as the tie at the top, are not resolved. The targets are drawings, and the reachable optimum for each one is unknown. The hand model has conservative joint limits, and real hands, with softer skin and more flexible thumbs, can make some shapes that it forbids. Whether a model peeked at the book was its own choice, so the comparison of episodes that peeked with those that did not is observational. We evaluated six models from two laboratories, and we plan to add others, and a human baseline, as we extend the benchmark.

Access to the environment, targets and trajectories, and citation

The environment, the twenty certified targets and the trajectories of all 360 episodes are public on Hugging Face. The dataset holds the environment as a Python package, together with the script that evaluates a model under the protocol of this report, and it records every episode call by call: the arguments of each call, the text and image the model received, the pose of both hands afterwards and the hidden score, which can serve as a dense reward for training. ShadowBench Explorer presents the results and replays any episode in the browser with the images the model saw.

@misc{perpetual2026shadowbench,
  title  = {ShadowBench: Posing Hands to Cast Shadow Puppets from Nineteenth-Century Books},
  author = {{Perpetual Research}},
  year   = {2026},
  url    = {https://perpetualresearch.si/shadowbench}
}