One scene + “open the drawer”
01 · Plan the chaindesk → drawer → handleThe image and “open the drawer” first go to the vision-language model. It returns the short chain desk → drawer → black handle. This supplies both an order and a containment relation: look for the drawer within the desk, then for the handle within the drawer. The first search receives the original image and the word “desk”.
02 · Propose boxesCurrent image → boxesAt each level, Faster R-CNN receives the current image and produces candidate boxes. On the last level, that image is the drawer crop. Each box marks a region that the next stage can inspect against the phrase “black handle”. The proposals provide locations; the text-guided evaluation decides which location fits the instruction.
03 · Select a boxBoxes + target → accepted boxFor each evaluated box, the program crops its pixels and asks Qwen2-VL how well they match “black handle”. The number of evaluations is limited in advance. Upper confidence bound (UCB) chooses which candidate to evaluate next, balancing its average score with how few times it has been examined. After this budget is spent, candidates are checked in average-score order. The program slightly varies brightness, rotation and scale; a candidate passes when its mean score is high enough and the range of scores is small enough. A rejected candidate leads to another box at this level. With no candidates, or no accepted candidate after the allowed retries, the chain stops.
04 · Crop and advanceAccepted box → crop + offsetAn accepted desk box becomes the input image for finding the drawer; the accepted drawer box becomes the input for finding its handle. Steps 02 and 03 repeat on each new image. In the schematic, the desk begins at (120, 140) in the scene, and the drawer begins at (90, 80) inside the desk. The drawer crop’s origin in the scene is therefore (210, 220). Keeping this offset makes it possible to search locally without losing the object’s original location.
05 · Map to the sceneFinal part in scene coordinatesSuppose the selected handle box runs from (200, 90) to (320, 120) inside the drawer crop. Adding the drawer’s origin, (210, 220), gives the box (410, 310)–(530, 340) in the original scene. Its width and height remain 120 × 30 pixels. The localization output now carries a part name and a position that another module can use in that scene.
Final target → 05
Next target: return the crop to 02
Stop the chain
No accepted candidate
A separate use across many images
Collect the localization results, then use them as supervision.
A dataset of part boxesImage · part name · box
Train a detectorPredict part locations from images
Repeat on many images