pix2pockets: Shot Suggestions in 8-Ball Pool from a Single Image in the Wild

Jonas Myhre Schiøtt*,1, Viktor Sebastian Petersen*,1, Dim P. Papadopoulos1,2,
1Technical University of Denmark, 2Pioneer Center for AI
SCIA 2025
Paper arXiv Video Code Demo Dataset

Abstract

Computer vision models have seen increased usage in sports, and reinforcement learning (RL) is famous for beating humans in strategic games such as Chess and Go. In this paper, we are interested in building upon these advances and examining the game of classic 8-ball pool. We introduce pix2pockets, a foundation for an RL-assisted pool coach. Given a single image of a pool table, we first aim to detect the table and the balls and then propose the optimal shot suggestion. For the first task, we build a dataset with 195 diverse images where we manually annotate all balls and table dots, leading to 5748 object segmentation masks. For the second task, we build a standardized RL environment that allows easy development and benchmarking of any RL algorithm. Our object detection model yields an AP50 of 91.2 while our ball location pipeline obtains an error of only 0.4 cm. Furthermore, we compare standard RL algorithms to set a baseline for the shot suggestion task and we show that all of them fail to pocket all balls without making a foul move. We also present a simple baseline that achieves a per-shot success rate of 94.7% and clears a full game in a single turn 30% of the time.

Presentation

Project Pipeline

The input image is run through the Ball Location Model to estimate the ball positions on the table, which is then handed to the Shot Suggestion Model. First, we obtain the dot- and ball detections, which we use to find the table lines and thus estimate a mapping from the image to a template. Then, we use the mapping to estimate the center point for the balls, resulting in the positions for the environment. The Shot Suggestion Model sends the state to the agent, suggesting the action. During training, the environment evaluates the action, and the agent receives a reward.

Environment Demo

In the 16 ball environment, the task is to complete a full turn (all blue balls and then the black) without violating the rules, such as pocketing the cue/black ball, or not pocketing a blue ball.

Mirror Environment

To include kick and bank shots, we establish an approach to mirror the balls and pockets to a separate space on each table side. Aiming for a ball or pocket in the mirrored space is equivalent to hitting a cushion and then a ball on the real table. The cyan points indicate the aiming points.

BibTeX


@misc{schiøtt2025pix2pocketsshotsuggestions8ball,
    title={pix2pockets: Shot Suggestions in 8-Ball Pool from a Single Image in the Wild}, 
    author={Jonas Myhre Schiøtt and Viktor Sebastian Petersen and Dimitrios P. Papadopoulos},
    year={2025},
    eprint={2504.12045},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2504.12045}, 
}