Benchmark Submission
SEANavBench
How to Submit
For a complete tutorial, please see: https://sean.interactive-machines.com/tutorials/submission
Currently, we require the submission to be a zipped ROS package that contains a submission.launch file inside the launch folder. In other words, we will run roslaunch $package_name/launch/submission.launch to benchmark the submission.
Here is an example submission. The submission.launch file launches exactly one node, the move_base node. If you use tmux and has a configuration similar to the example in the "ROS" section, you only need to replace roslaunch --wait social_sim_ros kuri_move_base.launch with roslaunch --wait $your_package_name submission.launch to test it on your own.
Make a Submission
Make your submission at the benchmark website: https://benchmark.interactive-machines.com
T2FPV
Overview
T2FPV is a new trajectory forecasting benchmarking track that focuses on the task of forecasting pedestrians’ future paths, while handling noisy sensing from a first-person perspective. The T2FPV benchmark track leverages SEANavBench to replay and record all pedestrians from the original ETH/UCY dataset. Trajectories are then generated using state-of-the-art methods for detection and tracking, incurring errors due to field-of-view, occlusion, and algorithmic limitations. The forecasting task comprises predicting the ground-truth future trajectories of both the ego agent and observed agents, while only having access to tracks provided from noisy perception. Across the five folds in ETH/UCY, T2FPV has the following statistics, demonstrating its difficulty:
- 49,116 total ego scenes, with 136,042 additional detected pedestrians
- Average detected pedestrian track MSE to ground truth of 1.38m
- Average “missing observation” rate of 40%
How to Submit
The challenge submission portal is hosted on the Eval AI platform, at the following url: https://eval.ai/web/challenges/challenge-page/2086/evaluation
Please follow the terms and conditions outlined there regarding expectations for training models and selecting the best-of-six predicted trajectories. Steps for attaining data, training models, and producing the prediction file are listed out on the GitHub, at https://github.com/cmubig/T2FPV