VLM-based Decision Making for Off-Road Autonomous Driving
Background
Eira Systems is a Swedish start-up developing electrically powered unmanned ground vehicles for logistics, CASEVAC and other supporting missions in demanding off-road environments. Eira has developed and tested the full-scale Eira-X prototype and has now received a contract to develop the framework for a second-generation platform together with a major Swedish defence actor. Eira is based at The Yard in Lindholmen, Gothenburg.
Unmanned operations need autonomy. Eira is developing both driver-assist features that make remote-controlled operation more reliable and point-to-point autonomous navigation.
Off-road driving often depends on context: whether a stretch of water can be crossed, or a gap is wide enough to pass. Vision-language models (VLMs) can reason about that context.
Goal description
The goal is to use a VLM to make high-level driving decisions off-road, and to test it on our vehicles. SOme possible directions are:
- Taking over in hard situations: a slow VLM that steps in for the big decisions
- VLAs: explore vision-language-action models that output driving actions directly
- More context: a slower VLM that uses more camera views, including thermal, the recent past and satellite images
- Route planning: planning routes with a VLM from satellite and height maps, position and cameras
- UAV collaboration: a VLM that works together with a UAV, using its view to make decisions
In every direction, the VLM should call the operator when it needs help and a link is available, and otherwise try to re-establish a link.
Our vehicle has stereo and thermal cameras, IMUs, wheel odometry, an Ouster lidar and an Nvidia Jetson AGX Orin. We also have a 1/10-scale vehicle with a similar sensor and compute setup.
Want to know more about how you could attack this topic? Reach out to us! We would love to support you in exploring your own ideas in a related topic. We also have a few more experimental topics if you want something even more challenging, as well as thesis proposals in other areas.
Student background
We are looking for students who are passionate about autonomous ground vehicles, off-road driving, self-driving or physical AI in general. You only need to be ambitious and curious to apply. Applicants must, however, be citizens of a NATO or EU member state. We assess applications on the following, in order of priority (attach something for at least one of them):
- Projects (school or hobby) in a related area
- Your personality and motivation for this project
- Merits
- Knowledge (your bachelor's and master's programmes, and relevant courses)
Apply before November 1st to andreas.munck@eirasystems.com