The New Zealand-US company has won a contract that may send two of its Photon spacecraft on a mission to Mars in 2024.
The company has been picked to design the craft for Nasa's Escape and Plasma Acceleration and Dynamics Explorers (ESCAPDE) mission, to put craft into orbit around Mars to study its magnetosphere, to better understand how solar wind removes atmosphere over time.
Two of Rocketlab's small Photon spacecraft would be carried to near the planet aboard a large Nasa craft on an 11-month voyage and then be launched into orbit.
Rocket Lab founder and chief executive Peter Beck said the mission was a way of doing big science in a small package.
"Planetary science missions have traditionally costed hundreds of millions of dollars and taken up to a decade to come to fruition. Our Photon spacecraft for ESCAPADE will demonstrate a more cost-effective approach to planetary exploration that will increase the science community's access to our solar system for the better."
A decision on whether the mission will proceed will be made later this year by Nasa.
Rocket Lab is also currently working on a project for Nasa to put a satellite into orbit around the moon this year, as well as on a private mission to Venus in 2023.
The company is currently finalising plans to list on the US Nasdaq stock exchange using a special purpose acquisition company (SPAC), which values Rocket Lab at $US4.1 billion.