The trusted face component of Lollipop's Smart Lock system is really neat, but it's a little buggy on the Nexus 6. Shortly after launch, users noticed that the front-facing camera wasn't very good at recognizing them. According to the Google Groups thread, Google has fixed the problem in the Android code and an update is on the way.
The issue stems from the way the camera adjusts to the lighting in your environment.
Upcoming Nexus 6 OTA (Probably 5.0.1) Will Include Trusted Face Lighting Fix was written by the awesome team at Android Police.