Should I trust user tests outside a controlled environment

Recently we started to do user tests for a new website we are launching. At first my boss was against user testing because 'reasons'*.

After I conducted the first few tests we already identified a couple of pain points. Solving these pain points made the product a lot better. My boss now had a renewed view on user tests and wanted to test it on more users** (mainly his neighbors / friends etc.), so i prepared a set of tasks for the users that I would use in our controlled environment and he could test tasks on users he knew.

My current problem

Tests done in our controlled environment reflect nearly the opposite results of his own user tests. I know he can be a bit projecting regarding, so this got me wondering: Should I trust user tests that are done outside a controlled environment?

*With reasons I mean the one that most people use is that 'I understand the target group better then the target group understands themselves etc'

**Yes these users fall within the target group, our target group is extremely wide. Basically anyone from 18 till 50 years old.