I think I need to move to specifying separate src.conf files for each build but I am going to get this working in a simple setup first.
It seems to compile the ports kernel modules into the OS, I need to be running the same kernel version as I am building, so I am putting it into its own VM.