After a fresh install of filemaker server a security setting must be changed in order to open the photos database on the server. To do this open a command prompt on the server and run this command.
fmsadmin SET SERVERCONFIG SECUREFILESONLY=false
It will then ask you to authenticate with the admin console account and after you will be able to open the photos database in the admin console
