Apache can't access file while is has permission

I don't want to allow others read access, so I have added my user and the apache user to a group called apachme and then set that group to all the files and folders I want apache to have access to, including the root of the webpage. I have then given group and owner all permissions. After this apache still can't access the files without setting the read permission to allow all(rwxrwxr--(0774)). The folder that contains the index file, it's owned by apache with the group apachme and the permissions rwxrwxr-x(0775) with this I get a 500 error, changing it to 0774 will let it run fine These where just to confirm what user apache was running under and that it had been added to the group.