This set of problems was anything but a fun error to encounter, even though the fix was easy. I troubleshot this problem for the better part of 3 hours before finding the solution. Most importantly noted, is that most of the information currently on the internet was of no help and pointed in the wrong direction (thus the post). I managed to find an obscure reference to permissions which, ultimately, ended up being the problem.
To begin with, I started getting the following 403 error when attempting to access websites:
Forbidden You don’t have permission to access / on this server.
Okay, great, sounds like a permission problem. I checked the permissions and they were all fine; I even opened them up further, all to no avail.
I started digging through logs and found the following entries:
(13)Permission denied: / .htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable
[notice] child pid 4081 exit signal Aborted (6) libgcc_s.so.1 must be installed for pthread_cancel to work
The first showed up across all site logs and the second began showing up in the main Apache log every couple of seconds. This gave me serious pause. It should also be noted that I was doing site and permission script work prior to the problems and I had just executed some permission changes. While this might seem like an obvious point, it's not as clear as one might think. Upon examination of ALL of the file structures I'd been working on, I was not able to find anything out of order. All the permissions seemed to be set correctly.
I then disabled ALL websites, turned on debug in Apache, created a basic link to an index.html file. Then, restarted the daemon and tried again; still resulting in the same exact errors. At this point, I was completely baffled because there wasn't even an .htaccess entry anywhere to be found in the configurations at this point, I'd removed it all. To be sure I then disabled ALL of the Apache modules and conf.d entries which STILL did not resolve the problem and the errors remained the same.
After almost 2 hours of working through this, Googling, and wracking my brain as to the source, I was certainly not feeling happy and getting near to the point where I was considering a full out rebuild at 1:00 AM. Not good…
I gave the system a full reboot just in case something was going to come out in the wash that way. Not only did it not fix the problem, but it actually made everything worse! Now I couldn't even log into the system without root!
I started to get the following error:
Could not chdir to home directory /home/user: Permission denied /bin/bash: Permission denied
Great, now I had a broken Apache instance and a broken server. I managed to get in with root and attempted to su into a user account which proved fruitless as well. When su – user doesn't work as root, you know something has really gone wrong. However, this started to lead me in the direction of the resolution.
After more research I was able to find that during my permission work I somehow accidentally changed / to a 700 permission set. This rendered the operating system almost non-functional since only root had ability to navigate properly.
The fix was amazingly easy, as shown below:
chmod 755 /
That was it, as soon as I did that, all functionality returned to normal. Hopefully this little writeup is able to give a heads up to an obscure problem.