Recently an issue surfaced with one of my Apache servers. Page loads were suddenly taking WAY too long to load. In some cases, I was seeing over a minute long load times.
After some head scratching and pointless page refreshes to see if maybe it would fix itself, I started digging.
Some quick Googleing netted nothing of use. Blarg...
Time to whip out the Chrome Dev tools to see if anything glares back at me...and it did. In the Network tab I noticed that each item requested by the browser (excluding items served elsewhere) all had something in common. They all took 10 seconds and some change to load.
That seemed too precise to be a random issue. Time to round up some suspects.
Firewall? Nope. A quick drop of the firewall and a page refresh answered that. Still taking 10 and some change.
DNS? I don't think so. My resolv.conf is fine and I can dig others and myself without issue.
Apache Configs? I haven't changed these in quite awhile, so if the issue is there, why would it suddenly decide to manifest now? Syntax checks returned OK and I couldn't see anything glaring back at me. So I started stripping items from my vhost conf.
SSL? Commented out. Still slow page loads.
Rewrite rules? Commented out. Still slow page loads.
Network ACLs? Commented out. BOOM!! The lights came on!!
Turns out, it was a combination of DNS and my Apache vhost config file.
Order deny,allow deny from none allow from all
Some of you seasoned SysAdmins may have already figured whats wrong with that code block, but for the rest of us, the offending line is line number 2.
More specifically, line 2 word 3.
none is not a valid Apache directive. Turns out, Apache was trying to do a hostname lookup on a site called
none*. Turns out, that times out after 10 seconds. TURNS OUT, I used 'turns out' a lot just then!
So, with each page load, it checked the ACLs, tried to figure out who
none was so it could deny it, failed after 10 seconds, and moved on to the allow line then returned the item requested.
Hopefully this will help someone in a similar boat. I sure could have used this info 4 hours ago...
Oh...and as to why this issue suddenly manifested when nothing had changed in the configs since inception...your guess is as good as mine. (aka I'm still looking into that)
* NOTE* Apache appears to still do hostname lookups here even if you have
HostnameLookups off in your main config file. (/etc/httpd/conf/httpd.conf in my case)
*NOTE 2: Revenge of the NOTE* Be sure to check for any
.htaccess that might have a line similar to this!
*NOTE 3: The NOTE-a-ning: This time it's personal* ALSO, this can effect
allow lines the same way! If it is not an IP or the word All, it will attempt a hostname lookup.
Ok. I think I found it.
(Some) Interfaces defined in /etc/sysconfig/network-scripts/ had an erroneous DNS defined. My guess is, the request would come in to the VHOST, hit the
deny from none line, do a hostname lookup with the DNS defined on the interface, timeout (since there was no nameserver there), and continue on to my
resolv.conf, get an answer (albeit a not found answer), and load the content.
I've not had a chance to test this hypothesis, but I'm willing to bet therein lies my issue.