This is way out of date now
The information in this post is so out-dated that I wonder why I'm keeping it around. I guess I'm a digital hoarder...
Recently an issue surfaced with one of my Apache servers. Page loads were suddenly taking WAY too long to load. In some cases, I was seeing over a minute long load times.
After some head scratching and pointless page refreshes to see if maybe it would fix itself, I started digging.
Some quick Googleing netted nothing of use. Blarg...
Time to whip out the Chrome Dev tools to see if anything glares back at me...and it did. In the Network tab I noticed that each item requested by the browser (excluding items served elsewhere) all had something in common. They all took 10 seconds and some change to load.
That seemed too precise to be a random issue. Time to round up some suspects.
Firewall? Nope. A quick drop of the firewall and a page refresh answered that. Still taking 10 and some change.
DNS? I don't think so. My resolv.conf is fine and I can dig others and myself without issue.
Apache Configs? I haven't changed these in quite awhile, so if the issue is there, why would it suddenly decide to manifest now? Syntax checks returned OK and I couldn't see anything glaring back at me. So I started stripping items from my vhost conf.
SSL? Commented out. Still slow page loads.
Rewrite rules? Commented out. Still slow page loads.
Network ACLs? Commented out. BOOM!! The lights came on!!
Turns out, it was a combination of DNS and my Apache vhost config file.
Order deny,allow
deny from none
allow from all
Some of you seasoned SysAdmins may have already figured whats wrong with that code block, but for the rest of us, the offending line is line number 2.
More specifically, line 2 word 3. none
Turns out, none is not a valid Apache directive. Turns out, Apache was
trying to do a hostname lookup on a site called none*. Turns out, that
times out after 10 seconds. TURNS OUT, I used 'turns out' a lot just then!
So, with each page load, it checked the ACLs, tried to figure out who none
was so it could deny it, failed after 10 seconds, and moved on to the allow
line then returned the item requested.
Hopefully this will help someone in a similar boat. I sure could have used this info 4 hours ago...
Oh...and as to why this issue suddenly manifested when nothing had changed in the configs since inception...your guess is as good as mine. (aka I'm still looking into that)
NOTE
Apache appears to still do hostname lookups here even if you have
HostnameLookups off in your main config file. (/etc/httpd/conf/httpd.conf in
my case)
NOTE 2: Revenge of the NOTE
Be sure to check for any .htaccess that
might have a line similar to this!
NOTE 3: The NOTE-a-ning: This time it's personal
ALSO, this can effect allow lines the same way! If it is not an IP or the
word All, it will attempt a hostname lookup.
Eureka!!
Ok. I think I found it.
(Some) Interfaces defined in /etc/sysconfig/network-scripts/ had an erroneous
DNS defined. My guess is, the request would come in to the VHOST, hit the
deny from none line, do a hostname lookup with the DNS defined on the
interface, timeout (since there was no nameserver there), and continue on to my
resolv.conf, get an answer (albeit a not found answer), and load the content.
I've not had a chance to test this hypothesis, but I'm willing to bet therein lies my issue.