#programming

I would like to find a method for sending alert email if my site starts to get new kind of traffic from other sites. Is there tools or solutions for this?

I think it could work like this:
Application would be an apache plugin. It looks referer-information of all http requests. If it notices several new referer sites/urls in short period of time it sends alert email to specific address.

Google Analytics won't send email to my knowledge, but otherwise it's pretty nice. As for noticing when you're tweeted, you could try scraping e.g. http://twitter.com/public_timeline, but you'd have to do that several times a minute to catch everything. Google Alerts is another useful tool for catching all kinds of mentions, I have alerts for some site URLs and my name.

This time I need instant solution when "1-day-old" is not good enough. Twitter was just an example. Trigger can be from any blog, news, etc. so I believe using information that is already fed into my server is the key.

I would expect this kind of filter to be only few hundred lines of code (just in-memory logic) but I believe somebody has to have build this already.

Nagios can watch your loads and response times and send alerts based on them. The good/bad part is that it will alert on any trouble.

Anyways there is no need to make apache module or even something like cronolog, just a static log analyzer run every 5mins or so checking the last 1k or so lines of log should be enough.

I don't know if there's any out of the box solutions, yet. If not apache level, then with PHP/Python, whatever this could be made quite lightweight by putting referers to memcache, or something persistent. Whenever some referer count raises above alert level, some callback function is called. This light snippet could be included to the site's code.

@rambo Maybe daemon for tailing apache log and running some scripts against that feed?
@tepheikk I don't like idea of any additional code to site's code. This should be easy to install solution for any webserver without touching pages itself.

@salainen Any other ideas of solving this?

AwStats does all kinds of analysis on apache logs - quite easy to use and configurable, for what I remember http://awstats.sourceforge.net/ e.g. domains where accessing etc.

http://www.pingdom.com/ offers one free site monitoring

@lle cronolog is basically a daemon for apache loggin. I don't recall the details but basically you can direct apache log "file" to an executable, whether it's just stdin or if there is some sort of socket protocol I don't know since I never bothered to check out the details.

In any case it's a bit overkill to use a daemon process, and if you use one you will need to make sure it doesn't randomly crash or leak memory or something, this is why I suggested a simple peridiocally run job.

awstats or other log analyzers don't really give you alerts, the performance monitoring systems are reactive at best.

Looking at referres in the log and if you start seeing increasing amounts of the same referrer (or similar patterns) allows one to take action before the server is on it's knees.

Thanks @rambo for your ideas, I'll take a look at cronolog...

@ile I think apache access log is the solution here. How you monitor it is a matter of finesse. The simple solution you should try first is just:

while sleep 60; do cat header.part; awk '{print $11}' access.log|sort|uniq -c|sort -rn|grep -v "\"-\""|awk '{print "<tr><td>"$1"</td><td>"$2"</td></tr>"}'; cat footer.part; done > referers.html

(where header.part and footer.part are html fragments to go around the referers table)

Then you have already a website that shows your top referers.

;-)

@salainen That's so kwl! IOU ;-)

@ile No problem. By the way, reading it now I see a flaw. Can you spot it?

The fixed version is below (I think):

while sleep 60; do (cat header.part; awk '{print $11}' access.log|sort|uniq -c|sort -rn|grep -v "\"-\""|awk '{print "<tr><td>"$1"</td><td>"$2"</td></tr>"}'; cat footer.part) > referers.html; done

@ile did you try it out already?

@salainen Yes, I did. First impression: This is a way to go.
Currently we were not logging refered but it's easy to change.

I found this:
> Multiple Access Logs
>
> Multiple access logs can be created simply by specifying multiple CustomLog directives in the
> configuration file. For example, the following directives will create three access logs. The first
> contains the basic CLF information, while the second and third contain referer and browser
> information. The last two CustomLog lines show how to mimic the effects of the ReferLog
> andAgentLog directives.
> LogFormat "%h %l %u %t \"%r\" %>s %b" common
> CustomLog logs/access_log common
> CustomLog logs/referer_log "%{Referer}i -> %U"
> CustomLog logs/agent_log "%{User-agent}i"
> This example also shows that it is not necessary to define a nickname with the LogFormat
> directive. Instead, the log format can be specified directly in the CustomLog directive.

Maybe I'll configure apache to write referers to another file. It seems your cool pipe-chain doesn't take too much cpu/memory.

@salainen Lupasit IRL ratkaista tämän 5 eurolla. Ratkaisit, voisinko saada laskun IRL? Onks siinä ALV mukana vai tuleeks se päälle?

@ile Kyllä siihen ALVi pitää pistää päälle. ;-)