Mail archive

[acf] acf-weblog

From: Timo Teräs <>
Date: Fri, 18 Sep 2009 13:15:36 +0300

Hi all ACFers,

I was just upgrading the acf-weblog on my box to the latest.

In the process, I fixed acf-weblog.apk to install weblogimport script directly
to /etc/periodic/hourly so it does not need user involvement to make it work.

I'm running it on my busy squid box now and seems to be working pretty well.
And looks nice!

I have some wish list items, so I'm posting them all here. Thoughts, opinions
and implementation (at somepoint) would be appreciated:

- the "local" source is very useful. might be an idea to create it
  automatically when the database is created. (might be actually done; i
  just tested upgrade from old version.)

- if userid is "-" because authentication was not required, use the source ip
  address as grouping factor. it helps to see where the traffic originated.
  this will be especially useful for me since i will soon have two separate
  subnets: other tracked with authentication, the other with IP / MAC address.
  So grouping primarily on username and if it's not present group by IP.

- if userid is "domain\theuser" it breaks. i see "domain heuser" and i don't
  get the matching lines in "search" page. unless i fix the backspace on
  search field. so escaping is broke somewhere. atleast in "audit" page's
  link to "search" page.

- might be an idea if you did not delete the squid access log. instead let
  logrotate delete them. and just mark internally that it has been imported.
  this is because there are other scripts/tools to analyze squid logs and
  they print you statics of bandwidth usage / requests done, how many hits
  vs. misses. type of traffic. graphs based on time of day, etc...
  I've personally used scalar.awk for this to get rough ascii stuff out.

- even better if on "usage" page you counted "bytes/requests in/out" +
  proxy "hit" ratio.

- it is possible that the pruning of old data is not working as expected.
  in "usage" page i see two months old stuff. with config being:
  "days to keep history: 14", "days before purge: 30"

- consider using prepared statements and transactions. those would speed up
  importing process. It took almost 20mins to import the big big log file
  I had.


Received on Fri Sep 18 2009 - 13:15:36 UTC