Well, it seems that SmarterStats still hasn’t fixed their biggest bug… when you import a bunch of log files, the service will mysteriously stop importing.
For example, I have log files starting from January of 2008. When it imports, sometimes it will get to the middle of June, sometimes the middle of December, and then it just stops importing. To reimport, it deletes all of the processed logs and starts from January 2008 again… it’s a vicious cycle.
I’m trying a solution where I move the log files that it has processed to an archive, and also move the distilled “processed” logs to another directory. When it compiles a new set of stats, I will merge it with the existing stats.
I also changed the settings to not pause. By default, the service pauses every 15 minutes to “give your hard drive a rest”. WTF is this crap? This is supposed to be enterprise-level software. When importing logs, I never see more than 5% CPU usage, and hard drive activity is nothing unusual, either. Why in the hell does it take so long to import logs when resources obviously aren’t being utilized to their fullest potential? Regardless, I set the “timeout” from 15 minutes to 999 minutes, and we’ll see if that resolves the problem.
It’s very tedious. This problem has been around in SmarterStats for years. If you troll their forums, most of the posts are about this mysterious behaviour. Like I said in my previous post, I have a love/hate relationship with SmarterStats. Once the logs are imported, it is very powerful… it’s just getting to the point where everything is imported.
1 Comment
I thought I would elaborate a bit on how SmarterStats works internally and maybe help you out in the process.
First of all, please bear in mind that SmarterStats is an Enterprise-level product, but it also has to run on low-end machines that don't really have any extra horsepower to give. It is also very frequently installed on web servers themselves, so the ability for SmarterStats to run without affecting your web delivery is the most important thing.
That said, there are a lot of settings you can alter to affect the way it imports data. I'll share a few of them.
First, the way SmarterStats works is that it loops through each of your sites and spends a maximum amount of time importing each one. By default, this is 15 minutes for each site. The reason for this is that hosting companies do not want one huge site to slow down the stats of the other 200 small sites they have on the server. After all sites have been processed, it does take a break from importing. The reason is that in the vast majority of cases, and after a site is originally imported, the update frequency does not need to be as high. IIS has a delay in writing logs, and typically importing more than every 15 minutes is overkill. This can be updated, as you found, by changing the reader time slice variable, which affects how long a single site can be imported.
Second, you can change parameters to give SmarterStats more CPU to use. Again, the defaults were chosen so the software would not be impactful on servers it is installed on. Unlike most products that require a server dedicated to stats processing, SmarterStats can run right on the web server.
Information about the fields to edit can be found at http://www.smartertools.com/portal/KB/a220/instru…. Making these changes can ramp stats up to 100% cpu if you want.
Third, make sure you're running the latest version. We release updates every couple of months with performance updates in them. I'm pretty sure you know this already, however.
Lastly, if you haven't talked to our support about the import stopping, it would be good to contact them. In almost all cases where this has happened, it has been related to too little memory or some other issue. And it's pretty much impossible to diagnose without taking a look at the server to see what's happening.
I hope this helps explain why things work the way they do. Please feel free to email our sales department and ask to have the email sent over to me if you need any additional information.