I voted last night.

It was the first time I can say that I voted BEFORE election day.   I'm not going to be around on election day this year, so I arranged for an absentee ballot.  It was SO nice!  I didn't have to wait in lines or worry about getting away from work for an hour to cast my vote.  I could also research information about the issues on the ballot with my ballot right in front of me.

It's not a big election.  There are only two issues in my particular precinct:  the issue of school vouchers and three vacancies in my local city council.  Yet I feel that it is still important to be part of the process regardless of the number or the importance of the issues on the ballot.

Now it's your turn.  Vote on November 6th!

Douglas Adams understands Software Engineering.  In a quote in this post he explained software engineering pretty well.

But here's another quote that explains software engineers pretty well:

I am rarely happier than when spending entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand. -- Douglas Adams

Yep.  He knows.

I'm writing this post from a new computer at work.

One of the things I work on is a Software Development Kit for Linux, UNIX and Mac. From that and the title, you can probably guess that the new computer is a Mac. It's an iMac to be specific.

I've always been of the opinion that Mac computers are good for something. They make great doorstops. The last Apple computer that I used was probably the Apple ][e that my dad bought when I was in Junior High. So to say that I've honestly given them a chance wouldn't be stretching the truth. It would be an outright lie.

But lately, Apple as done a couple of things that have piqued my interest:

1) They put Intel processors in them. Whether or not that is a GOOD thing is a religious issue. But I'm more familiar with Intel processors, so it's something that is a little more familiar to me.
2) They released OS X. For most people, it's just a pretty new front end, but for people like me that are doing development on Mac and that are used to doing development on other operating systems, this is huge. The non-pretty back end behind the scenes is a variation of UNIX developed by Berkeley Systems. So if I can do development on any UNIX or Linux system, the process is very similar, if not often identical, in OS X.

The bottom line: I believe that this shiny new iMac on my desk is 85% eye candy and 15% computer. But that's a 14.997% increase over the previous iMacs, as far as I'm concerned.

The lag is really gone.

REALLY.

The site has been up and running without a reboot for more than a week now.  Swap in/swap out statistics look perfect:  0 and 0 respectively.

*whew!*

The settings I tweaked for the database server? They were only part of the problem. They were just the beginning.

My VPS has only 128 megabytes of dedicated memory. No, that's not a lot of memory, but for a simple web server, it's sufficient. At least, it's sufficient IF some settings are right.

Unless you want to learn WAY more than you ever wanted to about memory management on Linux systems, you'd probably better stop reading now. This is going to get WAY beyond geeky here.

No. Seriously. If you're not interested in minute technical details, you'd better stop reading or your head just might explode.

One of the nice things about a Virtual Private Server that you don't get with most shared hosts is a command prompt with root access. With this type of command prompt, you have the ability to troubleshoot things like memory issues.

The first thing that I checked was /var/log/messages. I found regular messages in this log that said things like:

Out of memory: Killed process 7716 (mysqld)

Out of memory: Killed process 7725 (httpd)

This made it pretty clear that it really WAS a memory issue and that the programs suffering most from lack of memory were my database server (MySql) and my web server (Apache). Since these two things are, to say the least, critical for web sites (they're the core of most web sites, especially sites with blogs), I needed to figure out how to keep these servers from being killed. These servers were asking for more memory than existed, so they were being stopped by the machine.

My next step was to look at the memory usage on the machine. I saw that the total memory available was barely higher than the amount of memory in use. That worried me at first until I did some research and found that this situation is very normal with Linux systems that have limited memory. The next thing I looked at was the memory buffers used. This shows how much memory is actually being used by applications. The absolute maximum amount of memory available is the amount of physical memory plus the amount of "swap space" (which I'll explain in a moment). The disturbing thing about the memory buffers was that it appeared dangerously close to the absolute maximum amount of memory available.

So... How to fix this? I needed to find out which applications were hogging memory and I needed to figure out how to make them stop being such bloated pigs.

I found out that there is a way to display what percentage of total memory was used by each process. From that, I found out that the worst offender was the Apache Web Server and another big problem child was Spam Assassin. Apache will create multiple servers (to serve up multiple simultaneous requests), each of which can consume 15% of the server's memory. Apache alone was eating up about 60% of the memory available and Spam Assassin had multiple copies running, each of which was using about 7% of the memory. Add to that a MySql database server that was using about 15% of the available memory, and guess what. I was OUT of memory.

So what does the system do when it runs out of memory? It uses swap space.

Swap space is simply disk space that is used as memory by the system. When running a program a computer will first check and see if the needed part of the program is already in memory. If it is, it will use the program in memory. If not, it then checks to see if it's in the "swap space" on disk. If it's there, it has to copy it from the disk to memory. If there's not enough free memory to copy it into memory, it first has to copy something from memory into the swap space. Thus, it swaps different applications back and forth between system memory and disk. Why is this a problem? Memory access = usually very very fast. Disk access = usually very, very, very slow in comparison. Access to memory is measured in nanoseconds, or BILLIONTHS of seconds. Access to disk is measured in milliseconds, or THOUSANDTHS of seconds. So realistically, memory access is a million times faster. If memory could run the hundred yard dash in 10 seconds, a disk could run it in just under 116 days.

The other problem is that the swapping of memory blocks from memory to disk takes time and processing power. So as more swapping takes place, the entire system slows down.

So I checked to find out how much swapping was taking place.

The Swap In and Swap Out numbers were in the mid-hundreds. This meant that my server was doing way way WAY too much swapping. Good numbers would be close to zero. The less swapping, the more that memory is being accessed instead of disks and the faster the system will run.

So I needed to find out how to make the memory-hogging programs not hog as much memory. A little research later and I found out that I needed to:

1. Limit the amount of memory that MySQL was using by adding 3 or 4 lines to a configuration file.
2. Limit the number of web servers by changing about 3 lines in a configuration file.
3. Limit the number of spam assassin servers by changing a line in a couple of configuration files.

So by making a few small changes, I could solve this problem. I made the changes and the results are great! Apache is using about half of the memory it used to. Spam assassin is using less than a third of the memory it used to. MySql is using less than half of the memory it used to. These three changes dropped my swap in/swap out numbers from the mid hundreds each to 1 and 2 (at this moment) respectively.

The bottom line: You probably didn't have to wait forever to read this long, way-too-technical, very boring post. I learned a lot. I'll be able to fix problems like this in the future. And I didn't have to pay more to add memory to my server. All of those things are GOOD things!

Last night, I received a phone call that turned out to be rather enjoyable for me.

The caller identified himself as a representative of our current mortgage company.  They're calling people and making sure that their mortgage(s) are meeting their needs.  They want to make sure that we're in the "best position possible."

Translation:  "Are there any other services that we can sell you."

The phone call was pretty short.  He first asked if I was still comfortable with my mortgage payment.  I told him to look through my payment history and that would be self-evident.  He saw that we're paying extra principal each month and spontaneously reacted with "Whoa!  You're really trying to knock this one out, aren't you?"  After a rather long pause (where he was probably reading telemarketing scripts and looking at our information), he finally said, "Well, unless you're going to do major renovations or need us to increase your Home Equity Line of Credit, I don't think there's anything else we can really do for you."

I suggested that he might lower our interest rate, but he couldn't do that.

It really seemed to surprise him that there really wasn't anything for him to sell to us.  He said it was very rare that he had such a short phone call with absolutely no options.  I like being in that position.

Oy...

My site is set up on a VPS: a Virtual Private Server.  It basically looks like it's own separate, private machine with nobody else in the world using it.  It's not, because that's a lot more than I can afford, but nonetheless, it's better than the normal run-of-the-mill shared hosting that I used to use.  I can install software and I can do pretty much whatever I want... within reason.  I like the VPS idea and it's affordable.

Lately, if you've read my blog (or anything else hosted on my VPS), you've probably noticed a considerable lag.  It's been getting progressively worse.

I'm not exactly sure what it is, but I don't believe that it's due to the site being overloaded.  Few people regularly visit anything hosted here.

Last night, the whole thing came to a screeching halt.  It was so bad that I couldn't even log into the server to reboot it.  I put in a help ticket with the host and they were able to get the site back up and running.  But this is ridiculous.

I looked up some information on streamlining memory usage, especially with the database server that this site uses.  I tweaked some configuration settings and hopefully, I won't have the same problem with the ever-slowing-to-a-grinding-halt problem that I've been seeing lately.

If so, I guess I'll have to look into the issue more closely and more carefully.  The main problem is that when the site gets really bad, it has to be rebooted, which nukes all of the information that might tell me who the memory hog really is.   Maybe I can write a script that would regularly log such information to a file I can retrieve later.

But hopefully, it's fixed and you can waste less time waiting for my pages to load.

I've been playing a game on my iPod called "Zuma." It's hard to explain, but suffice it to say that it's a fun diversion. It's sort of a mix between a puzzle and an action game.

Someone needs to help them with their translations though... After each "temple" in the game (which consists of 3 levels), there's a bit of text. Here's a transcript:

Temple 1:

Interloper of puny! You suppose the secret of Zuma it is possible to take the knowing so easily? No! There is defect for your thinking! Three mystery shrines protect method! Never you will strike past them!

Temple 2:

I see what?!? You have lived still. Is it possible to be the one of which the ancient prophecy speaks? It was sent in order perhaps to release me from my wicked capture person finally. But no namely this was the story of the exactly old wive. You want to live, now retreat!

Temple 3:

As for me it is not possible to believe! Your power namely that is large! But there is one more temple which you do not find, that the Zuma deeply buried under the land. It was hidden, it is the temple of secret! You the final temple must fight in order to take the cover of position of my jail!!

Temple 4:

The final temple of Zuma was struck! Your extreme power is not possible to be defeated! As for me it can taste the taste whose almost freedom is sweet sweetly. But god of the sun can be imprisoned at only that true house.

We come from the star, return to the star! It is of you to rescue of my star.

Temple 5:

Excellent work, mighty frog one! You obtained your lawful place of the stars, at the side of the ancient ruler of the Zuma. In the future, as for me, you are known as our sibling. I will call you brother, and we control the outer space together! It is joyous news! Now we are to the dance!

It's actually pretty hilarious. I'm assuming it's a very direct translation from Japanese.

DELICIOUS!

It was a weekend of a little bit of experimentation and a lot of good food.  Yes, I'm talking dutch oven cooking again.

Ever since my younger son had dutch oven pizza at cub scouts, I've been wanting to try it out.  On Saturday, I gave it a shot.  I made the pizza dough and after it rose, I split it in half and spread it into the bottom of two dutch ovens.  I added homemade pizza sauce, cooked italian sausage and lots of cheese.  After baking the pizzas for about a half hour, they came out deliciously.  The crust was really puffy and wasn't as crispy as when we cook them on the pizza stone in the oven, but it was VERY delicious.

Then, on Sunday, I made pulled pork sandwiches and baked beans in my dutch ovens.  I put the pork loin in the oven and cooked it at about 250 degrees from 10:30 in the morning until about 4:45 in the afternoon.  It was EASY to pull apart with forks.  I added a little more barbecue sauce and piled it on toasted buns with cole slaw, and pickles.  The baked beans had pork & beans (of course) with bacon, garbanzo beans, lima beans, and kidney beans.  Added to that concoction was brown sugar, white sugar, dry mustard, and some other spices.

*drool*

Did I mention that my weekend was delicious?

To the person that followed me most of the way to work this morning...:

Thank you for following me so closely. If you'd just call me and tell me the results, I'll tell my physician that he can forego certain unpleasant parts of my next physical examination.

<expletive deleted>