The first thing you should know is that I had already been working professionally with Linux for ~10 years before I began studying for the exam. I doubt that so much experience is really necessary, but because the exam measures actual competency on live systems rather than your ability to memorize or “read between the lines,” I think it’s highly unlikely that someone with little or no working knowledge of Linux could successfully cram for this test. So I’d have to say that experience is an important prerequisite, but I’d also have to say that experience alone is almost certainly not enough. Considering the wide range of subject matter, it’s very likely that you’ll be tested on something you’ve never had to touch before.

In my opinion, the hardest thing about self-studying for the RHCE exam is knowing how much you need to know about each study point in the RHCE Prep Guide. In other words, when it says something like “you need to know basic configuration of x,” it’s hard to know exactly what “basic configuration” means. Although I’ve signed an NDA preventing me from revealing any details about the exam itself, I can tell you that nearly all of my study material came from two places:

1. RHCE Red Hat Certified Engineer Linux Study Guide by Michael Jang
2. Red Hat Deployment Guide

The rest of my study material came from search engines. I spent a lot of time comparing other people’s study notes to mine and reading anecdotes about the RHCE exam. While the result of this was usually a temporary blow to my confidence (there are a lot of horror stories out there!), I did find a few blogs and forum posts with some helpful information. Usually it was just an alternative way of doing something, but it was also nice to come across the occasional words of encouragement from a self-studied RHCE. Just knowing that other people had done it gave me a slight confidence boost (as I hope this post does for some of you).

The first thing I did was read Michael Jang’s book cover to cover, just trying to absorb what I considered to be the most important information from each chapter (i.e. concepts rather than commands). This took me a week or two. When I was done, I set up a test environment using VirtualBox. Then I decided it would help to have a condensed, single-page study guide to refer to, so I created a “wikified” version of the RHCE Prep Guide which I called my RHCE “cheat sheet”. For the next few weeks, I went through each study point on the “cheat sheet” and used my study material to test and document everything I thought I might need to know.

I found Michael Jang’s book to be a great study guide (the labs and example problems were a huge help), but I wasn’t completely happy with the amount of detail on some topics, as well as how it tends to follow the Red Hat course outlines rather than just sticking to the RHCE Prep Guide. I also found quite a few typos and just plain incorrect information, so that’s where the Red Hat Deployment Guide came in. While I was working on the RHCE “cheat sheet,” I would usually read the appropriate chapter(s) from Michael Jang’s book first, then I would supplement it with the appropriate chapter(s) from the Red Hat Deployment Guide. If I felt particularly weak in a certain area, I would also peruse the man pages and any HOWTOs I could find online. This was a long, arduous process, but it helped ensure that I wasn’t missing any important details.

For the last couple weeks, I tested myself by putting the “cheat sheet” away, doing a minimal install of CentOS in my test environment, and trying to configure everything I could without referring to any documentation whatsoever. Since the test takes place on a live system, I assumed from the beginning that the man pages would be available, but in order to save time, I wanted to make sure I could do everything off the top of my head. As a result, I ended up memorizing almost everything on the “cheat sheet,” which is probably why I was able to complete the exam so quickly.

Since I didn’t have a study partner, preparing for the troubleshooting section of the exam was a bit of a challenge. I did come across an interesting project called Trouble Maker which directly addresses this problem, but unfortunately, it has not been updated in several years and does not work on recent versions of CentOS (Update 2010-08-25: It appears this is no longer the case!). For a while, I actually considered writing my own trouble maker program, but I ultimately decided that this would be too much work. Luckily, I have a few friends who know enough about Linux to make a machine unbootable, so we made a game of it. I would give them my root password and challenge them to do something that would keep me from using my computer, then I would try to fix it as fast as I could.

When it was all said and done, I spent roughly six weeks (studying a few hours each day) to prepare for the RHCE. Considering how easy the exam was for me, I believe that I worked a lot harder than I needed to, but the results were clearly well worth the effort. The best advice I can give to prospective RHCEs is to take your time and practice until you can do everything in the RHCE Prep Guide off the top of your head. If you feel weak in anything, do yourself a favor and postpone the exam.

Even though I can do about 95% of the stuff on the RHCE Prep Guide off the top of my head, the horror stories I’ve been reading about this test are making me pretty nervous. Any tips from current RHCEs would be appreciated. Also, if anyone is interested in logging into my test machine at home and doing something to make it unbootable, that would help me practice for the troubleshooting part of the exam.

I’m scheduled to take the test in Philadelphia on September 4th, which coincidentally, is my birthday. I guess I won’t know if that’s a good thing or a bad thing until I get the test results.

First, a few tips:

• Clearly, it’s in your best interest to learn as much as you can about as many different technologies as you can get your hands on. However, don’t ever expect to know everything about anything. Definitely don’t waste your time memorizing things like command line options and function call parameters. That’s what reference manuals and Google are for. It’s infinitely more valuable to have a good understanding of concepts.
• This is a bit of a continuation of the previous point, but you need to realize that what you know off the top of your head is a lot less important than knowing where to find the answers. Read the manual, learn how to use  Google and IRC, and as frustrating as it may be, only ask for help as a last resort after you’ve done your own research and tried everything else. This will teach you how to teach yourself.
• Get in the habit of asking yourself “how would I do this if I had to do it 10000 times?” Computers are very good at performing repetitive tasks very quickly, but human beings are slow and error-prone. The sooner you learn how to leverage the computer to do your work for you, the better off you’ll be. Be lazy by eliminating as much manual work as possible. Your mantra should be “how can I automate this?”
• Never settle for “good enough.” Be a perfectionist, and strive to do everything the correct way according to best practices and/or accepted conventions. If you find that you’re doing something wildly different than everyone else, you’re probably doing it wrong, and in the real world, it will almost certainly come back to bite you or your successor(s) in the ass later. You might also end up reading about yourself on a site like The Daily WTF.
• Remember that the best solution is always the simplest and most elegant one. Avoid creating Rube Goldberg machines. Remember this quote by Antoine de Saint-Exupery: “A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.”

And now the fun stuff:

1. We’re going to start by building a network. In the unlikely event that you haven’t done so already, get yourself a broadband Internet connection and a cheap Linksys firewall/router. Make sure your ISP allows you to host a server (I strongly recommend DSL over cable for this reason alone). Contrary to popular belief, a static IP is not necessary, nor is lots of bandwidth.
2. Build your own server from scratch. It doesn’t have to be anything really expensive or powerful, but it should be dedicated (i.e. a different physical machine from your primary desktop) and capable of running 24/7 for the foreseeable future. Don’t waste your money on an expensive video card. Use that money to buy a cheap interruptible power supply and/or a tape drive instead.
3. Install Linux on your server. I don’t care what distribution you use, but you’ll find a lot more documentation for the more popular ones (I use Ubuntu server). Do the most minimal install for your distribution of choice (i.e. no optional packages). Do not install a GUI (or control panel) on your server under any circumstances.
4. Once your server is up and running, you are not allowed to reboot it unless it’s to swap out hardware or to install a new kernel. Your goal should be to have the longest uptime possible (for reference, my home server’s record is somewhere around 450 days). When something breaks, figure out what went wrong and fix it. Don’t just reboot and hope the problem will go away, because that will be a waste of an opportunity to learn something.
5. Without a GUI, you’ll need to do everything on the command line. Get familiar with Linux’s filesystem hierarchy. Learn about what files go where and why.
6. Register a domain name, but do not purchase off-site email/web hosting. From now on, you’re not going to pay anyone else for anything you’re capable of hosting yourself.
7. Make your domain resolve to your home Internet connection. If you don’t have a static IP address, get an account with a dynamic DNS provider and learn how to make this work.
8. Install Apache and start a blog. You’ll use it to document things as you learn them. Learn how to open the necessary ports in your firewall to make your website available to the whole world. Does this scare you a bit? It should. Your server is now wide open to attack. Start learning how to secure your server. Be paranoid.
9. Install SMTP, POP3 and IMAP servers, then tell everyone about your new email address. If you have other outside email accounts, forward them all to your new address and use it exclusively. Set up server-side spam filtering and learn how to defend yourself from spammers.
10. If you haven’t already, get a cheap tape drive and learn how to back up and restore your files. Then learn how to take automated nightly backups of your critical data.
11. Now that you’re your own webhost, give your friends accounts on your server and host their websites/email too (the more the better). Learn how to set quotas and secure your server on the inside so that your users can’t break it on you. Then test yourself by challenging them to break it on you.
12. Turn off the crappy DHCP server on your router and install something more powerful (e.g. ISC-DHCP).
13. Install BIND and use it in place of your ISP’s DNS servers. Learn how to edit zone files by configuring an internal DNS namespace and putting every device on your network into DNS. Don’t forget to create a reverse lookup zone. Learn how to use the dig command.
14. Combine your knowledge of DHCP and DNS to set up dynamic DNS updates.
15. Now that you know your way around a Linux box, back up your home directories and set everything back up again on a fresh install of Gentoo Linux.
16. Learn a popular web-programing language (e.g. PHP, Python, Ruby) and build your own website/blog from scratch. For bonus points, start by creating your own MVC framework.

That aught to keep you busy for a few years! And once you’re done, you’ll have more working knowledge than 99% of the people out there applying for entry-level IT jobs.

Step 1: Describe the problem

This is the easy part. Find out what the problem is and reproduce it. That 2nd part is important, because the typical user can’t always be trusted to know what they’re doing. By reproducing the problem, you can rule out user error and verify that there actually is a problem.

Step 2: Gather and analyze data

This is the part everyone likes to skip, and is the real subject of this rant. Step two requires direct observations in order to find out exactly what is happening. How you do this is very much dependent on the problem at hand, but it involves things like log and packet analysis (which may very well require a 3rd party tool not included within the base OS). In any case, please don’t just take a wild guess about what’s causing the problem and then proceed down the path of random experimentation. I don’t know for sure how people acquire this bad habit, but I have a strong suspicion that it comes from working in Microsoft Land, where the computers have personalities, rebooting is the inexplicable fix for everything, and a sea of GUIs makes it easy for the novice sysadmin to miss what’s really going on.

Part of problem in Microsoft Land is that proper troubleshooting tends to be a lot more difficult than it needs to be. For example, there is a severe lack of useful diagnostic tools included within the Windows OS itself. Why are the Windows support tools, resource kit tools, and IIS diagnostic tools still separate downloads? The same question can be asked about the Sysinternals Suite (which Microsoft has owned for several years now). Why are they still shipping obsolete utilities instead of their newer replacements (e.g. nslookup, which was obsoleted by dig many, many years ago)? And lastly, why does Microsoft constantly try to hide any information that could be useful for troubleshooting? Anyone who has ever had to view the message headers on an email in Outlook knows exactly what I’m talking about here, but I digress.

Step 3: Form a hypothesis

It isn’t until you figure out what‘s happening that you can address the question of why it’s happening. Step three is where you use the information you gathered in step two to determine a logical course of action. Remember that a hypothesis beginning with “maybe” or “I think” with little or no direct evidence to back it up is often a dead giveaway for someone who doesn’t know what the hell they’re talking about.

Step 4: Test your hypothesis

Perform your planned course of action.

Step 5: Analyze results and draw conclusions

Check to see if the issue is resolved. If not, revert your changes and go back to step three. When drawing conclusions, ask yourself how this problem occurred in the first place. Was your most recent fix permanent or just a temporary band-aid? If the fix was temporary, make sure you schedule a time to implement a permanent fix.

For the record, I am not just an elitist who advocates doing everything manually through a command line.  I understand that a wizard can help get you up and running quickly, and I think any wizard that tells you all the things it did would be a great learning tool.  However, I have yet to encounter a wizard that tells you much (if anything) about what it’s doing, and nobody is going to convince me that speed of implementation is more important than knowing how to configure something so that you can fix it when it breaks.

The bottom line is that if you feel the need to use the wizard (especially for critical security infrastructure like firewalls), then you have no business using it, because you obviously don’t know what you’re doing.