Every once in a while, I like to post on topics which are more accessible to a wider audience. Often times however (which will probably become even more often in 2014), I like to really just nerd out about things I think are cool to talk about. Unfortunately, this tends to make for less accessible posts. I would apologize for that, but whatever. It’s my blog, and I’m not sorry. The point is, this is one such post. If you aren’t either a Linux user or a morbidly curious individual, then this isn’t the post you’re looking for. Move along.

For the last two and a half years in which I’ve been a Linux user, two of those have been spent quite happily in the (not so) tender care of Arch Linux. Arch is a truly fantastic Linux distribution for a whole host of reasons which could merit an entire blog post of their own. Since we don’t have time for two blog posts though, I’ll just leave it at “Arch is awesome”.

It’s actually because I took such an early liking to Arch Linux that I was driven to try Gentoo for the first time. Gentoo and Arch share many similarities in both philosophy and design which I found myself liking in my Arch expericne. However, despite the commonalities, my first attempt at Gentoo convinced me that I absolutely hated it. All I could chalk it up to was linux for masochists, and every configuration I performed felt like a run through suicide linux.

Yet today, I write to you from the desktop of my fresh new Gentoo Linux installation, which now serves as my primary operating system. In fact, not only is it my primary operating system, but I now vouch for Gentoo as being one of the finest Linux distributions in existence, both for a general purpose computer and for a highly specialized machine with a specialized task.

So how did such a drastic 180 overtake me? The only way Gentoo can: in stages.

Stage1: Things I always liked that Gentoo and Arch share in common

These are the things which lead me to Gentoo in the first place, and which kept me coming back to give it second, third, fourth, and beyond chances.

• Rolling Release Model: Let me be perfectly clear. I don’t like static release distributions. For starters, upgrading a static release tends to mean “Perform a clean install of the new release version, migrate your data to the fresh install”. If that’s not enough of a pain in the ass for you, the speed with which your software become dated might be slightly more off-putting (Or if you’re into Debian, your newest release starts with the software already well out of date). For some reason, this is something SysAdmins have an enormous nerd crush on. They love this obsolescence so much, that they’ve even anointed a worshipful nickname for it: stability. Translated into plain English “stable” really just means static (well tested, but static none the less), which is how software goes out of date.

• The DIY Approach: Both of these systems are really about putting choice and power (and by extension, responsibility) in the hands of the user. There are no automatic installers. There are no desktop environments by default. A finished base installation leaves you with a command line and a blinking prompt. This is not newbie friendly stuff, but once you embrace it, it gives you a powerful feeling of ownership over your machine.Going through the manual process yields you a setup which is perfectly tailored to your needs and has what you need in order to do what it is that you do, and not a bit more.

• Top Notch Documentation: Arch is widely renowned for its incredible wiki. The excellent documentation makes the highly manual nature of the distro accessible to otherwise intimidated or relatively new users. Gentoo’s documentation is no slouch itself, being in some respects more complete than even Arch’s documentation. While the Gentoo Wiki may not cover as many “How to” type tutorials as the Arch wiki does, the Gentoo Handbook provides far more complete coverage of Linux fundamentals than anything in the Arch Wiki. In any case, both of these distros have great documentation. Their docs make solid sources for referencing general Linux material, and are not just specific to their own distribution.

Stage2: Things I used to hate about Gentoo that I grew to love

Eventually, Gentoo found a place in my Linux ecosystem as a server OS. Servers tend to require much simpler application stacks than desktops, and are more likely to leverage ebuilds which have shorter compile times (No X11, KDE, or GNOME here). This actually made Gentoo a reaosnable choice for my servers and helped mitigate some of the pain I had been scarred with in previous experiences.In actually taking some time getting to know Gentoo, I found myself growing to love many of the features which I previously believed had no purpose other than to cause anguish and frustration.

Compile Everything!

I came into Gentoo as a firm believer in binary package distribution. Like many users before me, I was utterly befuddled as to why anyone would bother compiling every piece of software they add to their system. It takes extra time, extra CPU cycles, more I/O stress, and can occasionally result in build failures (which 4 times out of 5, can only be resolved upstream). All for what? Actually, it turns out it’s all for a couple of pretty good reasons

1. Options galore: Those who have stuck to Binary packages probably don’t realize that most software comes with a complement of options which can be enabled or disabled at compile time. Often times, this can be used to prune external dependencies by disabling features which are not needed on the particular machine the software is being compiled on. Or vice versa, specialized features which are good in specific use cases can be enabled at compile time. In binary distributions, these decisions are made for you. Source distributions put this power back in your hands.
2. Architectures abound: The fact that Gentoo is a source based distribution means it can be deployed to a hugely diverse array of CPU architectures with relative ease. As long as the architecture type has a working C compiler, you have most of what you need to get up and running with Gentoo. I take advantage of this by rolling Gentoo on my Raspberry Pis, where Arch would require too much babysitting (by which i mean constant updates) and Debian is… well, old and crusty. For PPC architectures, this means Gentoo is probably the only OS available on your platform which can run current software.

Emerge

emerge is the command line tool used to manage software from Gentoo’s portage tree. My first experiences with it were (wait for it) decidedly negative. It’s slow. Half the time I want to install a package it complains with cryptic messages about USE or keyword changes. Then when I let it have its way with these changes, it asks me to run dispatch-conf. What then hell is dispatch-conf, and why isn’t my package manager just doing this for me? It actually turns out that gripes like these were really just carryover from my binary package management mentality. While emerge is admittedly slower than other package managers, much power is gained in exchange for the speed sacrifice.The power of emerge is exemplified in the quality that it is the most intelligent dependency calculator in all of Linux package management.In situations which it cannot automatically resolve dependency conflicts, it gives precise directions on how the administrator can manually do this. emerge can handle walking a 5+ year old system through an upgrade path to becoming current again (albeit not quickly) without the need to build a new system from scratch. Any issues I had with confusion were cleared up with a thorough read through of the Portage documentation, which helped me realize that this tool was not what I thought it was. Error messages are actually quite helpful, telling me exactly why emerge fails, and what I can do to resolve the failure. The dispatch-conf tool is actually my friend, preventing emerge from clobbering my config files. USE flags are an indispensable tool, allowing me to pick out exactly how I want my packages configured without ever having to touch a configure script. Indeed, Gentoo’s software management tools which I started out with so much contempt for have actually grown to become the feature I find most compelling about it.

Stage 3: Things which Gentoo gives me that Arch can’t

After spending some time with Gentoo on my servers, I began to realize that it’s really quite an elegant distribution, with a deep complement of outstanding tools. While learning to use these tools, I started to happen across some nice features which really add a lot to the rolling distribution experience, which you don’t find anywhere else.

Portage

Gentoo’s portage is one of the finest software management systems in existence, both in conception and implementation. The reasons why could be detailed entirely in their own blog post, as portage is an incredibly feature rich system, so I’ll keep it to a few highlights.

1. USE Flags: USE flags are keywords which represent common compile time options. For example, if the sound USE flag is enabled for an application, the application will be compiled with its sound features enabled. Or if the X USE flag is enabled, the application will be compiled with support for X11 based graphical features. If you’re running a server though, and you don’t really have need for either, you can set the flags as -X -sound, and you won’t pull in any X11 or sound related dependencies. Pretty nifty if you don’t want to pull in unneeded dependencies or want to add features which aren’t typically distributed as part of binaries.
2. User patches: Most modern ebuilds leverage the epatch_user function, which allows users to specify their own patches to apply to the source code of software in the portage tree prior to compiling it. So if you found a cool patch on GitHub to add functionality you need to one of your apps, you can integrate that patch into the regular build process of Portage, rather than having to download a separate source version of your application and manage it outside of your main software repository.
3. Multiple Versioning: Most packages in the portage tree will actually have multiple versions available which you can choose from to install. For example, if I wanted to install Apache, I have two minor versions to choose from (2.2 and 2.4) and each of those has three revisions to choose from. Although admittedly, desktop applications tend to have one or two versions available in the portage tree at a time (due in large part to the fact that only one or two version of a DE can be reasonably maintained in the portage tree at a time), server applications tend to have 3 or more versions available at the same time.
4. Slots: Usually, if you install one version of an application, it will replace whatever version previously existed. However, in the portage tree, every version of a package can be assigned a number called a “slot”. Versions of a package which occupy the same slot will replace each other when being installed, while package versions with differing slots can be installed simultaneously. Don’t even ask me how the hell the Gentoo devs pulled off this sourcery; the implementation of this feature is a total mystery to me (if you know better than me, I would love an educational comment). This allows maintaining legacy versions of packages on your system alongside newer versions which break interface compatibility.
5. And so much more: I could literally spend an entire day writing about all the things I love about portage which are unique to a ports-like system. Almost as easily as you could fall asleep reading that laundry list. So instead, I’ll settle fo just saying there are tons of other great features which you’ll pick up on if you ever choose to explore this frontier for yourself.

Stability

At the end of the day, this is the number one reason I ditched Arch for Gentoo. Even though I made fun of all the SysAdmins who believe in “stability” as the holy grail of server administration, there is some value to being able to stabilize software versions on the server (and on my desktop, where I have grown incredibly lazy about updating).I may like having recent releases of Tomcat, PHP, or uWSGI, but it might not be as pressing for me to have the latest release of glibc or X11.Under Arch, you could upgrade packages selectively, leaving your base system in tact, but a quick peek at the Arch upgrade path documentation provides a clear pictue of how often “breaking changes” occur. Upgrading a stabilized system through several iterations of these kinds of changes can be a nightmare at best, and totally break a system at worst. Some of Arch’s most seasoned users will be amongst those to tell you that you may as well go elsewhere if you want a static system. Portage on the other hand, has a “stable” branch which upgrades infrequently, and as mentioned earlier, the system can be years out of date and still be successfully upgraded by an emerge --update. Although the upgrade path certainly involves much manual dependency resolution on a system that far out of date, portage will guide you through the entire process, always giving explicit instructions on how to resolve conflicts. This approach doesn’t need to come at the cost of security either. Gentoo has developed the glsa-check tool to help you audit your system for any packages with known security flaws. Once you have identified affected packages with the tool, you can emerge the @security list set in order to upgrade only affected packages.

The Rest is History

Constantly updating Arch actually got to be a bit much for me. Once a week I would be installing a whole host of new packages, and ocassionally something would be acting screwy after an upgrade (phonon, VLC, and pulseaudio were the worst offenders by far), or I would come to discover that some package had just upgraded major versions and now had a completely different interface (e.g., my surprise bluez4 to bluez5 upgrade). Upgrades on Arch can also feel unnecessary sometimes (Do I really need to upgrade the entire KDE base every month to get the latest minor revision? Did I really need to upgrade my kernel from the one I got last month?). Despite my gripes in dealing with the constant upgrades, I never strayed from Arch because every other platform has felt absurdly stale and unnecessarily out of date. With my discovery of Gentoo and Portage however, I may have finally found a home in which I no longer have to pick solely between stability and modernity. Too good to be true? I suppose we’ll find out in time.