A Non-Developer GitHub Fever Dream

“Where am I? What… is this place?”

The Horde looked up from their work and saw me. Eyes wide with fear. One sniffed my hair but didn’t find what he was looking for.

“Oh, you’re a clever writer. One of those new-aged copywriters-turn-content-strategists who want to make things “easier” and “more intuitive.” That’s fine. Here in developer land, we have a thing called GitHub. A place where work lives. You could make pull requests if you had any talent, but you don’t, so you should probably just open a GitHub Issue and be as clear and specific as possible.”

Click click click click click click. What to name it. What to name it. Let’s look at some of the other Issue names so I can try to blend in.

//Punycode XSS on the production migration header. 2-Factor Auth zone file restrict. Infrastructure architecture failed the MCAT.//

Ok, how about I name this, “Idea for making the content easier to find.” Now, let’s give everyone a chance to comment.

Click click click click click.

The Horde recognized the foreign issue and stood. An understanding unit sighed and slowly lowered his Club-Mate. “Oooh. Yeah. This comment thread looks deep. Lots of differing thoughts. We should probably hold off on that until we finish with some of our Elixir Phoenix Commodore 64. That way we can complete the pipeline and pacify the wildflowers.”

“But, but, now what? Should I edit the top part to reflect the comments? Do I rewrite the whole thing in a new comment at the end? Do I close the issue and start a new one? How are people going know what’s actionable here?”

“Hmmm, that’s a good question. GitHub Issues isn’t meant for complicated ideas, y’know? It’s more meant for issues that can quickly be solved.”

“Maybe we should move these feature requests to a different platform?”

“GITHUB IS WHERE WE LIVE, CHILD. And please, stop trying to open new repos. That just makes finding things confusing. BEEP BEEP BEEP BEEP BEEP.”

Faint red flashes appeared in the distance. I’ve angered The Horde.

“What are you saying? Why do you keep beeping at me? AHHHHHHHH!”

Uh uh, what? Oh phew, just my alarm. I’m ok. This is all going to be ok.

It Might Be Dull, but Take Your Time

“couch”

It’s Sunday night; I have a kid learning to crawl on his mat, a football game on TV, an old fashioned in-hand, and all I really want to do is register a domain name for a project I’ve half thought through. Most of the initial planning is done, and there’s even some content written, but I haven’t spent any time on the name.

The name. It’s the one thing that’s inseparable from every brand. Fortunately, in 2016 you don’t need a great name to “make it,” but you do need a good one. More specifically, you don’t want a bad one. A bad name is a cancer. A demoralizing flag waving around announcing your mediocrity.

Don’t let that scare you. The real difference between the big budget brand name and your garden variety startup-on-a-whim name isn’t a special degree or talent; it’s perseverance. As Sun Tzu would say, it’s good to have a strategy to win, but winning demands a strategy for not losing (I’m pretty sure I made that as confusing as possible).

What I mean is that naming a brand/website/whatever is a process, even for the most seasoned branding pros. You might hit the perfect thing right away, but stopping there isn’t a winning strategy. No matter how clever you are, you’re doing yourself a disservice if you haven’t reached into the dark recesses of your grey matter to uncover even the worst of ideas. Don’t just come up with five names; get out a pad of paper and write out 100 (or more). And don’t get caught in trendy name schemes. After you come up with a handful on one train of thought, go in a different direction. Try to make up words. Use foreign languages. Play around with some name generators. Pull out the old thesaurus. Be creative.

So…

Dear Chris. Next time you find yourself sitting on the couch trying to rush a naming process, remember writing this, take a slow sip of that old fashioned, put down your computer, and spend an hour or five with a pen and paper. Your website projects will thank you, as will your bank account (I have a terrible habit of buying domain names, not liking them the next day, then setting them to expire).

If You Don’t Like Blogging, Start a Podcast

“mic”

I hinted at this theme yesterday, but the real dream of the internet is that anyone with a voice can stake a plot of land online and speak their mind. If you like writing, get a domain name and start a blog. But it’s 2016 – if you don’t like writing, don’t feel like your voice isn’t welcome. Get a domain name and start a podcast.

Think it’s too hard? It’s not. Here’s a quick primer on how to do it.

Get a domain name

Step 1 is easy. Just pick a domain name that’s easy to remember – the TLD really doesn’t matter. Like .pizza, .blackfriday, or .me? Go for it. Most people looking for podcasts find them through iTunes (or their podcast app of choice), so you don’t necessarily have to be as conservative with your choice as you would be if you were a restaurant or local credit union.

Pick a hosting platform

It might seem like podcasts require some sort of secret juju to appear on iTunes, but really, all you need to do is find a place to host the audio files, then follow some very simple iTunes submission steps to get it up. (Here are the Squarespace instructions, as an example, via their Support page… it’s easy).

If you want some suggestions on which platforms to use, here are three in our plugins page:

Get a mic

You could record a podcast with your internal computer or phone mic, but I’d recommend something a little more robust. There are a million sites offering recommendations (here’s a good one from Marco Arment)… I’ve used a Blue Yeti in the past and have been fairly happy with the results.

Find some recording software

You can go crazy with this, or you could just record on whatever free software you can get your hands on. As a Mac user, I’ve been pretty happy with GarageBand for podcasting. Again, there are a ton of sites with tutorials, but if you want a robust overview, check out the one on Lynda (a free trial is available).

Here’s a little suggestion: if you’re worried about spending too much time editing your audio, just don’t do it. Seriously, if you stay mindful of awkward pauses and bridge words like “like,” your audio will probably be fine without edits. (One of my favorite podcasts, Roderick on the Line, is almost entirely unedited, save for a cut in the middle for an ad spot and some intro music.)

Don’t be too self-critical

No one likes hearing their recorded voice. You’ll sound weird, and you’ll probably be ashamed of your speech patterns. That’s ok. Deep breaths. If you drive yourself crazy, be mindful of things you can change, and change them. But more times than not, being self-critical is simply a distraction. Your voice is yours, and if people like what you have to say, they’ll learn to be ok with it.

Post-Election Domain Thoughts

I’ll keep this short because just about everything there is to say about the US presidential election has already been said. Whether the person you wanted to win won or not, the United States has a proud tradition of being a place where you can speak freely. And now, more than ever, we need nuanced, thoughtful voices to rise above the noise.

If you have something to say, or a point of view that needs to be heard, get a domain name and start a blog. It’s not hard. Be the thoughtful voice people are sharing on social media, instead of hoping the things you want to read will be written.

A Blog About Blogging With a Pre-Ordered .blog Domain… Blog

  “typewriter”

Blogs about blogging have been written since the beginning of blogs, but none have had the privilege of ending in .blog. You’ll find your .com’s, your .net’s, and .me’s, sure, but sometimes you want a blog distilled to its truest .blog form. For those who wish to wish such a thing, .blog is arriving on November 21st.

I’ll repeat that for those lost in the Seussian nonsense. On November 21st, you’ll be able to register domains ending in .blog.

Here’s the important part. If you “pre-order” a .blog domain now, you’ll have a chance to get the domain you want before all the “regular folks” realize .blog has even gone live.

There are stipulations though. I put pre-order in quotes and used the phrase “you’ll have a chance” because the domain pre-order process isn’t a true guarantee you’ll get the domain you want. Basically, by pre-ordering a domain, you’re saying you have dibs on a certain domain – but there’s a chance other people from around the world want the same domain. At that point, who gets the domain is kind of random, so the more generic your selection is, the more likely you are to end up in one of these random drawings.

Regardless, getting your pre-order in now gives you the best shot at those coveted generic and single-word .blog domains, and demand seems to be quite high. So be sure to get your .blog domain on the list before the 21st…or else!

From Micro-Services to Monoliths

“frecks”

Over the last 18 months, we’ve re-platformed twice, hopefully without most of you noticing at all. The first major shift was for cost advantage, changing our host provider, and the second was moving to the latest Debian release, which brought with it a flurry of patches and changes — not all of it wanted. There was a feeling we were beholden to our OS and its patches, rather than to our customers and our business. systemd introduced a cascade of changes requiring changes to all of the services we run, and the changes continue to roll in over time.

Migrating hosting providers enabled us to move our core database servers, running Apache CouchDB, to much larger and faster machines, using SSD and much more RAM. Building indexes is faster, which allows us to deploy code faster as well. In fact, it’s now possible to cache our entire database and indexes in RAM, which has helped the responsiveness of our whole site.

We’ve always had a fairly micro-services-like architecture, and it’s stood the test of time. The front-end application is written in Perl using the Catalyst framework, and communicates with workers in a variety of programming languages, using RabbitMQ as a message broker between services that run on several different servers. Our two main databases, Apache CouchDB and Kyoto Tycoon, both have built-in replication, which provides both application-level redundancy, as well as simplifying operations when doing backups or upgrades.

Despite that, micro-services have introduced small delays at every step - network latency due to round trips, further reliance on stable internet connections, and extra conversions between JSON and the native programming languages used for each service. End-to-end testing of our application was also very complicated. Something needed to change.

We’re not Twitter scale, and we don’t need the feature set of Amazon or Google’s clouds. We’re also mindful of avoiding Cloud Lock-in. Our profitability as a business won’t change much if we halve or double the infrastructure we use. We asked, is there a way to have the best of both worlds? The decoupling of micro-services, without the latency? The debugging simplicity of a monolith, without the interruptions from continual upgrades? Could we have the operational flexibility of a cluster without the risk of a catastrophic melt-down? Could we have a test environment that ran on a laptop, but that still matched production?

Probably a number of you are mumbling Docker under your breath. Some of you are getting sweaty palms and thinking of Linux containers. But we’d tried them, and it wasn’t the answer. We spent more time trying to get the tightly coupled container stack working together than actually shipping code to production, or improving the stability and reliability of our services.

The key failure of the container vision today is that, unless you are Google scale, where you have a significant cost advantage from reduced server footprint, and where you can afford a fleet of container infrastructure engineers to keep up with the evolving landscape, the operational effort simply doesn’t stack up to deliver the benefits. It shouldn’t be necessary to rewrite how we do logging, monitoring, packaging, and deployment, and to dedicate engineers to maintaining and grooming the container monster, just to simplify shipping code to production.

What we needed was some Plain Old-fashioned Boring Infrastructure. Loosely coupled at each layer, without introducing latency, and still allowing the flexibility of the container-think movement. Something that had a long support lifespan if we needed it, but that didn’t compromise our ability to keep on the front foot for patches and security of both the OS and the apps that came with it.

In the end, we settled on three core changes:

  • move our OS from Debian Linux to FreeBSD
  • switch out distributed container-style VM microservices to paired physical servers
  • migrate our Perl-based Catalyst core app over to Elixir & Phoenix

Many of you will know of these already, but here’s a glimpse into our thinking.

FreeBSD

FreeBSD is one of the original Free UNIX-like operating systems, and is going stronger than ever. It powers Netflix’s mighty streaming servers, and is used in a modified form in Sony Playstations and Apple’s iOS and OSX. Most Internet Providers use FreeBSD in some form. FreeBSD also has a long support lifespan for the core OS while at the same time allowing us to use the latest ports and packages - a neat mix of backward compatibility.

And to be honest, FreeBSD already had a leg-up as a couple of us have been using it for a while.

The three big advantages of FreeBSD for us are zfs, jails, and ports.

zfs is arguably the leading filesystem of all time, with great flexibility, and power including inbuilt high-speed compression, data checksums (no bitrot or silent data corruption), snapshots for replication and backup. It also supports boot environments, which is a clever snapshot-based way of managing upgrades safely. This makes testing and ultimately deploying a new version of FreeBSD, or our apps and services deployed upon it, largely risk-free, and very very simple.

FreeBSD jails are about a decade old, and similar to Linux containers conceptually. However there’re no venture-backed companies fighting over turf in the hope of achieving a VMWare-like monopoly, and the software is well integrated into the operating system and community tooling. The performance of jailed applications is effectively the same as that of running in the main kernel, both from a network and a filesystem standpoint.

The ports tree is a significant feature that all BSD-derived operating systems share - a massive repository (subversion or git as you please) of every piece of open-source software you could possibly imagine. As our core business is providing simplified domain purchase and management through custom software, we are often in the position of needing a specific version of a tool or application, or needing custom patches deployed immediately while we wait for the upstream application owners to merge a patch, or for it to trickle down into the OS distribution’s package manager. With the ports tree, we have a custom private repo for our own packages, and the ability to carry and patches or to hold back specific versions for our own needs, in a very simple and straightforward fashion. Where we’ve needed new packages, or to get changes committed, it’s proved extremely simple to do so. This makes maintaining our own infrastructure very simple indeed - installing a handful of packages gets us up and running much faster than in the past.

There is a fourth advantage for us, however. The FreeBSD community is close-knit, with a reasonably consistent culture about doing things right. In practice, this means there’s very little gap between issues we identify or knowledge we are missing, and the developers and community creating it. The documentation on FreeBSD itself is part of this culture of doing things right, and the integration between the OS itself and the tools it ships with are the result, and we are already looking forwards to contributing further to the community.

Overall, as a result of moving to FreeBSD we hope to spend significantly less effort managing our infrastructure, and more time invested in improving our services and business.

A Pair of Servers

FreeBSD’s jails allow us to run micro-services on a single box. Arguably this is no different to Linux containers in practice, however we are not fighting to accommodate a stream of changes that add no value to our business along the way, and we are not forced to change out our operations tools and processes. The value to our customers is not in having the latest container tech running, it’s in having simple and reliable services for the infrequent times they need to acquire or manage their domains throughout the year.

In each region we are deploying a pair of physical servers, each one providing the same services and applications. Within each pair, we are using DNS round robin name resolution, and CARP, a low-level IP availability solution built into FreeBSD, to provide load balancing and failover at a network level between our boxes.

The next layer up in the paired server stack is the awesome haproxy load balancer, which we use to ensure that we direct our users to the closest and best performing application server. haproxy also allows us to dynamically remove and add back-end services from the pool, whether during deployment or maintenance, and communicates across the cluster to maintain a transparent view of services for our customers.

This consolidation brings disparate virtual machines back onto the same server, whilst still using FreeBSD jails to maintain the microservices-like separation. Luckily, none of our apps have required major changes to make them run on FreeBSD - UNIX standardisation has been a huge benefit here. When this is complete, we’ll have significantly reduced our app latency by removing the network round-trips that we have today.

The key difference here to the Docker-style container architecture is that there is very little coordination or dependency between these layers.

CARP is fundamentally a network protocol, and we could easily disable it should there be any issues, or choose some other facility. haproxy could be replaced by nginx which we already use today for slightly different functionality. zfs provides an incredible filesystem, available within jails and to the core operating system, as a solid and reliable platform for our services and your data. Logging, monitoring, and upgrades are all done using the same decoupled tools using well-known UNIX standards in place for decades — and there’s nothing wrong with preserving that simplicity where it suits us.

Elixir and Phoenix

Since almost the beginning of iwantmyname, the programming language Erlang/OTP has been at the heart of things. It’s the programming language that Apache CouchDB is developed in, as well as RabbitMQ, our message broker, and our core search application is also written in Erlang. It’s robustness has shown time and time again as it transparently deals with issues such as transient connection failures to our API partners, and has generally required significantly less maintenance effort than our other services. As a concurrent functional language with soft-real time characteristics, it is ideally suited to building websites and services that make heavy use of asynchronous internal and external APIs.

Our front-end app, written in Perl’s Catalyst framework, was leading edge when we first started using it, but it’s become more and more of a hindrance in evolving to a more robust, mobile first, system. Perl’s forking worker model means that we use a significant amount of RAM across our infrastructure, just to ensure we can handle what is definitely not “web scale” user and network load.

After experiencing several years of solid Erlang reliability, we picked Elixir, a new functional programming language that runs on the same Erlang VM, and the Phoenix web framework written in the same language, to upgrade and eventually replace our front end application.

Erlang’s BEAM virtual machine provides Elixir, and Phoenix, with a screamingly fast robust and reliable concurrent web framework, without the memory overhead of a forking model, and able to handle many concurrent connections transparently. Both Erlang and FreeBSD are used heavily by WhatsApp, and they shot to world-wide notice when FaceBook acquired the low-staffed app.

Live Debugging

Sometimes stuff breaks in production for no apparent reason, and we need to know why — in a hurry. In recent months we have traced & debugged transient internet outages, upstream API changes, timeouts and failures, unanticipated third-party library concurrency model changes, and much much more — much pain and frustration!

Debugging in our current infrastructure is a painful process, usually involving reading all the log files, using low-level Linux tools like strace, all the while inserting print statements and re-deploying furiously while trying to understand the underlying issues from the resulting flood of information.

Aside from being able to roll back safely any changes using boot environments and packages, our new stack provides some incredible introspective live debugging capabilities, which makes it easier for us to deal with problems in real-time and without downtime, and most importantly without needing to change compiler settings or edit our production code on the fly. These advantages alone would be reason enough to move. The Erlang VM provides a native erlang tracing library, and the community has extended this with erlang dtrace support, and the delightful recon. FreeBSD itself supports natively DTrace, the powerhouse introspection tool first developed for Sun Solaris, and since ported to several other platforms.

We’ve dubbed this setup the “FRECK Stack” - FreeBSD, RabbitMQ, CouchDB, Kyoto Tycoon. Yes, we had some closely related abbreviations in mind, but we managed to control our childish mirth.

Looking ahead

At the end of the day, iwantmyname is a domain registrar, and when it comes to domains, a “plain old boring infrastructure” is exactly what we — and you — should want.

With FRECKS, we’re now more stable and more secure than we’ve ever been — allowing us to truly focus our brain muscles on overdue UI and UX upgrades. More changes to our iwantmyname platform are coming, but behind the scenes, it’ll hopefully be smooth sailing out front — touch wood!