In the beginning

Some time ago, as I was learning how to write Python, how to interact with the terminal, and how to run small scripts, I decided that I wanted to – had to – go deeper into programming. At the time, I was working as a financial analyst. Until that point, I never quite felt the pull that programming suddenly had on me. The feeling was magical. I had power over the computer; I understood how and why it was doing something. So I wanted to learn everything I could, starting from the fundamentals and working my way up toward real, lasting understanding. But how would I even know where to begin?

This question has lingered with me since I started learning, and it certainly hasn’t left. I did some soul-searching (we’ll call it that) and eventually concluded that I had to direct my own learning. Naturally, I enrolled in the Recurse Center, which perfectly fit my self-directed learning style while also providing a supportive environment for the expected frustration of diving into a discipline on my own. But the question remained: what do I learn?

The Recurse Center gives a lot of valuable advice for learning programming. Almost all of the advice could be generalized for learning anything deeply, but one thought in particular has guided my learning so far:

Be rigorous. Understand how and why your code works. Understand your tools. If you’re working with a framework… learning to use it is just scratching the surface. Go deeper. Learn how it works.

To me, this is exactly what I wanted to do in my time at the Recurse Center: go deeper, understand how things really work. After learning the basics, I’d been continually frustrated in transitioning to a more advanced level on my own. The Recurse Center environment made it feel okay for me to be frustrated. It also helped me to turn frustration into curiosity – from “ugh, how does this work?” to “how DOES it work?” Finally, the environment convinced me that it’s not so scary to not know something — and even though it’s impossible to learn everything, it’s worth exploring what I don’t know.

One of my first goals at the Recurse Center was to learn and make something with Flask, a Python framework for building web applications. I quickly recognized all of the potential concepts I could learn more deeply. I’ll call them rabbit holes, which I feel effectively communicates both their potential for deep, lasting learning, and deep, lasting frustration. Any one thing that underlies a web framework could easily lead to a several-week investigation if one is persistent, patient, and genuinely curious. HTTP was one of those rabbit holes, but I decided it was one worth going down.

Why learn HTTP?

I decided to learn more about HTTP for a few reasons. First, it’s essentially universal – anyone who has ever used a web browser has almost definitely seen the letters “http” (or “https”) at the front of www.google.com or any website they’ve ever visited (and if you didn’t type them, your browser probably put them in anyway). Second, this universality indicated to me that, if I were learning to build web applications, it must be absolutely critical to know how HTTP works. Finally, I was just genuinely curious – “http” has been lodged in my brain since I visited my first website on good old Netscape Navigator, and yet I’d never taken the time to understand what it is or how it works.

I should also note that one of the best things about HTTP is that it’s an amazing abstraction of fundamental operations of the web. This is one of the enormous benefits of HTTP — it’s entirely possible to build a webapp without the faintest idea of how HTTP works. So the benefit of learning this isn’t to get something working — it’s to understand how and why it works, and how to fix it if something goes wrong.

HTTP – aka Hypertext Transfer Protocol

In the spirit of the Recurse Center, let’s dig deep and start at the very beginning – what does HTTP even mean?

By way of introduction, I’ll use Wikipedia’s summary. In my experience, the opening paragraph on Wikipedia for a given topic – particularly in programming – is usually an amazingly dense summation that makes little to no sense without an understanding of 5-10 other concepts. Because of this, anytime I’ve really wanted to learn something in programming, it’s been worth my time to understand every single word in the Wikipedia definition. By no means has it been my only resource, but it’s often been a great start.

The first sentence of the opening paragraph:

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web.

Given that definition, here’s a breakdown of what each letter in HTTP means.

  • Hypertext: At its most basic, text with hyperlinks. In the context of HTTP, this has grown to encompass all of HTML – aka Hypertext Markup Language – which defines the presentation/structure of web pages. This is effectively the data or content that is being sent using HTTP.

  • Transfer: The process of moving the hypertext between client and server. I’ll assume it’s just HTML from now on, but it could be other types of data, such as JavaScript source code, a PDF document, or JSON.

  • Protocol: The rules in place for transferring HTML. There are lots of different application protocols – rules that define how computers talk to each other over a network – HTTP is just the most widespread.

Most simply, HTTP is a set of rules that defines how computers send HTML over the Internet. The rest of Wikipedia’s definition (and opening paragraph) is important, but is essentially expanding on what “hypertext,” “transfer,” and “protocol” actually mean in this context.

But how does it work? And who decides these rules?

How it works

When you type in a URL (like http://www.google.com) in the address bar of your browser and hit Enter, your browser is acting as the client and is opening a connection to some server (like Google’s servers) and making a request to that server.

The request

More formally, the request is known as an HTTP request. In its simplest form, it looks a little something like this:

GET /index HTTP/1.1
Host: www.google.com

And it breaks down like this:

  • GET: The HTTP method. Essentially, when you visit a website, you’re receiving data from the server in the form of HTML – that’s a GET. If you were sending data to the server (e.g., you’re signing up for Facebook and entering an e-mail and password) that’d be a POST. There are a number of other methods, but GET and POST are the most common.

  • /index: The request-URL. You, through your browser, tell the server which page you’d like back; it’s the server’s responsibility to find and return that page.

  • HTTP/1.1: The version of HTTP (1.1 has been the standard version for a while – since 1999). Important so the server knows what kind of request to expect (HTTP has changed since the beginning).

  • Headers: Anything under that first line (aka the request line) represented by some text separated by a colon.

This message has just one header – the Host – but it could have any number – usually to give some more information about the request/requester. The request message could also have an entity-body – some extra info about the request, generally there for POST requests (as in, if you’re sending some data to the server, it’d show up here).

So what’s actually happening in that request? The browser is requesting something from the server – specifically, the resource (fancy term for some data - usually an HTML page) found at /index with host www.google.com. When the browser sends the message, it’s asking the server, “Hey, can you send that thing at /index from www.google.com?” And the server now has enough info to actually respond to that request. (Get excited!)

The response

HTTP/1.1 200 OK
Content-type: text/html
Content-length: 60
Hi, I’m a response! What a cool request you just made there.

Let’s break it down!

  • HTTP/1.1: The HTTP version, again. But it’s not too redundant – it’s important that your browser knows what version the response has. If it’s like, 0.9 – scary stuff – it’ll know to deal with it differently than trusty 1.1.

  • 200 OK: This is the fun stuff. There are really two parts to this – the status code (200) and the reason phrase (OK). The status code is used by the browser to determine what actually happened – like if it actually got something back, or if whatever the browser requested isn’t actually there. The most common are 200 (the browser got something!) and 404 (the server couldn’t find what the browser requested!). There’s a full list of status codes on Wikipedia and on the official site of W3C (more on W3C in a second).

Two fun facts – if you expand your definition of fun to include HTTP status codes – you can make up your own status codes (as long as they fit into the general buckets as described in the HTTP specification – e.g., any code in the 200s means a successful request) and you can make up your own reason phrase. So sending “200 OK” is the same as “200 Everything's A-OK.” The reason phrase is entirely for human eyes – the server and client won’t do anything with it, as long as it’s on the same line. It’s not really recommended that you make things up – but hey, it’s fun!

  • Headers: Same format as in the request, but generally they describe the data that’s being sent back (like the type, the length, etc.). You can also send whatever headers you want (if you’re feeling adventurous) but there are some typical headers that are generally used, like “content-length” or “content-type.”

  • Entity body: This is the actual data being sent – the entire point for HTTP existing is to send this data – usually, it’s HTML. But it’s also optional, if you’re into sending blank messages!

So that’s a pretty typical HTTP communication – your browser (client) makes a request, and the host (server) sends a response, which your browser renders (aka makes it look pretty). There is obviously a giant amount of complexity that can be introduced, but even in the most advanced web apps, the pages are almost always requested and sent using HTTP.

So, how have these standards been determined?

Who decides?

The W3C – aka the World Wide Web Consortium – decides. W3C is (according to their website) “an international community that develops open standards to ensure the long-term growth of the Web.” Here’s some more on their mission, and here’s Wikipedia’s summary of what they do.

Tim Berners-Lee, “best known as the inventor of the World Wide Web” (according to Wikipedia), founded the W3C in 1994. The W3C decides on standards for the Internet and attempts to enforce compliance with those standards. HTTP is just one of those standards, but it’s a pretty important one. I have the W3C to thank that when I go to www.google.com, I don’t get some crazy error. Because of HTTP, my browser knows how to request that Google send back a page, and Google knows how to respond to that request. That’s a huge deal. Without some semblance of standards, the Internet would’ve descended into chaos long ago. And it’s in the best interest of developers and organizations everywhere to conform to the standard – otherwise their sites and applications wouldn’t be supported.

Takeaways

I’m leaving out a lot – the history and evolution of HTTP, the future of HTTP (get excited for HTTP/2!; also see the Google-developed SPDY, which Google has been using and which “was chosen as the starting point” for HTTP/2), and more complex uses of HTTP. But I’ve used the simplest possible example to illustrate the basic idea. A client makes a request, usually through the browser, and that request conforms to HTTP. Given that request, a server will send a response, which also conforms to HTTP. The protocol itself is defined by the W3C, and it has changed since the beginning (and will continue to change). Beyond that high-level outline, my biggest takeaway was that HTTP underlies nearly every action on the Internet. For that reason alone, it’s worth researching even further.

It was well worth my time to dig a little bit into HTTP for a whole host of reasons, but most importantly because I no longer take it for granted that I can type in a URL and get back a web page. Honestly, it’s somewhat incredible that it works. Even more significantly for me, though programming is full of potential rabbit holes like this, I’ve become much more comfortable with the idea that I’ll never be able to go down every one. At the same time, I’m convinced that it’s sometimes worth going down the rabbit hole.

Some resources on HTTP (some linked in the article, some others):

  • Hypertext Transfer Protocol: This is literally what is meant by HTTP — the definition of the protocol itself. Not exactly a beach read, but a great reference.

  • HTTP: The Definitive Guide: A 658-page behemoth that spares no detail. I’ve only read the first 70-or-so pages, which go into some of the above topics (what is HTTP, how do you format messages, etc.) — so I’m no expert — but it’s well worth a skim of at least the first few chapters. It was written in 2002, but amazingly, most of what I’ve read is still relevant.

  • HTTP (Wikipedia): Gives a great overview of what HTTP is, its history, and where it’s going.

  • HTTP Status Codes: I’d also highly recommend 451: Unavailable for Legal Reasons (still a draft).

  • W3C: Site for the World Wide Web Consortium.

  • W3C (Wikipedia): A good rundown of the history/mission of the W3C.