Continuations

Tech Tuesday: Routing and TCP/IP

Tue, 07 Feb 2012 08:21:35 -0500

We are continuing along with the web request cycle. Last week we took a look at the HTTP protocol. There I already mentioned that HTTP requests and responses travel over a TCP/IP connection. Today we will dive a bit deeper into TCP/IP. This is technically not really necessary for understanding the request cycle because these lower levels of the network are completely abstracted away when you develop for the web (which is a fancy way of saying you get to use it without worrying about how it works). Yet, peeling the onion a bit further will turn out to be very useful to the overall understanding of how things work on the web.

In the Tech Tuesday on networking, I introduced the idea that the Internet is a packet switched network. As a refresher this means that data gets cut up into packets. The IP layer is responsible for how these packets move across the network. What follows is quite a bit of a simplification but good enough for our purposes here. Each packet (sometimes also referred to as a datagram) has its own header which contains among other things the source and destination IP addresses. These packets travel between machines along flexible paths known as routes. There is a tool called traceroute for examining what these routes are and it is worth trying this out.

On a Mac, use Spotlight to find and start the “Terminal” application. You will get a new window with a prompt which lets you type commands (this is known as the command line and we will learn a lot more about it in a future Tech Tuesday). Type “traceroute google.com” and you will see output that looks something like the following:

 1  192.168.1.1 (192.168.1.1)  1.987 ms  0.864 ms  0.794 ms
 2  10.32.128.1 (10.32.128.1)  9.576 ms  8.510 ms  7.638 ms
 3  gig-0-3-0-7-nycmnya-rtr2.nyc.rr.com (24.29.97.130)  7.983 ms  8.371 ms  8.123 ms
 4  tenge-0-5-0-0-nycmnytg-rtr001.nyc.rr.com (24.29.150.90)  12.007 ms  12.481 ms nycmnytg-10g-0-0-0.nyc.rr.com (24.29.148.29)  14.716 ms
 5  bun6-nycmnytg-rtr002.nyc.rr.com (24.29.148.250)  18.132 ms  11.899 ms  12.706 ms
 6  ae-4-0.cr0.nyc30.tbone.rr.com (66.109.6.78)  7.120 ms  8.395 ms  8.113 ms
 7  ae-4-0.cr0.dca20.tbone.rr.com (66.109.6.28)  13.161 ms 66.109.9.30 (66.109.9.30)  14.679 ms ae-4-0.cr0.dca20.tbone.rr.com (66.109.6.28)  13.992 ms
 8  107.14.19.135 (107.14.19.135)  14.153 ms  12.694 ms ae-1-0.pr0.dca10.tbone.rr.com (66.109.6.165)  14.154 ms
 9  66.109.9.66 (66.109.9.66)  15.230 ms 74.125.49.181 (74.125.49.181)  13.553 ms 66.109.9.66 (66.109.9.66)  13.315 ms
10  209.85.252.46 (209.85.252.46)  17.017 ms  14.467 ms 209.85.252.80 (209.85.252.80)  15.536 ms
11  209.85.243.114 (209.85.243.114)  26.926 ms 209.85.241.222 (209.85.241.222)  25.348 ms  25.406 ms
12  216.239.48.103 (216.239.48.103)  25.799 ms 64.233.174.87 (64.233.174.87)  25.046 ms 216.239.48.103 (216.239.48.103)  32.101 ms
13  * 209.85.242.177 (209.85.242.177)  40.436 ms *
14  vx-in-f103.1e100.net (74.125.115.103)  25.568 ms  26.283 ms  26.659 ms

Each one of these lines represents a so-called “hop” — meaning packets traveling between two internet devices. The first hop is from my computer to my home switch. The second hop is from there to my home VPN device which is connected to a cable modem from Time Warner. From there the packets travel over a whole bunch more intermediate switches and routers until the get to a server operated by Google. You can try this with other servers as well, such as “traceroute www.dailylit.com” — if the output get stuck with lines containing just “* * *” instead of information on hops, then you can terminate the process by pressing Ctrl-C. For those of you on Windows, here is how to run a traceroute.

Now the really important part to keep in mind about the IP level of the protocol is that it is strictly best efforts. This means that packets can travel different routes, can get dropped and can arrive out of order at the destination. So how in the world do we get an HTTP request and response across such a fundamentally unreliable network? Well that’s where the TCP portion comes in. TCP the Transmission Control Protocol sits on top of IP and provides for guaranteed in-order delivery of packets. How does it do that? Well, the details are complicated, but for our purposes it is sufficient to understand that it starts with a fair bit of initial “handshaking” (back and forth) where the two endpoints (sender and receiver) agree on what they will do. Once that “connection” has been established it becomes possible to keep track of which packets have been received and which have not and to cause packets that might have been dropped to be resent.

What are some of the takeaways here? First, having fewer hops will make things faster. If you try different servers with traceroute, you will see that a lot of servers are more hops away than Google’s — Google has invested heavily in shortening the paths to their servers. This is also what so-called CDNs or Content Delivery Networks do. They bring content (e.g., images) closer to the “edge” of the network so that requests have fewer hops. Second, setting up a TCP connection involves a fair bit of overhead. In the first version of HTTP each request required a new connection which was very inefficient. With HTTP 1.1 a single connection is kept alive for a sequence of requests and responses (a session). But there is still a separate connection required for each different server and so a web page that connects to many different resources incurs more overhead. Third, if you really want a lot of speed it helps to reduce the number of packets that need to be sent. In the early days, the entire home page of Google was optimized to fit into a single package.

Learning to Love Sales

Fri, 03 Feb 2012 08:43:36 -0500

I started out in business selling development services. OK, so that’s hugely glorified. I was a teenager desperate to get my driver’s license in Germany where that is a costly process (lots of mandatory lessons). So I figured out how to make money programming custom applications for people, including a driving school. Ever since then I have had a love/hate relationship with sales.

The hate portion is easy to understand for any engineer. The things you have to do in selling run counter to a lot of things you care about as an engineer. For instance, you need to spend a lot of time explaining stuff to people that should be, well, obvious. And most importantly, time spent selling is not time spent creating. The same reasons for hating sales seem to apply to product and design folks.

But I also love sales and not just because it helped pay for my driver’s license. People paying for your product is what enables you to grow your business without giving up (more) equity. And selling is what provides critical feedback about what you should build, making your product better. If you have the right kind of product or service (one with network effects), then selling has the additional benefit of making the product/service better for everyone. Finally, selling is about educating users who otherwise wouldn’t know how or why to use the product or service.

Unfortunately, I see all too many product and/or engineering led organizations that are in thrall to their hate or disdain or at a minimum personal dislike for sales (and by extension sales people). That’s highly unfortunate because it dramatically reduces the overall chances of success for these organizations. And one of the grand ironies is that many of these organizations think that they are in some way emulating Google, which has somehow managed to create a myth that Google got big without sales (nothing could be further from the truth). A similar myth seems to be in the making about Facebook.

Incidentally, I don’t mean sales here just as in having sales people. I also mean selling as in convincing endusers explicitly of the value of a product (ok, so technically that’s marketing but it has many of the same aspects). I haven’t figured out yet how to help people to learn to love sales. But maybe a starting point will be to point at how important selling has been to the success of companies such as Google. And of course selling has been critical to the company currently so beloved by many engineers, designers and product folks alike: Apple.

Facebook's Valuation

Thu, 02 Feb 2012 12:35:58 -0500

I was going to write a post about Facebook’s valuation, but Bill Gurley has done such an excellent job, that the better idea is to point at his post explaining “Why Facebook Clearly Belongs in the 10x Revenue Club.” There is one other important point to consider in thinking about Internet company valuations in the current economic environment: low interest rates. Companies that are still growing and have a lot of room for future growth have a fair bit of their value sitting in the future — that’s certainly true for a company such as Facebook. As we are currently in a global deflationary environment (*) the discount rate being applied to these future cash flows is lower than it has been in a very long time and possibly ever. When I started learning about DCF models in the late 80s, a common rule of thumb was to use 7% for the risk free rate of return!

The (*) above is to indicate that we have been expanding the money supply like never before but the lending multiplier has contracted even faster and supply is far outstripping demand in combination resulting in a deflationary environment.

PS If anyone has seen a good analysis of revenue composition for Facebook (advertising versus credits, assuming that’s even disclosed in the S1) please let me know

The Challenge: A Decentralized Rights Registry

Wed, 01 Feb 2012 11:50:45 -0500

Since my post last Friday about alternatives to SOPA/PIPA, I have started to talk to a bunch of people about the idea of a decentralized content registry. Here are some thoughts and questions that I have been kicking around since then.

A decentralized system could offer a new revenue stream for existing registrars and certificate authorities who are already at least partially equipped to deal with issues of verification. By having a competitive situation from the start the price for content registration can be determined by the market rather than set by the government.

We should figure out how to leverage DNS and DNSSEC in this context. The direction in which I am thinking is some kind of analogy to DKIM. Part of what this would require is an efficient way to create signatures even for large pieces of content, such as a feature length movie.

The scheme should probably specifically *not* support DRM and the idea of windowing either by time or geography. Why? Because those are exactly the types of artificial scarcity that piracy exploits. Instead all the content in the registry should be unencumbered and even externally cacheable.

Here then are some of the big questions to consider:

1. If the registry allows content owners to set a price (and likely a separate price for download versus streaming), then how does that money get remitted to the content owners? And how/where does usage get metered?

2. What is an efficient method for discovering copies that are not participating in the scheme? One idea here would be that content that participates in the registry gets fingerprinted. That would enable third parties to build services that report (presumably unsigned or mis-signed) copies that match the fingerprints. To that end it might be worthwhile considering a NIST competition for a publicly available content fingerprint technology (similar to NIST’s hash function competition).

I would love to hear from people who have thought about content rights registries before and/or work with some of the existing centralized ones. For this to work at Internet scale whatever solution comes out cannot be centralized.

Tech Tuesday: HTTP

Tue, 31 Jan 2012 09:04:48 -0500

Today we are continuing on with the web request cycle. After the browser has parsed the URL and obtained the IP address of the server via DNS, the browser now has to communicate with the server. That is done using the so-called Hypertext Transfer Protocol or HTTP for short. The beginnings of HTTP go back to the early 1990s when Tim Berners-Lee first devised it drawing inspiration from Ted Nelson, who had coined the term Hypertext in 1963. For an even earlier description of a similar idea it is worth reading Vannevar Bush’s amazing “As We May Think” from 1945!

HTTP builds on top of the lower level Internet protocol TCP which permits establishing a connection between two machines (see my introduction to networking). A so-called HTTP session consists of a series of requests from the browser followed by responses from the server. Each request consists of a request method, a resource (URL), a set of headers and optionally a request body.

The most common HTTP request methods are verbs such as GET, POST, PUT and DELETE (I am capitalizing them because that’s how they appear in the protocol). What’s great about these is that they are wonderfully descriptive of what you expect the request to do. GET is supposed to, well, get information from the resource. I say resource rather than server because that is the right level to think about with regard to HTTP — it is about manipulating abstract resources rather. PUT on the other hand puts information at the resource (without regard to what’s already there). DELETE — you get the idea — deletes the information at the resource. This relative obviousness and some associated expectations around how these methods behave provides a powerful foundation for the transfer of information (more on that in a future post on so-called RESTful APIs).

The headers contain additional information about the request. For instance, the “Date” header field contains the date and time when the request was sent. Or the “Referer” header (misspelled in the protocol and in most implementations!) contains the URL of the page on which the currently requested resource was found. It is worth looking at the list of possible HTTP headers, which also shows the headers for a response (see below). It should be pointed out that the HTTP protocol allows for the creation of additional headers which can carry custom information (not always what you would want as in the recent case of O2 sending users’ phone numbers!).

The request body is used for POST and PUT requests to carry the data. For instance when you encounter a registration form on the web that asks for your name and email address, the information you type into the form fields is (generally) carried in the body of the resulting HTTP request.

Once the server has received and processed the request it will send an HTTP response. The response has a structure that’s quite similar to the request. Instead of the method, the server returns a status code, then some headers, and finally a response body.

The status code indicates what happened at the server and hence what to expect in the body of the response. The standard code is “200 OK” which means the server processed the request and everything went well. There are more precise responses in that vain, such as “201 Created” which means the server created a new resource (e.g. in response to a PUT request). There are a series of codes to deal with resources that have moved, such as “301 Moved Permanently” which provides a new URL that should be used for all future requests for this resource. And there are a bunch of codes to indicate various error situations such as the famous “404 Not Found” for which some web site return very funny contents in the body of the response. Again, it’s worth browsing the complete list of response codes.

The response headers contain a lot of additional information about the response. For instance the “Content-Type” header field describes what kind of content the response body contains. Examples of different values for this header field are “text/html; charset=utf-8” for a web page in HTML using the UTF-8 character set or “image/jpeg” for an image that is compressed using JPEG. Without knowing this the browser would have to infer the content type from inspecting the body of the response which would be very cumbersome. There are a ton more headers in a response that are similarly critical to the proper functioning of the HTTP protocol, such as how long a recipient can “cache” (locally store) the body (in order to help speed up a subsequent access and also relieve the server and network).

Finally there is the body of the response which contains the actual information. The body is a bunch of bytes. What they represent can vary wildly as explained above. It could be an HTML web page or an image or something altogether different. One of the great powers of the HTTP protocol is that it is really content agnostic.

Because there is a lot going on with the HTTP protocol under the hood and much of it matters it is a bit of a shame that many people including active developers don’t really understand it and as a result either create things that don’t work as expected (e.g. making resources change in response to a GET request) or re-invent features on top of HTTP that HTTP already contains (e.g., content caching). If you do any work on the web it is well worth digging deeper than this post!

ReRAM: An Exciting Hardware Innovation

Mon, 30 Jan 2012 08:14:49 -0500

In my Tech Tuesday posts I have covered main memory and storage (by the way, coming up tomorrow: HTTP). If you have read those or otherwise follow hardware, then you will find this short piece from BBC Technology News on a new technology known as ReRAM quite interesting. Essentially, ReRAM holds the promise of providing non-volatile storage at the speed of memory.

That would provide a major breakthrough for database applications. Not only is ReRAM even faster than the Flash memory used in the SSDs (which are currently replacing traditional disk drives for high end database applications) but it obviates the need for going to disk in the first place. That means the whole intermediate software layer that controls disk access falls away as well.

What is amazing is not just that this is possible at all, but also the history of this technology. ReRAM is based on something called a Memristor which was invented as a theoretical possibility in 1971 (much after the Transistor which was invented in 1947). Then it took until 2008 to build a Memristor. From that breakthrough by HP it seems that commercial products may be available as early as next year!

Thinking About Alternatives to SOPA/PIPA

Fri, 27 Jan 2012 12:08:09 -0500

With SOPA and PIPA shelved at least for the moment, it is time to start thinking about alternatives. It would be a shame if we limited our collective thinking here to slightly different versions of those bills instead of exploring what a different approach to copyright could be that doesn’t try to fight the characteristics of the Internet but rather embraces them, providing value for rights creators/holders, technology companies and endusers.

One interesting entry here is Ian Rogers (from Topspin Media) proposal for a rights and media registry. It’s worth reading the entire post and also the comments, which include good questions from Andy and clarifying answers from Ian. In essence such a registry would enable tech companies to deliver innovative user experiences on top of content, as long as they respect the prices set by the rights holders. Rights holders would be entitled to enforcement only if they participate in the registry.

I believe this direction is very promising and is also something that was recommended by a report that the UK government’s copyright office had commissioned. An important addition though would be that this should not be a centralized registry (which then requires an operator and become a single point of control and failure) but rather a standard for publication that would allow for a decentralized implementation.

Apple Is Slow Boiling Developers

Thu, 26 Jan 2012 15:25:53 -0500

How do you boil a frog? Slowly. Apparently the same is true for endusers and even software developers. That at least is what Apple seems to believe. And while this has been debunked for frogs (they do jump out as the water gets too warm), it’s not clear that the same is true for humans. We seem all too willing to trade off having a shiny device for accepting ever more restrictions on what we can do with that device.

I wonder how long it will take before people realize how much they are losing when instead of a general purpose computer they have a locked down device controlled by a central choke point. I am especially curious when developers like Marco will conclude that this is no longer in their interest. And I am fascinated to see Gruber write a long post arguing that Apple’s new ebook “standard” is not a classic case of embrace, extend and extinguish. What line of control does Apple have to cross for him to say it’s actually a step too far?

The latest tightening of control by Apple is making some APIs accessible only to applications sold through their store. I am not talking about apps for the iPhone or iPad here but applications for laptops and the Mac Mini. You can read more about it here. This whole direction is rather upsetting because I really like my MacBook. But I don’t enjoy being boiled, not even slowly.

Supermodularity And Service Bundling

Wed, 25 Jan 2012 14:00:00 -0500

This will be a bit of a wonky and short post with a longer and less technical one to follow some time soon. Google has just announced a coming update to their privacy policy which will essentially make it possible for Google to integrate all the information it has about a user across its many different services. This comes at the same time as the revelation that Larry Page apparently explicitly stated the goal of building “a single unified, ‘beautiful’ product across everything.”

While one can come up with many possible verbal explanations for why Google might want to go this direction, there is some powerful math that lies at the heart of it: supermodularity. Here is the definition:

A function

is supermodular if

for all x, y R^k, where x y denotes the componentwise maximum and x y the componentwise minimum of x and y.

If a production function is supermodular then x and y are strongly complementary. If you want to read the bible on this consult Don Topkis “Supermodularity and Complementarity.”

A firm such as Google for which the production function relies almost exclusively on information (yes, there are servers and people as well) will exhibit super modularity almost by definition. Why? Because if X and Y are different information vectors, then as long as they carry some joint signal, the inequality will be met as you can always choose to discard additional information (meaning you always have access to the component wise minimum). In plain English: if you have access to both the search history (X) and the social graph (Y) of a user, you can always “do better” than two separate services that only have access to one of these respectively.

Tech Tuesday: DNS

Tue, 24 Jan 2012 08:00:05 -0500

Today we are continuing with the web cycle that I outlined two weeks ago. After a URL has been parsed in Step 1, the browser needs to determine the IP address for the domain as Step 2. Reprising the previous example, let’s consider the domain name dailylit.com. How does the browser determine that in order to retrieve information from this domain it should access a server at IP address 72.32.133.224? This is accomplished via a system called DNS, which stands for Domain Name System, and provides essentially the equivalent of a telephone book which provides IP addresses (telephone numbers) for domain names (people names).

In ARPANET, the predecessor to the Internet, there were so few domain names that this telephone book was simply a file called HOSTS.TXT that was retrieved from a computer at SRI and stored locally. There were only a few domains (mostly universities) and the file was relatively short. Today on the Internet there are over 200 million domain names of the type dailylit.com, which are further subdivided through subdomains such as blog.dailylit.com. So the idea of having every computer maintain a complete and up-to-date copy of the telephone book locally doesn’t make sense any more.

Thankfully in the early 1980s, which depending on your perspective is either ancient pre-history or not that long ago, DNS was born as a service that would allow the registration of domain names and maintain a mapping between the names and IP addresses in a robust fashion. In fact, without DNS it would be hard to imagine the Internet having grown as dramatically and we probably wouldn’t have nearly as many domains to begin with.

There are many ingenious ideas in the design of DNS and I won’t be able to cover them all here. Instead, I will focus on some key concepts. The first and central one is that there is a hierarchy of authority which allows for the delegation of both registration of domain names and the lookup of IP addresses. The hierarchy starts with the 13 root servers which together make up the so called root zone from which all authority flows. It is here that the so-called Top Level Domains or TLDs get resolved. Going back to blog.dailylit.com, the TLD is the “.com” part. You can think of a domain name like nested Russian dolls, where the outermost doll, the TLD, is the rightmost part of the name.

The most common TLDs are .com and .net which together account for about half of all domain names. There is of course also .org, .gov., .edu and an ever increasing number of other TLDs such as most recently .xxx. And then there are TLDs for countries which all consist of two letters, such as .uk for the UK (duh) or .ly for Libya, popularized by bit.ly, and .us for the US, which made the domain del.icio.us possible. Each TLD has one or more registrars associated with it who are in charge of letting people and companies reserve names in that domain.

The root servers point to name servers for each of these TLDs. Since blog.dailylit.com is in the .com domain the next place to look is the .com name servers. The .com name servers in turn point to the name servers for dailylit.com itself. Currently those name servers are at Rackspace. Since Susan and I registered and control dailylit.com, we are the ones who get to decide which nameservers should be queried to find the IP address for dailylit.com and its subdomains, such as blog.dailylit.com. The way this generally happens is by logging into a system run by a registrar and setting which nameservers are to be the authoritative sources of IP addresses for the dailylit.com domain. That then gets recorded in the nameserver for the corresponding TLD.

The lookup process that started with the root, went to the .com TLD, is now at the dailylit.com nameservers at Rackspace. They in turn contain information on dailylit.com itself and its subdomains, such as blog.dailylit.com. The whole process of starting at the root and working towards the subdomain (right to left) in a series of separate lookups across different servers is called a “recursive lookup.” If this sounds complicated to you, that’s because it is. It is so complicated and resource intensive that we don’t want the web browser to have to do this each time it encounters a domain name. It would not only be slow, but it would also swamp the root servers, the TLD servers and possibly even the name servers for dailylit itself.

So instead of doing a recursive lookup every time, the results of these lookups are stored on so called DNS cache servers. For instance, most ISPs through which you access the Internet will operate their own cache servers. After they have looked up blog.dailylit.com once, these servers will “cache” (meaning temporarily store) the result of the lookup, thus providing a much faster lookup the next time. In fact, your own computer will often cache the results of lookups locally for super fast access. This is important both because even a single web page generally involves multiple requests (e.g. for images) to the same server. The duration for which the results of a recursive lookup can be cached locally is known as the Time To Live or TTL and is controlled by the owner of the domain (and generally honored by the cache servers).

The existence of cache servers (sometimes also referred to as non-authoritative servers — although technically not exactly the same) provides a critical security vulnerability for DNS. Let’s say you have gone to your favorite coffee shop and logged on to the WIFI network there. Where do your domain lookups go? Well to the cache server of whatever ISP the coffee shop uses or possibly even cache servers on the coffee shop’s own network. An attacker with access to those local cache servers could insert falsified records that could have the effect of say pointing chase.com to some rogue server that wants to steal your bank username and password. This would allow for a so-called man-in-the-middle attack (more on this in a future post). Fortunately, some security additions to DNS known as DNSSEC will in the future prevent these kinds of attacks. As more and more of our access to the Internet is over wireless networks this becomes particularly important.

If you made it this far, I hope you have a (newfound) appreciation for the complexity of a system that is used billions of times per day behind the scenes of nearly every access to the Internet. In addition to the technical issues there are also important political issues surrounding DNS. Most recently the proposed SOPA and PIPA legislation would have mandated nameserver operators to make changes that would have interfered with the implementation of DNSSEC. Then there is also the question as to who really controls the root zone which turns out to be the US Department of Commerce. Yes, for the *entire* Internet, which is all the more reason why we should make DNS better not worse.

Why Software As A Service (A Personal Reminder)

Mon, 23 Jan 2012 08:00:06 -0500

I am in the process of migrating a bunch of stuff off an ancient server which has been up continuously for 5 years and is being sunset (the hardware is tired and RedHat is ending support for RHEL4). The migration process has been a potent reminder as to the many hidden costs of installed software. For instance, at the time that the old machine was set up I used Subversion for version control and Trac for documentation and ticketing. Once these were working nicely for my purposes, I stopped upgrading them. Now you could say that’s a pretty stupid thing to do, but when you have very limited time would you rather fix a bug or upgrade your tools, especially when those are working!

So now of course I had a brand new machine running RHEL5 and much more up-to-date versions of Subversion and Trac. I have been using Git and Github for a more recent project but didn’t want to make a switch here as there is a lot of info in Trac that is nicely sync’d to the source. So how hard could it be? Well, the Subversion upgrade was a cinch. I dumped the repository on the old machine and loaded it on the new machine. Including time to set up Subversion, configure Apache to serve up the repo, and my laptop to connect, this all took less than 1 hour.

But then I got around to Trac where I was on version 0.10.4 and the current version is 0.12.2. As it turns out in between they switched templating engines and went from SQLite2 to SQLite3. I had to upgrade the old machine through several versions before I was able to export my data in a format that the new install could consume. Even then the documentation on how to do this was spotty at best. At one point I got so frustrated that I thought it might be easier to downgrade my new install to 0.10.4 but an attempt to install that old version on RHEL5 went nowhere. In the end I got it all done but it took almost 3 hours. Granted that someone who does this more often than once every other year might have been much faster, but my Google searches suggest that I wasn’t the only one to find this a bit tricky.

I still have quite a few other things to migrate to the new machine which I am sure will produce some similar problems. I could of course try to find someone to do all of this, but (a) that’s not easy given that this is a small one-off project and (b) I like to stay connected to how things actually work. And this object lesson in the cost of installed software was a good reminder for when this topic comes up with startups. For anything that’s not core to your success, consider using a Software As A Service offering over installing and running your own.

Some Quick Observations on MegaUpload

Fri, 20 Jan 2012 09:13:22 -0500

Yesterday an international police operation resulted in the shutdown of MegaUpload and the arrest of at least four MegaUpload employees in Auckland, New Zealand. This action resulted in a large scale DDoS attack by the group known as Anonymous on web sites including the MPAA, RIAA, DoJ and even the White House. While I don’t have time today for a full scale analysis here are some salient point:

1. The fact that this shutdown and the arrests were possible shows quite clearly that existing laws already provide a meaningful ability to deal with large scale copyright infringement even when sites operate from abroad. That’s all the more reason why we don’t need additional new legislation.

2. According to ArsTechnica, MegaUpload was brazenly flaunting the DMCA by only disabling links to infringing content instead of actually removing it or blocking access to it entirely. That is a violation of both the letter and the spirit of that law and should not be allowed to continue.

3. As with any digital locker site, there were also legitimate uses of MegaUpload. Many people who had work or personal files on MegaUpload have taken to Twitter to complain about a lack of access to their files. This operation and others before it (such as the server seizure that brought down Curbed, Pinboard and Instapaper) raise the question how to minimize “collateral damage.”

4. The retaliation by Anonymous has the potential to meaningfully escalate the push for government intervention in the Internet for cybersecurity reasons. This comes at a bad time as we are trying hard to keep the government out of controlling the Internet.

What a week this has been! Apple too dropped another interesting copyright bomb yesterday by claiming sales rights to any books created with iBooks Author. It seems like we are at the beginning of what Cory Doctorow has characterized as the “Coming War on General Purpose Computing.” We live in interesting times indeed.

The Day the Internet Stood Still

Thu, 19 Jan 2012 10:48:22 -0500

Yesterday (Wednesday, January 18), has a good chance as being remembered as the day that the Internet first truly showed its political clout in the US. So far we have largely pointed at events abroad when discussing the Internet’s potential to shift power. Web sites and services large and small (including Continuations) either forcefully alerted their users to the problems with SOPA/PIPA or blacked themselves out entirely. At the 12th hour even Facebook’s Mark Zuckerberg took a (by now very safe) stand on the issue.

The results from a political perspective were impressive. SOPA had already been stalled a bit but PIPA support was still strong. Following yesterday though 18 Senators including 7 co-sponsors withdrew their support for PIPA. Someone with a better knowledge of the history of American politics will probably know the correct statistics but this is a massive erosion in the support for a bill. Together with the White House’s stance against the bills in their current versions I believe that there is now a good chance to stop both SOPA and PIPA.

What’s next? First, as Ron Wyden makes clear in his terrific letter to the Internet there is still one more vote coming up on PIPA on January 24th so it is too early to declare victory. Second, it is worth reading the MPAA’s reaction to yesterday’s expressions to see just how cynical their view of what happened is. Third, unless we want a new wave of slightly different versions of these bills following the next election we need to proactively outline an alternative that is not based on government intervention in the Internet. Fourth, and maybe most importantly, we need to start the long work on using the Internet to shift political power back to the voters and away from special interests more generally and not just with respect to bills that directly affect the Internet.

Tech Tuesday: Anatomy of a URL

Tue, 17 Jan 2012 07:30:05 -0500

Last week’s overview of “How the Web Works” introduced the URL (Uniform Resource Locator) as the fundamental way things are addressed on the web. Before we pick apart some actual URLs, it is worth looking at the name itself. The promise behind “Uniform” is that this addressing scheme can be used across all kinds of resources and that explains why URLs are so powerful - they can be used to address content such as a blog but also services such as the Twilio telephony API. On the web a blog entry and an incoming phone call are both simply “resources”. That means a resource is a highly abstracted concept and as you will learn if you stick with Tech Tuesday, abstraction is amazingly powerful. And on the web the URL is the most powerful abstraction of them all!

So here is a URL to pick apart:

http://blog.dailylit.com/2012/01/16/in-honor-of-dr-martin-luther-king-jr/

The very first part, the “http:” indicates which protocol to use to access this resource. What other protocols might we find there? The obvious one is https: the secure (meaning encrypted) version of http:. Here some other protocols that you may have encountered around the web “mailto:” which indicates that the resource that follows is an email address and the protocol to speak to it is SMTP or you may have seen “ftp:” for resources that are accessible via File Transfer Protocol (FTP). Another protocol supported by many browsers is “file:” which means that the resource that follows is a file on the machine on which the browser is running.

Following the “http:” are two forward slashes “//” — these indicate that this URL starts with a domain name, which in this case is “blog.dailylit.com” — we will dissect domain names in more detail in the Tech Tuesday on DNS. There we will investigate the relationship between domains and actual servers but for now it is worth pointing out that grouping resources by domain serves an important trust purpose. Your expectations about accessing content at chase.com are meaningfully different from wepretendtobechase.com. Of course it’s not always that obvious and people go to great lengths to pretend to be someone else. There is a good test of your knowledge of which domains to trust.

Following the domain name is the location of the resource within that domain. This is the “/2012/01/16/in-honor-of-dr-martin-luther-king-jr/” part in the URL above. There are several things going on here that are worth noting. First, this location is structured in an easily human readable and comprehensible form. Just by looking at the URL you can infer that this is a post about Martin Luther King on MLK day. We call this kind of location a “pretty URL.” Having pretty URLs is a good idea not just because it helps humans figure out what they are likely to get when they access the resource but also because search engines, especially Google, make pages with pretty URLs rank higher in search results (assuming that the page content actually appears to be a match for the URL).

But there is even more to a pretty URL like “/2012/01/16/in-honor-of-dr-martin-luther-king-jr/” — the slashes “/” in the URL indicate some notion of hierarchy or of a path to the resource. It also suggests that the following shorter URL should point to something useful http://blog.dailylit.com/2012. In fact this retrieves all the blog posts from 2012. There is no requirement that the domain fulfilling the request understand this shorter URL, but the fact that it does corresponds both with intuition and allows for additional degrees of automation and discovery. For instance, without any further knowledge you should be able to construct the URL for finding all the blog posts from November 2011. Here it is http://blog.dailylit.com/2011/11/ . Again, there is no requirement on the server to respond to this with a list of posts and it could instead respond with say a 404 Page Not Found. The http protocol does not speak to this, which is one of its many strengths as it lets the person or organization controlling the resource decide how to respond.

Now not every URL starts with a “//” — there are also URLs that don’t contain a domain but instead just a path to a resource. Consider for instance the following http:/ — where the resource pointed to by this URL is located depends on the context in which it is encountered. This is an example of a relative URL. It points to a resource within the context of another resource. If you are reading this in the context of the Tumblr dashboard, the link will take you to your dashboard. If you are reading this on my blog, which is at the domain “continuations.com” it will take you to the home page of my blog. Relative URLs allow for more compact expression of the location of a resource but they can also introduce interesting errors. For instance, think about what resource that relative URL will point to if you simply copy it and send it to someone via email and they open it in a web mail client!

This post is getting quite long and I haven’t yet covered fragment identifiers or query strings. Instead of going on, I will cover fragment identifiers in the context of HTML and query strings when describing how URLs can be used to transmit additional information that can be used by the server in deciding how to respond to the request to the resource, so keep following Tech Tuesday!

Covestor: Getting the Message Out!

Mon, 16 Jan 2012 09:44:53 -0500

Our portfolio company Covestor today rolled out a tremendous overhaul of their web presence. The goal was to dramatically simplify Covestor’s message to make it easier for that message to spread and to improve conversion. The team has done an amazing job with a process that used extensive quantitative and qualitative research into consumer reactions to inform the redesign. While it’s too soon to tell how well they will hit their numeric goals, my immediate reaction to the new site is: wow! I encourage everyone who has wondered in the past what Covestor is about to go and check out the new Covestor site.

The overhaul touched every aspect of the site from updating the logo to changing pretty much all the copy. Here is the old logo on the left and the new one on the right:

What is externally visible though is only the tip of the iceberg. Internally, Covestor has done amazing work over several years to be in a position to roll out this new site. For instance, they have recruited hundreds of model managers to the platform. They have also developed a sophisticated risk score to help match models to investor strategies. And they have honed the trade replication engine that powers covesting.

Congrats to the entire Covestor team!

P.S. If you want to know more about Covestor in person, you can meet the team and others interested in covesting at their “Take Stock Mixer” on January 31st from 6-8pm.

Moving Back to New York City and Homeschooling

Fri, 13 Jan 2012 08:06:23 -0500

In the middle of 2010 we started to seriously consider moving back to New York City. At the time one of the considerations was that it would be possible to experiment with homeschooling the kids. I am excited to report that we are doing both. We have a place in Chelsea that is a short walk from the Union Square Ventures office and almost as importantly around the corner from Murray’s Bagels. We are also homeschooling our kids for at least the next six months. Now the “we” here shouldn’t be read to imply that Susan and I are doing the teaching - we both work full time. Instead, we have worked with Teri and Melissa from QED to recruit some amazing tutors. Susan has all the details on that over at a special blog about our homeschooling experiment.

Presenting Option Grants to Boards

Thu, 12 Jan 2012 07:07:27 -0500

One of the nearly routine items at startup board meetings is the discussion and ratification of option grants for new employees and possibly refresh grants for existing employees. Too often unfortunately this information is presented to the board in a way that requires way more time than should be necessary because critical pieces are missing.

Here is what you should always include as a bare minimum when presenting option grants to the board

1. Employee name

2. Title/role at company

3. Absolute size of grant in number of underlying shares

4. Percentage size of grant fully diluted

5. Total size of option pool and remaining available pool (absolute numbers and percentages fully diluted)

Just #1 and #3 are not enough because it assumes that board members will immediately recall the number of shares outstanding which may or may not be the case and/or be able to quickly do the percentage calculations.

In addition here is additional information you should provide for context

6. Grant size bands by role (if you have established those already) — if not, include existing employees in similar roles for comparison (including their start dates)

7. Indicate if there are any special vesting considerations that differ from the plan

8. For refresh grants: how many options does the employee already have and how far are those vested?

Providing all of the above ahead of time (or even at the board meeting) will not only make the grant process quick an painless but also assures that you can actually get meaningful input from your board on the size of grants (as opposed to spending time digging for information and calculating percentages).

Google Going All In

Wed, 11 Jan 2012 07:53:00 -0500

Last July I had predicted that Google would go all in by bundling Google+ aggressively with search and that is exactly what was just announced yesterday with Search, plus Your World. The “plus Your World” part right now refers “your world on Google” as only Google+ profiles, posts and shared images are included and not content from Twitter, Facebook or others. John Batelle’s capture this well in his aptly titled “Search, Plus Your World, As Long As It’s Our World.”

Also worth reading are Danny Sullivan’s excellent overview of what Search+ offers and his detailed analysis of whether or not Google could already include some Twitter content without a commercial arrangement with Twitter. Danny’s analysis has actual comments from an interview with Eric Schmidt. Finally, the most scathing reaction has come from MG Siegler who flat out titles his piece “Antitrust+.”

While it’s too early to know how all of this will play itself out over time (there has already been some public back and forth between Google and Twitter), two things seem fairly clear. First, in the near term this will be bad for end users. Second, the root of the problem are Google’s economics for search. The two point are intimately related.

On the first point, John Perry Barlow aptly tweeted:

We are becoming helpless collateral casualties in the war between Google and Facebook. bit.ly/WorldWarIII
— John Perry Barlow (@JPBarlow) January 10, 2012

From an enduser perspective the best web is one of little pieces loosely joined. That kind of web allows for lots of innovation and individuality. Instead, we are currently headed for big chunks of experience provided by just a couple of players. While a high degree of integration may look appealing to some under an “ease-of-use” type argument, all you have to do is look at the enterprise where a few large vendors have dominated for years (SAP, Oracle) to know how undesirable that is.

On the second point. the root cause of all of this are search economics. Google keeps one hundred percent of the search revenue from searches on Google. The explicit quid pro quo has always been that Google sends traffic to a site in return for getting to include the content among the search results. No search revenue is shared with the sources. During days when Google was just a search engine that seemed like a reasonable quid pro quo. But two things have happened to make this balance not work. First, Google has gradually entered many businesses that compete directly with providers of content and second we have seen the emergence and inclusion of many content “micro chunks” that will hardly ever generate traffic to the originating site, such as a restaurant rating from Yelp. I have argued before that some kind of revenue sharing will be required to break through this.

When Larry Page became Google’s CEO I had hoped that he would maybe pursue a vision of the web of little pieces loosely joined with Google providing a lot of that glue. It is by now amply clear that Google is going exactly in the opposite direction. That’s a shame in the near term. In the long run I agree with John Batelle that the web will find a way to route around all of this (assuming we don’t let the politicians screw it up in the meantime).

Tech Tuesday: How The Web Works (Overview)

Tue, 10 Jan 2012 07:00:19 -0500

As promised at the end of last year’s Tech Tuesday, we are starting this year with a cycle on how the web works. Just as a reminder, Tech Tuesday’s aim is to require no previous knowledge other than what has been covered before. So this overview may be trivial for some readers but I wanted to make sure to bring everyone along.

Let’s assume you have fired up your favorite web browser. Now you type the address “dailylit.com” into the address bar (if you always go to web sites by typing their name into a search engine, I urge you to discover the address bar and type in the address).

What happens now? How does the browser go from a web address for a site to that site’s content on your screen? That turns out to be an amazingly complex series of steps:

Step 1: The address “dailylit.com” is part of what is known as a URL. The full URL is “http://dailylit.com” and your browser automatically pre-pends the “http://” to save you the typing. The HTTP bit indicates to the browser which protocol to use to speak to the server (more on that in Step 3 below). Typing a URL into the address bar starts the same sequence of steps as if you had clicked on a link (e.g. among a set of search results) pointing to the same location as in DailyLit. In the very first step the browser “parses” the URL (meaning it takes apart the URL into its various parts) in order to determine where it is supposed to look for content.

Step 2: The “dailylit.com” portion of the URL is the domain name (you can get your own from a domain registrar). Think of this much like the name of a person. If you want to call a person on the phone you need to look up their phone number based on their name in some phone book (e.g. the contact list on your cell phone or in the dark ages some paper book made from dead trees). Similarly in order for your browser to retrieve the content from DailyLit, it needs to first lookup the IP address of the server on which the content lives. This is done by consulting a “phone book” known as DNS which stands for Domain Name System and is a near miraculous invention.

Step 3: Now that the browser has an IP address, in the case of DailyLit currently 72.32.133.224, it makes a request to “GET” content from 72.32.133.224. GET is capitalized and in quotes here because it is one of several defined requests supported by the so-called Hypertext Transfer Protocol or HTTP — which was what the beginning part of the URL. In essence this request simply says GET me the content that resides at 72.32.133.224. This is the protocol that got started with Tim Berners-Lee’s work in the very late 80s and early 90s and is to this day the underpinning of the interaction between web browsers and web servers.

Step 4: Through the magic of Internet networking, that GET request is routed via a whole bunch of intermediary devices (routers, switches, firewalls, load balancers oh my!) to the machine with the IP address. In fact, you can look up for yourself how many intermediate hops exist between you and the server and that’s something we will do in an upcoming Tech Tuesday

Step 5: We are now on the Server. The server receives the incoming GET request. On the server machine the work is co-ordinated by a program known as a web server (something like Apache or NGINX). The web server retrieves the contents for the page and starts sending them back to the browser again over the Internet. Important side note: because the Internet is packet switched, the content is cut up into smaller parts (packages) that may travel different routes to get back to the browser. All of that cutting up and re-assembling is handled by lower levels of the network and is transparent to both the web server and the web browser. That too is one of the many awesome features of the Internet that we easily take for granted.

Step 6: Back at the browser. The browser is receiving the content in the form of an HTTP Response. That response contains a bunch of different stuff. For instance, it contains a so-called HTTP Response Status Code to indicate to the browser whether the server thinks it has some useful information [Response Code 200 OK]. If the server had a problem, e.g. it didn’t find have any content for this URL it will send a different code, such as the famous 404 Page not Found. The browser needs to start parsing the response to figure out what to do next. That will in all likelihood include many additional requests by the browser to the same and possibly other servers to retrieve content that was referenced in the initial response, such as CSS and Javascript files. Every one of these additional requests involves all the steps from 1-6 AGAIN!

Step 7: Even while it is still waiting for the responses from these additional requests (and possibly even more pieces of the original request) to arrive the browser will start to figure out how to render the content that it has received on the screen. That means figuring out what to show where, which is made incredibly complex by the interaction between the HTML (roughly: the content itself), the CSS (roughly: the styling or look and feel of the content) and the Javascript (roughly: the dynamic behavior of the content). This work involves a so-called rendering engine and also a full fledged computer language interpreter (for Javascript).

Step 8: The browser continues to execute the Javascript code (which might, for example, animate an object to move across the page) while at the same time waiting for input from you. For instance, when you hover with the mouse above a link that might change the look and feel of that link. In the early days of the web, the most that would happen now is that a click on a link will start the whole process over at Step 1 for the next page. Today, however, many additional requests to the web server may occur without the page ever refreshing as new content is dynamically fetched and added to the existing page and other content written back to the server.

In the upcoming Tech Tuesdays we will look at each of these steps in some detail, starting with the anatomy of a URL next Tuesday. In the meantime, I hope I have managed to convey some of the amazing complexity that is involved in something that we now take for granted and people interact with billions of times every day around the world. And all along you should keep in mind that I haven’t even mentioned any of the complexity behind the scenes, such as the browser interacting with the computer’s operating system to make all of these steps happen.

The Internet Is a Human Right

Mon, 09 Jan 2012 08:00:05 -0500

At first, I was surprised to see a New York Times OpEd by Vint Cerf with the title “Internet Access Is Not a Human Right.” But once I started to read I began to understand the point that Cerf was trying to make. It comes out clearest in the sentence “technology is an enabler of rights, not a right itself” which in modified form is also part of our investment thesis at Union Square Ventures. We don’t invest in technology per se, but what that technology allows startups to build. That’s an important distinction. We don’t go looking for “mobile startups” but rather startups that use “mobile” to do something that wasn’t possible before.

Yet I think Cerf is selling the Internet short, which is ironic given that he is one of its co-creators. The Internet is not really a technology but rather a set of principles that have become embodied in a bunch of different technologies. I am going to quote at some length from a document that Cerf also co-authored about the history of the Internet:

The Internet as we now know it embodies a key underlying technical idea, namely that of open architecture networking. In this approach, the choice of any individual network technology was not dictated by a particular network architecture but rather could be selected freely by a provider and made to interwork with the other networks through a meta-level “Internetworking Architecture”

and

Four ground rules were critical to Kahn’s early thinking:
[1] Each distinct network would have to stand on its own and no internal changes could be required to any such network to connect it to the Internet.
[2] Communications would be on a best effort basis. If a packet didn’t make it to the final destination, it would shortly be retransmitted from the source.
[3] Black boxes would be used to connect the networks; these would later be called gateways and routers. There would be no information retained by the gateways about the individual flows of packets passing through them, thereby keeping them simple and avoiding complicated adaptation and recovery from various failure modes.
[4] There would be no global control at the operations level.

Those turned out to be a powerful set of principles that enabled the massive innovation that the Internet has brought about (for some more context you can also read my Tech Tuesday post on networking). These may seem like dry technical principles but embedded in them are some profound social implications. This is probably most obvious with the last of Kahn’s “ground rules” stating that “there would be no global control at the operations level.” While it has been pointed out that this may have been in part motivated by wanting to avoid a single point of failure/attack to create a network that might be sustainable even during or after a nuclear war, it also meant that decentralization was architected into the very heart of the network.

Black box routing too may seem like a solely technological concept. Yet it embeds within it an important separation of labor between various parts of the network that has had a profound impact on how innovation can take place and who wields power. In particular that ground rule is what has given us so far a fairly neutral network. And as longtime readers of this blog know, I am an ardent supporter of preserving that network neutrality and making sure it extends to wireless networks as well.

If you paid close attention, the headline for my post it is not the exact inverse of Cerf’s who wrote “Internet Access” - I simply talk about the “Internet” by which I mean a set of ideas that is grounded in these original principles behind the architecture of the Internet. At their heart all human rights are ideas and highly abstract ideas at that, such as equality and freedom. How we concretely instantiate these ideas through legislation and social norms has changed dramatically over time and much of that change has been driven by technology.

So when I claim “The Internet is a Human Right” I mean that the legislation and social norms that we use to operationalize abstract rights such as freedom of speech should be embracing not fighting the principles of the Internet. For example, freedom of speech will be a hollow right if movie studios can make entire web sites disappear off the Internet without due process, as is currently contemplated by the legislation known as SOPA. That is the exact opposite of the principle of decentralized control. To be clear, I am pretty sure that Cerf shares this view as he has come out against SOPA. That’s why I wish his OpEd had focused on the Internet as a set of ideas rather than a technology.