Last Tech Tuesday I wrote about how engineering led companies sometimes underrate the importance of marketing and sales. One of the comments on Twitter pointed out that the converse is also often true and I replied that I will write a post about that, so here it goes. Many companies founded by a pure business team tend to struggle with technology and fail to deliver the products or services they envision. There are at least three distinct failure modes that I have seen here over time.
The first are teams that outsource their initial technology development entirely. By this I mean that they don’t have a truly technical person on their own team. This hardly ever works. I am sure people will post examples of successful counter examples but most of the teams I see who fall into this category are struggling mightily. Some of them fail to ever ship a product. Others ship something but it is delayed and a far cry from what they had in mind. Based on lots of experience my clear and unambiguous advice here is: if you don’t have a technical member of your founding team stop everything else you are doing and find somebody (I will have a separate post on how to do this). In many situations you would be better off not continuing at all if you can’t do this and instead take time out to learn enough technology yourself to build an initial version.
The second are teams that add a technical person early on but always treat engineering as a substitutable appendage to the company cycling through many engineering leaders and accumulating massive technical debt along the way. Things may look like they are going well initially, but over time the product development cycle for companies like this starts to slow down dramatically as the technical debt makes innovation harder and harder. Churn in the engineering organization also means that there is less institutional knowledge about why things were built in a certain way to begin with.
The third are teams that bring on board a technologist but then proceed to let that person build an engineering fiefdom that’s disconnected from the rest of the organization. The failure mode here tends to be an over-engineered solution that invests heavily in technological infrastructure before having found product market fit. There may be no technical debt here in the classical sense but it generally won’t matter because many of these companies never really get off the starting blocks.
The root cause in pretty much all of these situations is a lack of understanding of technology. It is unfortunately easy to go through business school and/or work on the business side of startups or large companies without ever developing any meaningful understanding of technology. That, however, is a recipe for mismanaging technology as it will translate into a fear of engaging and an inability to ask the right questions. If that happens to describe you, then you might want to go and read Tech Tuesday from the very beginning.
Today’s Tech Tuesday is a relatively short note on the “Bleeding Edge” — technologies that are even beyond the cutting edge. Every technology tends to start life at the bleeding edge with early releases that are unstable and poorly tested, many different ways of doing something (instead of an agreed upon best way) and with upgrades that are not backward compatible. It is tempting for startups to embrace bleeding edge technology for several reasons: first, you are starting from scratch, so why not use the very latest? Second, if it works out well it can give you a competitive advantage. Third, it can be a recruiting tool for people who want to get on the next technology trend.
Before you adopt the bleeding edge though, make sure you understand the downsides. In the worst case scenario you find yourself sitting on technology that is no longer supported at all. That will leave you with a tough decision between maintaining it yourself or migrating away. Even in the best case though you are likely to face changes that come from the external technology that force you to rewrite your code (because the changes are not backward compatible). There too you face a tough choice — if you stop upgrading you avoid any rewrite but in all likelihood you are not stuck with whatever version you have which at some point (often very soon) will no longer be maintained.
As a general rule then I suggest to folks that unless you absolutely definitively need it, wait to adopt a technology until it has started to settle down a bit with the vendor (or vendors) reasonably firmly established, the number of releases slowing down considerably and — most importantly — a commitment to avoid backwards incompatible changes unless absolutely necessary. The node.js project for instance seems to be settling down (there are drawbacks to this too as people who were hoping for a different module system were quick to point out), but in terms of adoption this is a big plus.
One of the questions asked by a Tech Tuesday reader was about using multiple different programming languages for different parts of a site. Back in the early days of the web that wasn’t really an issue because you would write the page generation in something like Perl or TCL and that was all there was. Well, not really because even then you had to know HTML but of course early HTML was so minimal that it didn’t really matter much.
Having said that, I think the prime directive should be to use the right tool for the job even if that results in multiple languages. The benefits from a proper match will ultimately outweigh the added cognitive and tooling overheads. Also, you should be very suspect of any engineer who has only ever programmed in one language and does not feel comfortable or interested in picking up a new one.
PS Please ask more questions!
PPS Minimal links today but I am running to catch a flight!
The topic fo today’s Tech Tuesday is one that every startup I have ever seen struggles with: what to do about technical debt. First off, what is technical debt? The Wikipedia page explains it as “the eventual consequences of poor or evolving software architecture and software development” which result in more effort for subsequent work.This is actually quite good because it makes clear that technical debt doesn’t just result from mistakes (the “poor” part) but also from “evolving.” And as you may recall my advice has been to startups to evolve their technology as they grow.
The direct corollary from the above is: every startups has technical debt. And I mean every. Show me a startup that doesn’t have it and I will show you a company that has vastly over-invested in engineering relative to their scale and relative to product market fit. So if every startup has it, the question becomes, what should you do about it? The answer here is not unlike what you should do with financial debt: don’t try to pay it all back at once but also don’t ignore it and let it pile up further.
Why not pay it all back at once? First, because often that will result in the temptation to do a complete rewrite, which as I wrote before is a very bad idea. Second, even short of a complete rewrite trying to pay it all back will tend to slow down development of new features or even plain old bug fixing too much relative to the needs of customers (and hence the business). The best way to think of it is that there is some natural level of technical debt that exists at all times.
Of course you shouldn’t ignore technical debt either and especially not let it continue to pile up. If you do that, eventually making any change to the system gets so hard that your product becomes frozen. So instead what you need to do is allocate some fraction of engineering at all times to paying back some technical debt. This is one of the key reasons why your total engineering productivity (as measured from a user visible standpoint) will never be as good a year or two into the life of a startup as it was in the early days.
What is the right fraction? The answer is of course: it depends. But as a broad rule if you are a couple of years into it and you are not spending at least about a third of engineering time on rewriting some code (paying back technical debt) you are probably in denial about the extent of your technical debt. And conversely if you are spending more than two thirds of engineering resources on a sustained basis you are probably being overly aggressive.
If you have great examples of different types of technical debt that you have encountered, I would love to hear them. And of course if/how you managed to deal with them.
I still have quite a few great topics suggested by readers to work off for my Tech Tuesday series on technology in startups. But today I want to take on a different issue because it has been on my mind. I have now been programming computers in one way or another for over thirty years (yikes) and have seen a lot of technologies come and go during that period. So when something new comes along I tend to think of the great French saying: “Plus ça change, plus c’est la même chose” (the more things change the more they stay the same).
One of these constants has been that new technologies all too often wind up being seen as a silver bullet: something you can use to solve a difficult problem (werewolves, server scaling) with a solution that is simple and yet super effective. Unfortunately, technology silver bullets are as mythical as their werewolf counterparts. That’s not to say that we haven’t made progress and that every once in a while something that was previously hard becomes easy (or at least easier) but those are few and far in-between.
For a while recently people were treating node.js as a silver bullett and now it appears to be Go. So you wind up hearing things like “our xyz isn’t scaling so we should rewrite it in Go.” Now for some cases it may well be true that you need to rewrite a portion of what you have and use a tool that gives you more speed. But before you jump from there to the conclusion that you need the silver bullet du jour keep in mind that people have been writing high performance systems for a long time in existing languages.
So almost inevitably the answer is: you need to either restructure your existing team and/or hire some new folks. When you do that beware of anyone claiming to be an expert in a specific language or technology, especially one that is less than a decade old. If you need say a super low latency in memory server find someone who has written a bunch of those in whatever language rather than a self-declared expert in node or Go.
PS I spent a couple of hours over the weekend looking at Go and it does indeed seem like a promising alternative for situations in which you might have previously resorted to C or C++.
Last week in Tech Tuesday I asked for topics to write about in my series on technology in startups. There seemed to be a fair bit of interest in security, so here we go. First off a disclaimer. As with any general purpose advice, you need to think a lot about what it is you are trying to do. The security requirements for a bitcoin startup are vastly different from those for a social media one.
When you are just getting going you should treat security the same way as scalability: make sure you have the basics covered but don’t spend too much time on it as your bigger problem is to build something that people actually want to use. Again, please keep the disclaimer from above in mind though!
As it turns out even the basics still seem harder than they should for a lot of folks. Here is what I consider to be included: hashed paswords, SSL for all logged in users, safeguards against SQL injection and cross site scripting attacks, two factor auth or VPN requirement for web based site administration, key based auth for all server access (and limit dramatically who has server access), disciplined access to all cloud services.
One way to get a lot of the basics is through widely used web development frameworks. That comes with a *very* important caveat. Because those frameworks are widely used lots of people are looking for exploits and when a zero-day exploit is found you will be vulnerable and you *must* apply all security patches immediately and generally stay up to date with the framework.
For managing coud services access there are two promising startups: Meldium and Bitium. These are both relatively young and so might turn out to have their own security issues but they are a lot better than emailing cloud services passwords around or keeping them in Google Docs which is what a lot of startups are doing right now.
Bottom line: when you are just getting going be pragmatic and focus on the must have items. Once you start to grow though make sure not to neglect security — you will need to upgrade as you scale.
The current Tech Tuesday series is about technology in startups. I started the series out with posts on initially choosing and evolving your technology. Since then I have been writing about best practices for growing and structuring your engineering team, with separate posts on hiring and retaining engineers and one on remote team members.
Now that I am eight posts into this series, I am curious what questions readers may have that I should cover in upcoming posts. Here are some ideas to get things going: addressing security (when and how?), using frameworks (good, bad, indifferent?) and technical debt (what to do about it?).
Please ask away in the comments!
If ever we needed a reminder that technology can both make our lives better and more terrifying this week provided plenty of them. Pressure cookers for cooking and bomb making. Fertilizer for agriculture and explosions (sadly also used in the Oklahoma City bombing). Social media for collaborative (re)search and public witch hunts. This is why we need values. Without being guided by strong values we will not enjoy the benefits of technology but will be leaving in fear of it instead.
The last two Tech Tuesdays were about choosing technology as a startup. Today we are talking about what to do once you are up and running. Let me start by categorically stating what you should never, ever do: rewrite everything from scratch. There is a terrific piece on why this is a bad idea written in 2000 by Joel Spolsky that you should go off and read right now (Joel is a very entertaining writer).
Etsy early on made this mistake and it almost killed the company. When I first invested in Etsy, the marketplace was already up and running and growing nicely month over month. The team felt they had learned so much and were so unhappy with the existing codebase that they decided that Etsy V2 would be written from scratch. This meant that known bugs in the existing system weren’t fixed which is not good. And of course as with every rewrite ever V2 dragged on much longer than anticipated. The delays compounded the problem of the unfixed bugs as the marketplace continued to grow putting huge pressure on the team as sellers grew increasingly irate (at one point I had to provide “shuttle diplomacy” between Brooklyn and Jersey City). In the end, Etsy rolled out a V2 that was much stripped down from its ambitious plans, just in the nick of time for avoiding a full blown meltdown.
So what should you do instead? Rewrite piecemeal. The image that people sometimes use and is quite apt is one of rebuilding a plane while in the air. You have to pick pieces that are small enough so that you can replace them in flight. This approach seems less efficient because you will have to write the new pieces to fit with the old ones and also modify the old ones to make it work (even though you already know that you will be rewriting those next). And you may not be able to get from where you are to where you want to be until many iterations. Yet, based on everything I have seen over three decades of being involved with commercial systems (meaning systems deployed in a commercial rather than academic setting) it’s the only viable approach. In upcoming Tech Tuesday posts I will have more to say about best practices for doing this.
Last Tech Tuesday I offered some very rough guidelines for making technology choices in a startup. I was going to write today about how to evolve that over time, but one of the comments suggested that it would be great to pick something that won’t immediately require a massive rewrite if you hit user growth. So a bit more today on this initial choice. I agree with the suggestion but with an important caveat: make a choice where you don’t sacrifice agility. At the earliest stages the single most important technology feature you need is the ability to make rapid progress and make changes quickly.
Until very recently you were faced with a big tradeoff here: if you wanted agile you were likely making a big scalability tradeoff. Why is that? Because all the frameworks that were developed to support agile ran on top of relational databases with a “magic” ORM layer in between (object-relational mapper). The magic of an ORM works well at small loads but tends to have poor scaling characteristics. Now just to be clear here because I anticipate a fair bit of protest — your mileage may vary greatly. Modern hardware is crazily fast and you can get a long way with a beefy machine and with SSDs. But the further you go with that kind of setup the harder the eventual changeover is likely to be. In fact, so many of the companies in our portfolio and beyond had that experience that about 5 years ago I wrote a post titled “I Want a New Platform.”
Thankfully there is now an alternative. New databases and data stores have been developed (and continue to be developed) to allow for both agility at the start and very significant scaling. Collectively this group of databases has become known as NoSQL which is an unfortunate moniker that stuck. I am of course partial to MongoDB in this group as we are investors in 10gen, the company behind MongoDB. But depending on your specific application you might also want to look into Cassandra, HBase, Neo4J or DynamoDB. What all of these have in common is that they are schema free which eliminates the need for an ORM. In fact in many scripting languages you can persist or retrieve objects in a couple of lines of code.
The combination of scripting languages and NoSQL stores is a very powerful one. It gives you both agility and a much wider range of scaling. That’s not to say you might not eventually need to rewrite code to make it scale, but you will be able to get much much further and for many startups you might never need to resort to the highly custom code of a Twitter, Facebook or Google scale company.