Tech Tuesday: Security in Startups

Last week in Tech Tuesday I asked for topics to write about in my series on technology in startups. There seemed to be a fair bit of interest in security, so here we go. First off a disclaimer. As with any general purpose advice, you need to think a lot about what it is you are trying to do. The security requirements for a bitcoin startup are vastly different from those for a social media one.

When you are just getting going you should treat security the same way as scalability: make sure you have the basics covered but don’t spend too much time on it as your bigger problem is to build something that people actually want to use. Again, please keep the disclaimer from above in mind though!

As it turns out even the basics still seem harder than they should for a lot of folks. Here is what I consider to be included: hashed paswords, SSL for all logged in users, safeguards against SQL injection and cross site scripting attacks, two factor auth or VPN requirement for web based site administration, key based auth for all server access (and limit dramatically who has server access), disciplined access to all cloud services.

One way to get a lot of the basics is through widely used web development frameworks. That comes with a *very* important caveat. Because those frameworks are widely used lots of people are looking for exploits and when a zero-day exploit is found you will be vulnerable and you *must* apply all security patches immediately and generally stay up to date with the framework.

For managing coud services access there are two promising startups: Meldium and Bitium. These are both relatively young and so might turn out to have their own security issues but they are a lot better than emailing cloud services passwords around or keeping them in Google Docs which is what a lot of startups are doing right now.

Bottom line: when you are just getting going be pragmatic and focus on the must have items. Once you start to grow though make sure not to neglect security — you will need to upgrade as you scale.

Tech Tuesday: Topics to Cover?

The current Tech Tuesday series is about technology in startups. I started the series out with posts on initially choosing and evolving your technology. Since then I have been writing about best practices for growing and structuring your engineering team, with separate posts on hiring and retaining engineers and one on remote team members.

Now that I am eight posts into this series, I am curious what questions readers may have that I should cover in upcoming posts. Here are some ideas to get things going: addressing security (when and how?), using frameworks (good, bad, indifferent?) and technical debt (what to do about it?).

Please ask away in the comments!

Tech Tuesday: Remote Engineers

I have been writing the last couple of Tech Tuesdays about hiring and retaining engineers. One of the questions that comes up a lot in that context is what to do about people who are in other locations. Is it a good idea to have remote engineers? There are people who seem to have near religious believes about this ranging from it can never work to it being the only thing to get anything done.

In practice I have found that pretty much anything can work if you support it with the right culture and systems. I have seen both success and failure with entirely distributed teams (everyone is remote) and with entirely centralized teams. So what do you need to do if you have some remote engineers? Much of the following applies to whether they work individually from home or from a satellite office.

You need to invest heavily in communication. Having some kind of realtime channel seems to help a lot and IRC still appears to be the best way to do that. But in addition to realtime you also need to spend time on communicating company strategy, goals, values.

Having people come and visit so that they can meet in person at least once in a while also makes a big difference. There is some sense of being connected that comes from having met someone in person that is still quite difficult to establish purely online. 

It is very difficult to handle remote engineers (or remote employees of any kind for that matter) if you have a face time culture where being in the office for long hours is how performance is judged. In order to make remote work you have to have good systems for tracking progress and measuring individual productivity.

The one thing to be super careful about is to avoid any us vs. them mentality emerging. If there is any sign of that whether between two offices or between office and remote employees you need to get to the bottom of it immediately. That can become very corrosive quickly and hard to recover from if it goes on for some time.

Finally, not everyone is good at working remotely. Some people need the structure of an office environment. Others need the in person social interactions. So if you recruit someone to work remotely make sure there are some indications in their personality or history that this will be a good fit for them. 

I would love to hear from readers what has worked or not for them in having remote engineers or a second office.

PS Speaking of hiring engineers, Ziggeo is also looking for engineering summer interns.

Tech Tuesday: Retaining Engineers

Last Tech Tuesday I wrote that the companies which do well on hiring engineers are the ones that “prioritize both hiring *and* having a great work environment.” The post went on to talk about some of the best practices in hiring. Today’s follow-up is about creating a great work environment for engineers.

There is obviously a lot that goes into creating a great work environment including overall company culture, office space, company mission etc. Instead of those more general points, here are some key dos and don’ts that are more specific to engineering

Dos

- Ship early and often - getting code into production is hugely motivating. In fact Etsy has new engineers deploy a small code change into production on their first day.

- Hold internal hack days - letting engineers work on something they really want to with some frequency will make it much easier to also get those fixes to the the backoffice systems done (or whatever else is the least exciting work in your specific case)

- Invest in training - engineers tend to want to improve their skills and knowledge so learning something new makes them both happy and also more productive (as does teaching so you can keep some of this in-house)

- Remove dependencies - this is not just about loose coupling from a technical perspective but also about having enough product management and design folks around so that engineering doesn’t have to do NOPs waiting for specs or layouts

Dont’s

- Treat engineering as order takers - engineers are not production machines or robots - they need to buy into the company’s mission and strategy and understand how what they are working on fits into that and be able to provide feedback

- Hang on to low performers - if someone is turning in sub par work again and again it becomes hugely demotivating to the rest of the team. Even if you feel resource constrained already it is better to let them go.

- Give star performers too much leeway - this one may be the hardest of all because it is easy to be afraid of pushing back for fear that someone might quit. But again if they take too many liberties eventually it becomes destructive for the cohesion and productivity of the team (I am sure someone with more familiarity with team sports could provide a great example here).

Like last Tuesday on hiring, I would love to hear from readers what Dos and Don’ts have worked well for you. And of course any and all comments on the ones that I listed above.

Tech Tuesday: Hiring Engineers

The last two posts on Tech Tuesday were both about growing your engineering organization. Someone wrote to me saying: it’s all well to talk about best practices for organizational design, but we can’t find enough qualified engineers in the first place! So today a post about hiring.

I have been able to observe many companies over the years and even in the toughest markets some have a steady flow of high quality candidates and others always seem to struggle. The main distinction between the two: companies that prioritize both hiring *and* having a great work environment are the ones that do well. Neglecting either one results in trouble with recruiting engineering talent.

Here are some Do’s in no particular order:

- Have an active internship program, ideally fed directly by professors you know or work with a group like the wonderful HackNY.

- Create opportunities for engineers to work on things they are passionate about (you can help that with internal hack days)

- Have an engineering blog and get engineers to participate in conferences

- Be present at relevant hackathons

- Hold meetups and have guest speakers at your offices

- Use in-house recruiters (start with part-time if you are small)

- Include both writing and reading of code in your evaluation of engineers

And here are some Don’ts

- Don’t set an artificially high bar for performance - you cannot build a balanced team out of “super stars” only

- Avoid puzzles, unless you use them very carefully and playfully

- Don’t hire only from one place or build a mono culture in other ways (eg all guys)

- Avoid mercenaries at all cost — only hire people who genuinely want to work at your company and on your set of problems

Would love to hear what else has worked for (or against) you in recruiting. I am sure I am forgetting a whole bunch of important ones!

Tech Tuesday: More Best Practices for Growth (Engineering)

Last Tech Tuesday, I wrote about growing your engineering organization and the week before that about evolving your technology. Today here are three more best practices for successful engineering growth:

APIs - I will have a lot more to say about how APIs relate to strategy in a separate post, but they are also a key internal tool as you grow even if not exposed to third parties. Having clearly defined APIs is what lets different parts of the engineering organization work with different technologies and at different rates of progress. To make that work even when used only internally the same key rules apply to the evolution of an API: make sure your API is well documented (eg using Apiary) and don’t break existing calls as you make changes.

Loose Coupling - sometimes people think just because they have defined services internally and use APIs they have achieved loose coupling, but the two are not the same. I have written a whole blog post before where I called loose coupling a board level issue. The key idea here is that parts of your system should be able to slow down or even fail without causing failure of everything. If all your API calls are synchronous then you don’t have loose coupling. As you grow, loose coupling becomes more and more important because you may encounter bottlenecks that you have not anticipated.

Engineering Metrics - having some measure of the throughput of your engineering organization is critical. One system that seems to work well is to assign some point measure to each ticket and make sure to break work items down small enough so that you only have say one, two and three point items (anything that would be more points will have to be broken down into subtasks). You then measure how many points are being achieved each development cycle. Cycle time should be short, ideally one week but no more than two. This will give you rapid feedback on how many points you are covering per cycle. You can keep track of it by team for additional granularity and in particular see how it changes as you grow the number of engineers.

If you have your own best practices that you believe are essential to growing engineering please let me know. I am thinking of compiling a list potentially at a separate web site.

Tech Tuesday: Growing Your Engineering Organization

Last Tech Tuesday I wrote about evolving your technology as you grow. As, if not more important, is how you grow your engineering organization. They after all are the ones who need to make the right choices about the technology! Here I have observed three common mistakes: staying flat too long, not investing early on in devops and missing the co-founder to manager transition (if such a transition turns out to be necessary).

Most startups begin with just a couple of engineers who tend to be enormously productive. Productivity continues to be great as you add early hires (assuming you are hiring well — more on that in a separate post). As you approach ten engineers the incremental productivity of new hires tends to degrade very rapidly. Why? Because a completely flat structure doesn’t scale. The instinct is almost always that people don’t want to create hierarchy so early on and that’s a good instinct. But you have to form teams. Teams that have team leaders and that work on clearly identified areas. Remember how productive you were when you got going? Forming teams of 3-4 engineers, one of whom is the lead will retain that. I will expand in a future post on why that is and how to make those teams work best.

For years now I have heard the same comment when asking about plans for Ops or Devops hires. It used to be “oh we are on managed hosting — they’ll take care of it” and then it became “we run on a cloud provider” or more recently “we are on platform as a service.” All of those have been great innovations for startups but they don’t eliminate the need for Ops. They take care of what people used to think of as system administrators quite nicely and you won’t need someone just to get your favorite OS to run on a particular machine but that is just some tiny aspect of actual operations. What are your build and test systems? Do you have a staging environment? Are you secure? Are you backing up and have you tested that? Are your systems properly instrumented for tracking bugs, performance, usage, etc? And the list goes on. Once you fall behind badly on any of these you will be fighting an uphill battle for a long time. So as soon as what you have built shows meaningful growth in usage you should invest in Devops. If you get to 10 engineers and don’t have someone who has this as their primary focus you are likely already behind.

Finally, technical co-founders sometimes hang on too long in running the engineering organization. I am a big believer in helping people grow on the job and just like our approach at USV is to help founders stay as CEOs of their companies we do the same on the engineering side. But here too, sometimes it works and sometimes it doesn’t. It’s hard to fathom when you get going just how different your job will be when you have twenty plus engineers from what it was when there were two or three of you. What you spend your time on and what you need to be good at will be radically different. Some technical co-founder turn out to be great at making that shift (either because they have done it before or because they pick it up quickly *and* enjoy it) — others don’t. When they don’t it often turns out to be a particularly rough and emotional transition. After all, it is often still a co-founder in the CEO spot who has to make that call.

So to phrase these in the positive, here are three critical things to growing your engineering organization at the early stage: Introduce a team structure early on, don’t forget about DevOps and make sure you either grow as a manager or transition to an experienced manager.

Tech Tuesday: Evolving Your Technology As You Grow

The last two Tech Tuesdays were about choosing technology as a startup. Today we are talking about what to do once you are up and running. Let me start by categorically stating what you should never, ever do: rewrite everything from scratch. There is a terrific piece on why this is a bad idea written in 2000 by Joel Spolsky that you should go off and read right now (Joel is a very entertaining writer).

Etsy early on made this mistake and it almost killed the company. When I first invested in Etsy, the marketplace was already up and running and growing nicely month over month. The team felt they had learned so much and were so unhappy with the existing codebase that they decided that Etsy V2 would be written from scratch. This meant that known bugs in the existing system weren’t fixed which is not good. And of course as with every rewrite ever V2 dragged on much longer than anticipated. The delays compounded the problem of the unfixed bugs as the marketplace continued to grow putting huge pressure on the team as sellers grew increasingly irate (at one point I had to provide “shuttle diplomacy” between Brooklyn and Jersey City). In the end, Etsy rolled out a V2 that was much stripped down from its ambitious plans, just in the nick of time for avoiding a full blown meltdown.

So what should you do instead? Rewrite piecemeal. The image that people sometimes use and is quite apt is one of rebuilding a plane while in the air. You have to pick pieces that are small enough so that you can replace them in flight. This approach seems less efficient because you will have to write the new pieces to fit with the old ones and also modify the old ones to make it work (even though you already know that you will be rewriting those next). And you may not be able to get from where you are to where you want to be until many iterations. Yet, based on everything I have seen over three decades of being involved with commercial systems (meaning systems deployed in a commercial rather than academic setting) it’s the only viable approach. In upcoming Tech Tuesday posts I will have more to say about best practices for doing this.

Tech Tuesday: Choosing Your Technologies (Continued)

Last Tech Tuesday I offered some very rough guidelines for making technology choices in a startup. I was going to write today about how to evolve that over time, but one of the comments suggested that it would be great to pick something that won’t immediately require a massive rewrite if you hit user growth. So a bit more today on this initial choice. I agree with the suggestion but with an important caveat: make a choice where you don’t sacrifice agility. At the earliest stages the single most important technology feature you need is the ability to make rapid progress and make changes quickly.

Until very recently you were faced with a big tradeoff here: if you wanted agile you were likely making a big scalability tradeoff. Why is that? Because all the frameworks that were developed to support agile ran on top of relational databases with a “magic” ORM layer in between (object-relational mapper). The magic of an ORM works well at small loads but tends to have poor scaling characteristics. Now just to be clear here because I anticipate a fair bit of protest — your mileage may vary greatly. Modern hardware is crazily fast and you can get a long way with a beefy machine and with SSDs. But the further you go with that kind of setup the harder the eventual changeover is likely to be. In fact, so many of the companies in our portfolio and beyond had that experience that about 5 years ago I wrote a post titled “I Want a New Platform.”

Thankfully there is now an alternative. New databases and data stores have been developed (and continue to be developed) to allow for both agility at the start and very significant scaling. Collectively this group of databases has become known as NoSQL which is an unfortunate moniker that stuck. I am of course partial to MongoDB in this group as we are investors in 10gen, the company behind MongoDB. But depending on your specific application you might also want to look into Cassandra, HBase, Neo4J or DynamoDB. What all of these have in common is that they are schema free which eliminates the need for an ORM. In fact in many scripting languages you can persist or retrieve objects in a couple of lines of code.

The combination of scripting languages and NoSQL stores is a very powerful one. It gives you both agility and a much wider range of scaling. That’s not to say you might not eventually need to rewrite code to make it scale, but you will be able to get much much further and for many startups you might never need to resort to the highly custom code of a Twitter, Facebook or Google scale company.

Tech Tuesday: Choosing Your Technologies When You Start

This is the first of a new series of Tech Tuesday posts on Technology in Startups. Today’s topic: choosing your technologies. As a startup when you are just getting going, you have the luxury of picking any technology you want — you don’t yet have a legacy set of choices to contend with. It’s completely greenfield and that level of choice can be both exhilarating and daunting (what if I pick the wrong technology? cf Paradox of Choice). So what are some guidelines for thinking about that choice?

The most important initial consideration is the nature of innovation for your particular startup. Is your innovation primarily a social/behavioral/market structure one (think Instagram, Tumblr, Etsy) or a technical one (think v1 of Google, Twilio)? If it is the former you should tend to go conservative in your technology choices whereas in the latter you must pick something risky.

Why? Because risk is in startups is compounding/multiplicative rather than just additive. If you are already taking market or behavioral risk you shouldn’t also take technology risk (if you can help it). But in the case of early Google people were already searching or Twilio sending and receiving calls/texts and the companies innovation instead was a breakthrough way of doing that which requires taking technology risk to deliver that breakthrough.

Now there are at least two important footnotes required to that basic rule:

1. For social/behavioral/market structure startups: Your choice of technology also impacts your ability to recruit talent. If you go too conservative you might have some trouble getting the most talented engineers to work with you. You will need to decide how important a problem that is in your specific case keeping in mind that if you succeed and you will become attractive by virtue of scale. This is also an area where having a strong mission can help early on.

2. For technical startups: Riskier doesn’t necessarily mean more cutting edge base technologies it could simply mean more custom code in a well established technology (e.g. C). For instance, if you are creating a new database as in the case of 10gen your technology risk is writing a lot from scratch and so you probably don’t want to do that in a language that just emerged and for which tooling is still lacking and performance characteristic are poorly understood.

One additional consideration that may matter in all of this, including the original choice of what kind of startup to have in the first place, is the degree of technical competency of the team. If the founding team is more business and design oriented then all the more reason to go for a social/behavioral/market structure startups and make conservative technology choices. Next Tech Tuesday: how to evolve your technology once you are up and running.