Upgrading from Postgres 9.2 to 9.3 (and my troubles with Heroku)

(Cross-posted with Holger’s blog)

It is no secret that we build launchd.io using Ruby on Rails and are running it on Heroku. It was the fastest and easiest way to get the platform up and running one year ago. But while the experience of using Ruby on Rails has been nothing short of transformative, my experience with Heroku has been a decidedly mixed bag.

From the looks of it a Platform-as-a-Service (PaaS) is just brilliant. It takes away all the pain of managing your deployment on an Infrastructure-as-a-Service (IaaS) while preserving the inherent scalability. And Heroku makes on-boarding super easy. Literally signing up and a ‘git push’ and you are on your way. But baked into a PaaS are certain mental models and assumptions about how your app is suppose to be like. Maybe call it best practices, but in the end it might just be that “my best practices are your worst nightmares”. Our first rookie mistake cost us 2 weeks in launching launchd.io. We had simply forgotten to fix the version numbers in our Gemfile. And since half a year had passed from starting to code to actual launching, the initial deploy to Heroku resulted in countless gem upgrades which ultimately broke our app. Fine – lesson learnt (since we have day jobs to attend to, the resulting forklift upgrade cost us 2 weeks.). Then – a couple of months later – came a system wide upgrade of Rails (or was it Ruby) which cost us another week or so. And so on and so on.

Remember, the initial promise of a PaaS was that we did not have to worry about the infrastructure  – yet the reality of running a SaaS platform on a PaaS is that you still need to care, just at a higher level. True, I don’t need to provision machine instances and load balancers, but now I need to worry about forklift upgrades of frameworks I am relying on (which force me to deal with version incompatibilities of something which worked previously when I should be focussing on new features for my customers). This does not even mention problems with the Rails asset pipeline apparently working differently on my local dev system vs in Heroku.

What motivated me to this column was the latest episode of such an upgrade. We had exceeded the free database tier and were forced to upgrade to the first paid tier. Except that the free tier was Postgres 9.2 and the paid tier was Postgres 9.3. I need to add that we have wrapped our deployment process in a script which backs up our customer database both via the ‘pgbackup’ plugin of Heroku and to local file. After pushing the latest version of our code repo we would reset the database and restore from backup. This allows us to extend existing database schemas “in place” rather than keep adding consecutive migration files (I really dislike a directory with lots – and I mean lots – of files). Unfortunately the pg_dump for PG 9.2 will not work with a PG 9.3 instance – so for all sense and purpose our deployment process was broken after we switched to the paid tier. And sure – you could argue that it was our choice to structure our deployment process in this way, but guess what – this is exactly what I mean by “every PaaS has certain practices and mental models built in”. You follow them – your are fine. You break them knowingly or unknowingly – you are in for a lot of pain (and guess what, most of the baked-in assumptions are not even documented anywhere). To tie it back to my day job – PaaS are likeAPI SDK’s – they work until they don’t.

But fair enough – with a university wanting to use our platform for their entrepreneur course and certain features and bug fixes I needed to get in before the deadline, I had to upgrade my local environment from Postgres 9.2 to 9.3. I did not want to risk a single byte of data of my customers and was not willing to drop the ability to create a backup of the data locally.

As you can imagine (why would I write this blog otherwise), the upgrade itself turned a bit into a nightmare. Even though it was suppose to be the most simple of all – I had no data locally to preserve – simply dropping PG 9.2 and installing PG 9.3 should have been sufficient. Except I struggled to get PG 9.3. to work with my local Rails installs for a couple of long nights (remember, I have a day job). So if you think you did everything right upgrading to PG 9.3 but encounter the following problem

could not connect to server: No such file or directory
Is the server running locally and accepting connections on Unix domain socket
"/var/run/postgresql/.s.PGSQL.5432"?

I can hopefully save you a few hours by pointing out that PG 9.3 appears to have changed its default port from 5432 to 5433. The corresponding configuration file

/etc/postgresql/9.3/main/postgresql.conf

#——————————————————————————
# CONNECTIONS AND AUTHENTICATION
#——————————————————————————

# – Connection Settings -

listen_addresses = ‘localhost’ # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to ‘localhost'; use ‘*’ for all
# (change requires restart)
port = 5433 # (change requires restart)
max_connections = 100 # (change requires restart)

The TCP port shows up as part of the UNIX domain socket filename. So if your Postgres instance is listening on 5433 but your Rails app is expecting it on 5432 (Rails will default to 5432 if you do not specify Port in “config/database.yml”) you will get above error. I ended up adding both the Port directive to my database.yml file and configuring Postgres to listen to port 5432. This – btw – also allowed phpPgAdmin to connect to the database again since it was also implicitly defaulting to port 5432 (it had shown a ‘Login Invalid’ error when it could not connect).

Here are the links which helped me to get a working Postgres 9.3 instance up and running:

So what are my conclusion? Well – I am already knee deep into moving our platform into Docker, well Vagrant + CoreOS + Docker to be precise. First with our development setup and then onto Amazon for production. Ultimately control over my runtime environment in an IaaS trumps the backloaded convenience of a PaaS. I will keep you posted.

Lean (Entrepreneurship) Reading

Photograph by Victoria J Baxter/Getty Images

Photograph by Victoria J Baxter/Getty Images

Vacation time is upon us. And if you are like me, you will have an impossible list of books you would like to read.

But rather than telling you all the books I still want to read, let me share with you my list of books I have already read and can recommend for one reason or another. They all share a common thread – they are all exploring ‘Entrepreneurship”. Both inside of established companies or in startups.

The term Lean Startup was first publicized by Eric Ries in his Book with the same name. (Cashing in on the popularity he subsequently trademarked the term). In his book he is describing his experiences of implement the ‘Customer Development’ process from Steve Blank. Steve Blank was a member of the board of his startup at that time.

Which brings us to Steve Blank. He is teaching Entrepreneurship at the Stanford University and has formalized his experience as entrepreneur and mentor in a process called ‘Customer Development’. He published the process the first time in his book ‘The four steps to epiphany’. It is still a worthwhile read. Steve Blank subsequently incorporated the works from Eric Ries and Alexander Osterwalder into his own thinking and published ‘The Startup Owner Manual‘ last year.

Alexander Osterwalder is best known for his ‘Business Model Canvas‘ which he subsequently publicized in his book ‘Business Model Generation‘.

Based on the Business Model Canvas Ash Maurya developed his ‘Lean Canvas‘ and published ‘Running Lean‘.

Unfortunately (or fortunately depending on your viewpoint) the above approaches are heavily influenced by the Silicon Valley startup culture and have only limited value within established companies.

In those kind of circumstances I found the work from Clayton Christensen on his ‘theory of disruptive innovation‘ highly relevant. His classic ‘The innovators dilemma‘ is still a fascinating read. But beyond just analyzing the root causes of why well managed companies fail, he proposes an alternative framework for dealing with disruptive innovation in ‘The innovators Solution‘. If you are an intrapreneur or managing an inhouse-incubator, this is the one book you ought to read.

Interestingly only Clayton Christensen is referencing one of the foundational works on which Lean Startup/Customer Development appears to be based upon – the so called ‘Emergent Business Strategy‘. It is building upon research by Rita McGrath (and others) at the Harvard Business School. Rita McGrath and Ian McMillan published a first paper on ‘Discovery-Driven Planning‘ in summer of 1995. They further refined their work in the book ‘Discovery-Driven Growth‘.

Since there is now an almost inflationary ‘Lean’ hype, it often pays to go back to where ‘Lean’ started – to ‘Toyota’. While ‘Lean’ is mostly associated with the production processes, I personally find the book ‘The Toyota Product Development System‘ more relevant. It is an inspiring read, even though most of its concrete practices are rooted within the Toyota culture and can not be applied literally in other settings.

To step outside of the ‘Lean’ hype, there are books which describe very similar approaches without using the word ‘Lean’ at all. The book ‘If you build it will they come‘ by Rob Adams describes a process very similar to Customer Development, but from a marketing perspective.

And while the hype is all about startups and venture funding I have a special fondness for the little guy, the SMB market. I think Tim Berner ought to be included in any kind of review of books and tools for entrepreneurs. What is makes the SMB market special is that it is not funded by  VC money, but instead relies on bank credits. And no bank is willing to give you a credit for a ‘Business Model Canvas’, but instead requires an old fashioned business plan. Tim Berner has been pushing on combining the traditional business plan with the kind of agile, discovery-driven planning processes mostly characterized as ‘Lean Startup’. I found his book ‘The plan-as-you-go Business Plan‘ a refreshing reminder of the other 95% of businesses which do not have the luxury to burn through VC money but have to make ends meet the old-fashioned way.

if I can convince you to read at least one of the books from my favorite Lean (Summer) Reading list, then this blog will have been worth it. I would love to hear from your what other books you have found and why you would recommend them.

we are live (launchd.io that is)

I think every entrepreneur can immediately understand what it means to say ‘we are live’. It means a bottle of Champagne sometime around 1 am (always way later than you had planned) and lots of more coffee the next morning. Our moment of glory was sometime during the night from July 11th to the 12th. Almost two years after conceiving the idea of a Business Plan as-a-Service platform at the Old Town Coffeehouse in Louisville, Kentucky and more or less 9 months after we started finally putting code towards making it a reality, we are finally ready to learn if our idea can fly.

We had quite a few up and downs along the way. We were shopping it around for close to a year without finding anyone willing to do it for us (lesson to be learnt – if you won’t do it, no one else will).

But we had this itch which kept us going. We had this sometimes overwhelming clear vision of how we would use http://launchd.eu to develop all those other cool business ideas we had. It was us who wanted to have this platform. It was never just about launchd, but about what it would enable us to do.

Maybe now is the time to say what we envision launchd to be:  a platform for entrepreneurs and companies to develop business plans.  CRM meets Project Management meets Business Model canvas meets Market Validation. Got it?

Let me try that again: When we first started to look into developing business ideas, we quickly found ourselves using Teambox (very cool), Excel spreadsheets (heck, there is nothing you can not do in Excel), Dropbox, Wiki, Word documents, Google docs, Survey Monkey and so on. At some point it just became too much – and we thought that there must be a better way to bring all those disjunct pieces of information together. Around this time we stumbled upon Customer Development and Lean Startup.  And suddenly everything fell into place – at least in our minds – and we knew exactly how our dream ‘business development’ platform should look like.

So this it .. we did it. And I think it is only appropriate in this first blog to give thanks to my wife Gabby for enduring my sleep deprived crankiness and to Nele, the wife of my co-founder Jonas for doing very much the same.