Maybe I could also call this "what was i doing for the last three-and-a-half years?" Today could really be described as the final day of Camvine's existence, even though I left at the end of December (something I would rather not have done, I'll explain later) this is something immensely dear to my heart, so I thought I would weave my thread through it's patchwork. So, what was Camvine? It was an Internet connected digital signage system where you could control your displays anywhere around the world through your web browser.
I was working at DisplayLink, a company set up by Quentin Stafford-Fraser, but wasn't enjoying my work, so I was looking at startups for a good challenge and found a startup founded by Quentin Stafford-Fraser... The weird thing was I had never met him, but thought it would be an interesting challenge. I still have no idea what made them take a chance on me, although my profile on the website called me a veteran, which definitely made me feel old. I think one of the major factors was that I at least had an idea of how the hardware worked, you see the first product of CodaBox + CodaLinks was based on the ethernet nivos from DisplayLink. They were an interesting proposition, very low power and low cost, but also not massively capable pieces of hardware, so one small low powered Intel Atom powered central machine could run up to 20 displays.
So I joined Camvine mid-2008 and I ended up working in "The Shed" which is a studio at the bottom of a garden in Cambridge, where ndiyo and DisplayLink also started. It was my first chance to do commercial Python development and it was an exciting free-flowing environment. We even had Fun Fridays where we could work on stuff not on the roadmap, quite a few cool features came out of that single day of the week, unfortunately some did not get integrated like transitions code I had working quite sweetly on the CodaLinks. The website was really awesome evolving into a drag and drop user interface for controlling your displays and content modelled on the iTunes style UI.
Around that time my wife fell pregnant with our first child, you would think that startup life and children do not mix, but I went head-first into it all and it was immensely rewarding.
Soon, we are heading towards a "The Pivot"... The team needs introductions, we have Quentin Stafford-Fraser, CEO, ideas, maverick, like Hannibal in the A-Team. We have Michael Dales, CTO, uber-hacker. Thomas Hunger, developer, the brains. Sarah McKeon, the office manager, the glue that holds the company together and keeping it running (every company needs a Sarah). And me, lowest rung on the ladder.
Not long after me a sales and marketing director joined the company whose expertise was in the advertising arena, so we started to target advertisers. Whilst CodaLinks were in interesting esoteric proposition for communicating they were not what the advertising industry craved which is tons of rich content (read: Flash) and high definition videos. The advertising industry and digital signage are not a good place to be, but I'll come onto that. So now we hit the pivot point around April 2009, which incidentally is the month my daughter was borrn. We grab some Asus EeeBoxes and decide to use an Intel Atom N270 based system and start writing code for it (partially based on porting our playback code for the CodaLinks). I start hitting the base platform with coding hammers (we were based on Ubuntu so we stauck with that) and we ended up with a Jaunty franken-distro set up mainly by me and Thomas. By September we are shipping EeeBoxes to a potential customer needing to get rid of an unmanageable legacy system, the weird thing was I had this Acer Revo sitting on my desk and I couldn't get the damn thing to boot Linux on a USB key, then somehow I worked out how to make it work by altering the heads/sectors on the key (I'm sure I blogged about it at the time). It had an nvidia ION as the GPU which was very interesting, for a start thhe binary drivers were much better than the Intel ones at the time (I was one of the original users of the KMS drivers trying to get it all working), also it had the ability to do video decoding on the GPU. Because the driver quality was so good it blew the Eeeboxes out of the water with the same CPU, so this became our hardware platform in what seemed like overnight, unfortunately we did have to write off about 20 Eeebox units at the expense of the company, embarressing on my part, but totally the right decision.
Sales and marketing decided the new product should be called the CODApod, urgh, horrible name! Internally I called it the incredicoda which not only sounding quite unhinged is really easy to grep for in source code, as a developer this is important.
The advertising industry is rubbish for digital signage, they want everything perfect, which is surprisingly difficult to achieve. You end up having to trade things on multiple axes which are robustness, speed, quality of output, and accuracy. You can throw more staff at solving these issues but that is not an option at some startups like this running on a shoestring. One of the things you can't explain no matter how hard you try is that displaying a web page is not a no=op, you have to download resources, start rendering, get some more resources, render some more, then you might be complete ready for display. Even with everything cached in memory that is not a no-op.
The most important thing to get the software shipped just 5 months after deciding on this was a robust software update, then things could be fixed in the field remotely. I had made software update more robust since joining Camvine, and continually strived to make it unbreakable. We evolved the software over the next six months into something quite impressive with hardware accelerated codecs (we paid Fluendo to finish development of VDPAU), newer, better web browsers and robust PDf viewers. Unfortunately a bombshell hit in December 2009 when Thomas Hunger decided to leave for the bright lights of London. The months that followed were a blur, me and Michael were support and development, answering the phone and support requests as well as me developing the client hardware and him on the website. It was simultaneously awesome and horrible, horrible volume of work, but I think we were a well oiled machine with just two people in the shed. We stuck our headphones in and got our heads down and moved forward as well as maintaining all that was there.
Some more people joined, Quentin stepped down as CEO, and we got a new CEO. The next bombshell for me was losing Michael as CTO. I was profoundly gutted, not only was it amazing to work with him, I consider him a good friend. His reasons for leaving, I will leave up to him, but losing him as CTO and protector of the Camvine faith is the worst thing to ever happen to the company. It took me a while to realise that he should have become CEO (although obviously he would not want that job), but that would have been what was right and proper for the company. I had a crazy idea that I should take over website development and hand CODApod development over to someone else, since I knew the most about the system and Camvien was proving to be very poor at hiring, even not allowing me to advertise on some of the places I wanted to. Michael left October 2010, and the handover was an unmitigated disaster. I was forced to work on attempting to make a playlist perfect for an advertising potential customer, who was never going to put their money where their mouth was. I was doing this instead of doing a decent handover, luckily Michael was still available one day a week.
I got my head down and obsessed about the website. Christmas was a nightmare because I was now the only thing between our website and total oblivion, especially with issues like the server going up to load 30 at 8:30am every morning. I ended up by March 2011 restructuring pretty much all the deployment to use nginx, gunicorn, and Fabric, the tools of the modern Django coder. The databases were split onto new machines, the load balancing worked pretty great.
Unfortunately, the CODApod was essentially dead. The company had decided that they wanted a Windows CODA player, so this would now suck two man-years of effort into a black hole as it was never officially released. During this time the only development on the CODApod was what I did in what i laughably called my "spare time", also the CODApod software had to be ported to work on as many hardware platforms as possible (ATi, nvidia, Intel Core) where there were no scheduled developer resources. I was doing all this whilst also running and developing and being on-call for the website. I felt really sorry for the customers knowing no client side developments were happening that could improve their hardware further. Somehow I managed to have a release of CODAsoft available two months after starting work in my spare time, and I kept it up to date until the day I finished. It worked flawlessly on Intel graphics (945 through to I think Ivy Bridge), ATI (open-source and binary drivers with hardware acceleration) and nvidia (everything the binary driver supports and with hardware acceleration).
We left the shed in 2011. A very sad day for me. The weird thing was the physical seperation matching the logical seperation of the company actually was a great thing that was not appreciated by some people. By moving into one huge space things started to bleed into one another and there was an immense lack of self-control by certain roles.
The moment I knew I had to leave Camvine completely and utterly was Father's Day 2011. We were out in the middle of nowhere with me, my wife, daughter and father-in-law, and there was basically no phone reception. I got an odd phonecall and discovered the servers were down and attempted to debug what was going on through a phone call and intermittent data coverage. Basically we were screwed. For a bit of context, I was forced to outsource database management to someone else in order to alleviate my workload, and we ended up with two SSD based servers with master-slave, back-ups and database dumps every day. Everything failed, but it had been failing for a month. Hard drives? Broken serving corrupted data silently. Backups of the HDs? Failing silently. Database dumps? Failing as well. What happened was they decided to upgrade a backup program which caused the mysql process to restart, which is where all hell broke loose. I managed to get in the office at 5pm Sunday and was there reconstructing enough of a business to open doors again the Monday morning otherwise we should have just shut up shop. I have no idea how I did it, but enough was workking by 1am and Quentin was great moral support. The next two weeks of 14 hour days involved recreating as much data as possible which was almost everything, which was something to at least be proud of. The thought of this still makes me feel sick. The write rate of the web service was blamed as it was doing something like 50 per second, so I rewrote portions and got it down to 0.5 per socond. This was not the problem and it happened again but the recovery procedure was much more robust in that instance.
The real kicker - it happened after we have a Pingdom report with one month being 100% uptime.
I said I wanted to leave but I was convinced to stay by being told that things will be worked out to make my workload more manageable. Needless to say this never happened, and I got a payrise which was definitely not what I wanted. What I wanted was Camvine to succeed and I wanted to have a healthier relationship with my work which was continually getting more and more as time went on. I was forced out not by a specific person but the system I was working under, with four developers being wasted on professional services and the Windows player, it required way too much heavy lifting.
So I started to look, and ended up at a great start-up called Bromium. This all took longer than I would have liked. When I handed in my notice, I found that period was made to be immensely uncomfortable. At least I had Italian Spider-Man to brighten the days. I finished December 2011 with my DNA all over the place, massive amounts of website and codapod source.
So what did I use to build the CODApod? Ubuntu, GTK, WebkitGTK, poppler, clutter, connman, Python, and lots more. I was really proud of the clutter Python based desktop compositor I wrote. If people tell you Python is too slow they are wrong, 60 FPS every time. The whole thing used process abstraction for robustness so bits could fail easily without taking the whole system down. Everything failed gracefully. People told me they could tell it was a Camvine system because of the smooth tickers (using OpenGL textures).
The website was all Python and Django. I'm really glad I took that on, because I discovered I love website development and I think I did a really half decent job, apart from the database disaster of 2011. The site was dealing with 1 million requests per day and didn't break sweat on very modest hardware.
One month after I left it was announced Camvine was going into administration. This was a massive shock to me since I didn't know it was on the cards and thought there was 6 months money still in the bank. Subsequently on seeing some financials the bloodletting had to happen and the company was never going to turn the corner to profitiability. I find it annoying myself because I did not want to take out too much money from the company so made sure my wages were kept at the low-end whch most people see as idiotic, but I see as my commitment and investment in the company. I made sure I helped people as much as I could to find more work and to help out in any way I could, after all it felt like a piece of me still.
In administration it feels like people are dividing up a piece of me. I care immensely about the product and what the company could have been capable of. The codapod is the almost as old as my daughter and the time I have sepnt working on Camvine beyond what is contractually obligated is time I haven't spent with my daughter. I still wouldn't trade it (apart from the horrible ending) as technically I feel I have done great work and I have grown immensely.
The pain I feel that people are trading my code without me with is huge and I look forward to not being near to it in the future because maybe the distance will let me move on.
