28 March 2012

Camvine - The Highs And Lows

These are some other random highs and lows from the Camvine experience, lifting the veil on some bits and pieces.

What would I have rather have done with those three-and-a-half years?  Nothing else.  It'll be in my blood for a while longer.


We did the Cambridge Tech Demo Night.  Quentin had prepared lots of slides to cram into 7 minutes, and I was going to simultaneously do a full unboxing from scratch (with no cheating) simultaneously to how how easy it was to set up.  Unfortunately projector connection failure caused us to throw out the slides and Quentin fired up a web browser on some Windows machine that was part of the University network that by default was controlling the projector.  He talked, fired up the browser and I started unboxing and connecting cables, within about three minutes we were connected, registered and uploading content to the display.  The projector failure was the best thing to happen to us because we showed the product end-to-end in such a short period, we even got a standing ovation - scary and heartwarming stuff.


I was always asked to evaluate a new hardware platform, which I would get working.  Then I was always expected to support it without any hardware because £200 was too much to keep a small PC around.  Unfortunately for other things like Windows licences the company was able to pour money down the drain.


We had a crazy test rig.  It involved a full server in a Xen VM with two NICs so there was a fake Internet hanging off the back of it.  This was always our saviour, we insstantly knew if software updates or networking were broken because we had the physical hardware dangling off the back, not some mock representation.  This obviously couldn't save all problems but it meant the customers rarely saw a broken release.  It was horrible and verbose, but I saw the power in it being an integral part of the software development.


Not only did we have the major SSD failure of 2011, we had them replaced.  Then they died again, but this was a quick hotswap because we did not trust the management services we were paying for any more.  What I also discovered was the server grade Intel drives were actually cheap'n'nasty Crucials for consumers.  Because of the work I did reducing the database load we were completely fine back on spinny disks.


Working on Linux full-time.  Working on Mac full-time.  Going nowhere near Windows.  And making sure my Macs and Linux machines had awesome decals.


Stopping working on the Mac.  I realised I was using a VM with Linux the whole time on my Mac so I got myself an Asus U36JC which was an awesome laptop that I had configured perfectly for Linux.


Rewriting the network stack to be pure Python.  The code is all here https://github.com/garrybodsworth/coda_network - this means NTLM proxies worked, streaming worked, and all the intricacies of networking worked as they were supposed to.  Although I never got an official document to say I could open-source it, I did it anyway.


The Windows player.  As alluded to we had to have a Windows player and the whole feature creep and ignoring my initial specification documents helped make it a disaster.  To be honest, I have no problem with a windows player, but it took two developers for over a year and it never got released in the end.  If two developers had been working full time on the thing we were actually selling then I think it would have been a huge win all round for the customers because all the features needed would have been done.


Having an iOS player.  This did get released.  Michael Dales did excellent work on one day a week to get this done in a realistic timescale.  I think technically it was a damn good idea and something to promote, unfortunately the comany itself didn't know what to do with it so it languished in marketing purgatory.


Battling with ATi binary drivers, that was massively hard work, but I eventually had it so it could stay up for a significant period of time before the ASIC locked.  The main culprit was the XVBA decoding seemed to cause memory to eventually go mad.  The nvidia drivers had their fair share of issues like breaking vsync rendering for rotated displays which meant I ended having to make everything a surface in clutter and rotate that(!)  Adobe Flash support was similarly problematic mainly because of the underpowered CPUs we were using.


Programming the initial 35 batch of CODApods on my own in the shed on a Friday with Tron playing on the TV above my desk.  Tron was being played through a CODApod of course!


Being forced to evaluate hardware from people that were basically chancing it and were realistically years away from having a fully working system.  I ended up sinking way too much time into that.


The people who I feel embodied Camvine, Quentin, Sarah, Michael, and Thomas, without them there would have been no company to try and build.  John Naughton was also integral to the initial spirit.

And all the other people in no order and if I forgot anyone slap me round the head in the pub - Brian Boakes, Martin Waddell, Joseph Newman, Jonathan Thackray, Richard Morrison, Tomasz Wojciechowski, William Baer, Rob Berwick, Nick Brook, Dan Clemens, Allison Holloway, Steve Hales, John Martyn, John Naughton, Diego Barrow, saafad Khan, Emily Poulton, Breton Saunders, Justin Drake, Glenn Moodie, and Adam Brunning.

27 March 2012

Camvine - An Obituary - From Start-Up To Stop In Four Years

Maybe I could also call this "what was i doing for the last three-and-a-half years?"  Today could really be described as the final day of Camvine's existence, even though I left at the end of December (something I would rather not have done, I'll explain later) this is something immensely dear to my heart, so I thought I would weave my thread through it's patchwork.  So, what was Camvine?  It was an Internet connected digital signage system where you could control your displays anywhere around the world through your web browser.

I was working at DisplayLink, a company set up by Quentin Stafford-Fraser, but wasn't enjoying my work, so I was looking at startups for a good challenge and found a startup founded by Quentin Stafford-Fraser...  The weird thing was I had never met him, but thought it would be an interesting challenge.  I still have no idea what made them take a chance on me, although my profile on the website called me a veteran, which definitely made me feel old.  I think one of the major factors was that I at least had an idea of how the hardware worked, you see the first product of CodaBox + CodaLinks was based on the ethernet nivos from DisplayLink.  They were an interesting proposition, very low power and low cost, but also not massively capable pieces of hardware, so one small low powered Intel Atom powered central machine could run up to 20 displays.

So I joined Camvine mid-2008 and I ended up working in "The Shed" which is a studio at the bottom of a garden in Cambridge, where ndiyo and DisplayLink also started.  It was my first chance to do commercial Python development and it was an exciting free-flowing environment.  We even had Fun Fridays where we could work on stuff not on the roadmap, quite a few cool features came out of that single day of the week, unfortunately some did not get integrated like transitions code I had working quite sweetly on the CodaLinks.  The website was really awesome evolving into a drag and drop user interface for controlling your displays and content modelled on the iTunes style UI.

Around that time my wife fell pregnant with our first child, you would think that startup life and children do not mix, but I went head-first into it all and it was immensely rewarding.

Soon, we are heading towards a "The Pivot"...  The team needs introductions, we have Quentin Stafford-Fraser, CEO, ideas, maverick, like Hannibal in the A-Team.  We have Michael Dales, CTO, uber-hacker.  Thomas Hunger, developer, the brains.  Sarah McKeon, the office manager, the glue that holds the company together and keeping it running (every company needs a Sarah).  And me, lowest rung on the ladder.

Not long after me a sales and marketing director joined the company whose expertise was in the advertising arena, so we started to target advertisers.  Whilst CodaLinks were in interesting esoteric proposition for communicating they were not what the advertising industry craved which is tons of rich content (read: Flash) and high definition videos.  The advertising industry and digital signage are not a good place to be, but I'll come onto that.  So now we hit the pivot point around April 2009, which incidentally is the month my daughter was borrn.  We grab some Asus EeeBoxes and decide to use an Intel Atom N270 based system and start writing code for it (partially based on porting our playback code for the CodaLinks).  I start hitting the base platform with coding hammers (we were based on Ubuntu so we stauck with that) and we ended up with a Jaunty franken-distro set up mainly by me and Thomas.  By September we are shipping EeeBoxes to a potential customer needing to get rid of an unmanageable legacy system, the weird thing was I had this Acer Revo sitting on my desk and I couldn't get the damn thing to boot Linux on a USB key, then somehow I worked out how to make it work by altering the heads/sectors on the key (I'm sure I blogged about it at the time).  It had an nvidia ION as the GPU which was very interesting, for a start thhe binary drivers were much better than the Intel ones at the time (I was one of the original users of the KMS drivers trying to get it all working), also it had the ability to do video decoding on the GPU.  Because the driver quality was so good it blew the Eeeboxes out of the water with the same CPU, so this became our hardware platform in what seemed like overnight, unfortunately we did have to write off about 20 Eeebox units at the expense of the company, embarressing on my part, but totally the right decision.

Sales and marketing decided the new product should be called the CODApod, urgh, horrible name!  Internally I called it the incredicoda which not only sounding quite unhinged is really easy to grep for in source code, as a developer this is important.

The advertising industry is rubbish for digital signage, they want everything perfect, which is surprisingly difficult to achieve.  You end up having to trade things on multiple axes which are robustness, speed, quality of output, and accuracy.  You can throw more staff at solving these issues but that is not an option at some startups like this running on a shoestring.  One of the things you can't explain no matter how hard you try is that displaying a web page is not a no=op, you have to download resources, start rendering, get some more resources, render some more, then you might be complete ready for display.  Even with everything cached in memory that is not a no-op.

The most important thing to get the software shipped just 5 months after deciding on this was a robust software update, then things could be fixed in the field remotely.  I had made software update more robust since joining Camvine, and continually strived to make it unbreakable.  We evolved the software over the next six months into something quite impressive with hardware accelerated codecs (we paid Fluendo to finish development of VDPAU), newer, better web browsers and robust PDf viewers.  Unfortunately a bombshell hit in December 2009 when Thomas Hunger decided to leave for the bright lights of London.  The months that followed were a blur, me and Michael were support and development, answering the phone and support requests as well as me developing the client hardware and him on the website.  It was simultaneously awesome and horrible, horrible volume of work, but I think we were a well oiled machine with just two people in the shed.  We stuck our headphones in and got our heads down and moved forward as well as maintaining all that was there.

Some more people joined, Quentin stepped down as CEO, and we got a new CEO.  The next bombshell for me was losing Michael as CTO.  I was profoundly gutted, not only was it amazing to work with him, I consider him a good friend.  His reasons for leaving, I will leave up to him, but losing him as CTO and protector of the Camvine faith is the worst thing to ever happen to the company.  It took me a while to realise that he should have become CEO (although obviously he would not want that job), but that would have been what was right and proper for the company.  I had a crazy idea that I should take over website development and hand CODApod development over to someone else, since I knew the most about the system and Camvien was proving to be very poor at hiring, even not allowing me to advertise on some of the places I wanted to.  Michael left October 2010, and the handover was an unmitigated disaster.  I was forced to work on attempting to make a playlist perfect for an advertising potential customer, who was never going to put their money where their mouth was.  I was doing this instead of doing a decent handover, luckily Michael was still available one day a week.

I got my head down and obsessed about the website.  Christmas was a nightmare because I was now the only thing between our website and total oblivion, especially with issues like the server going up to load 30 at 8:30am every morning.  I ended up by March 2011 restructuring pretty much all the deployment to use nginx, gunicorn, and Fabric, the tools of the modern Django coder.  The databases were split onto new machines, the load balancing worked pretty great.

Unfortunately, the CODApod was essentially dead.  The company had decided that they wanted a Windows CODA player, so this would now suck two man-years of effort into a black hole as it was never officially released.  During this time the only development on the CODApod was what I did in what i laughably called my "spare time", also the CODApod software had to be ported to work on as many hardware platforms as possible (ATi, nvidia, Intel Core) where there were no scheduled developer resources.  I was doing all this whilst also running and developing and being on-call for the website.  I felt really sorry for the customers knowing no client side developments were happening that could improve their hardware further.  Somehow I managed to have a release of CODAsoft available two months after starting work in my spare time, and I kept it up to date until the day I finished.  It worked flawlessly on Intel graphics (945 through to I think Ivy Bridge), ATI (open-source and binary drivers with hardware acceleration) and nvidia (everything the binary driver supports and with hardware acceleration).

We left the shed in 2011.  A very sad day for me.  The weird thing was the physical seperation matching the logical seperation of the company actually was a great thing that was not appreciated by some people.  By moving into one huge space things started to bleed into one another and there was an immense lack of self-control by certain roles.

The moment I knew I had to leave Camvine completely and utterly was Father's Day 2011.  We were out in the middle of nowhere with me, my wife, daughter and father-in-law, and there was basically no phone reception.  I got an odd phonecall and discovered the servers were down and attempted to debug what was going on through a phone call and intermittent data coverage.  Basically we were screwed.  For a bit of context, I was forced to outsource database management to someone else in order to alleviate my workload, and we ended up with two SSD based servers with master-slave, back-ups and database dumps every day.  Everything failed, but it had been failing for a month.  Hard drives?  Broken serving corrupted data silently.  Backups of the HDs?  Failing silently.  Database dumps?  Failing as well.  What happened was they decided to upgrade a backup program which caused the mysql process to restart, which is where all hell broke loose.  I managed to get in the office at 5pm Sunday and was there reconstructing enough of a business to open doors again the Monday morning otherwise we should have just shut up shop.  I have no idea how I did it, but enough was workking by 1am and Quentin was great moral support.  The next two weeks of 14 hour days involved recreating as much data as possible which was almost everything, which was something to at least be proud of.  The thought of this still makes me feel sick.  The write rate of the web service was blamed as it was doing something like 50 per second, so I rewrote portions and got it down to 0.5 per socond.  This was not the problem and it happened again but the recovery procedure was much more robust in that instance.

The real kicker - it happened after we have a Pingdom report with one month being 100% uptime.

I said I wanted to leave but I was convinced to stay by being told that things will be worked out to make my workload more manageable.  Needless to say this never happened, and I got a payrise which was definitely not what I wanted.  What I wanted was Camvine to succeed and I wanted to have a healthier relationship with my work which was continually getting more and more as time went on.  I was forced out not by a specific person but the system I was working under, with four developers being wasted on professional services and the Windows player, it required way too much heavy lifting.

So I started to look, and ended up at a great start-up called Bromium.  This all took longer than I would have liked.   When I handed in my notice, I found that period was made to be immensely uncomfortable.  At least I had Italian Spider-Man to brighten the days.  I finished December 2011 with my DNA all over the place, massive amounts of website and codapod source.

So what did I use to build the CODApod?  Ubuntu, GTK, WebkitGTK, poppler, clutter, connman, Python, and lots more.  I was really proud of the clutter Python based desktop compositor I wrote.  If people tell you Python is too slow they are wrong, 60 FPS every time.  The whole thing used process abstraction for robustness so bits could fail easily without taking the whole system down.  Everything failed gracefully.  People told me they could tell it was a Camvine system because of the smooth tickers (using OpenGL textures).

The website was all Python and Django.  I'm really glad I took that on, because I discovered I love website development and I think I did a really half decent job, apart from the database disaster of 2011.  The site was dealing with 1 million requests per day and didn't break sweat on very modest hardware.

One month after I left it was announced Camvine was going into administration.  This was a massive shock to me since I didn't know it was on the cards and thought there was 6 months money still in the bank.  Subsequently on seeing some financials the bloodletting had to happen and the company was never going to turn the corner to profitiability.  I find it annoying myself because I did not want to take out too much money from the company so made sure my wages were kept at the low-end whch most people see as idiotic, but I see as my commitment and investment in the company.  I made sure I helped people as much as I could to find more work and to help out in any way I could, after all it felt like a piece of me still.

In administration it feels like people are dividing up a piece of me.  I care immensely about the product and what the company could have been capable of.  The codapod is the almost as old as my daughter and the time I have sepnt working on Camvine beyond what is contractually obligated is time I haven't spent with my daughter.  I still wouldn't trade it (apart from the horrible ending) as technically I feel I have done great work and I have grown immensely.

The pain I feel that people are trading my code without me with is huge and I look forward to not being near to it in the future because maybe the distance will let me move on.

02 March 2012

New Github Repo - polipo on Microsoft Visual Studio

Today I've just uploaded a fork of polipo to my Github account.  it is a caching proxy server which works really well with a tiny amount of resources. It is primarily driven by Unix development so it has a version that works on Cygwin and a version that is not considered complete or tested that can use the Mingw toolchain.

I wanted to get it working with Visual Studio and I stumbled across some work here at polipovc where someone got it working with Visual Studio.  It is a code dump rather than a series of patches, so I took the polipo git repository on Github and forked it.  I then applied the changes as a series of patches making minimal changes in each one and tried to normalise it with the project coding style.  I also created a solution for Visual Studio 2010.  It compiles for 32-bit and 64-bit (but just ignore the swatches of warnings).

The on-disk cache seems to not be working properly right now so I might look a bit more closely into why that is the case.  To be honest the directory checks need to be factored out.

You can see my polipo github repository here.