30 September 2012

Estimating Development Time?

It has been a crazy week where at the beginning nothing existed and on Friday the project was demoed live with all the functionality working.  In this compressed five day lifecycle it brought into sharp clarity  many aspects of software development.  I've always been able to estimate project length but there was no science in it, I always operated on gut feeling and put in healthy margins of error.

I was one of the cogs in this project but before I go into breaking down the project there were some other factors helping this development be a success:

  • Someone leading the project who made sure requirements were locked down and made sure there was no feature creep (within reason).
  • Testers were on from day one, planning how to test and then seeing through all manner of test regimens.
  • Working with smart people.
  • Good tools for source code management, documentation, and bug tracking.
The thing that was really scary for me as a project was how my time broke down:
  • 1 day coding
  • 2 days integration
  • 2 days testing and polishing
The critical aspect here is I spent one day out of five working on pure code and design.  This should be apparent to developers that coding is not the only task in delivering software projects.  By getting something that was working in half a day also gave other people something real to code against rather than trying to hope integration would happen.

The next bit was two days of integration, this involved tidying up the code for deployment, working out installers, dependencies and so on.  Then also it was a case of working out how all of this will work seamlessly for the end user.  This also involved working out how certain bits of the operating system would work when you don't have user contexts to depend on that run as administrator, because after all most devs have their machines configured like this.  Having this installer is critical for running proper end to end tests and ensuring the end-user will get the best experience.

Then we've got testing, this is unit-testing (well actually not a lot of that in a way), functional testing, and integration testing. On a tight deadline some of this is manual and not fully automated.

In amongst all this it was a case of making sure documentation was available for all aspects of the project, from end-user, through development and API right through to testing documents and lists of tests.

I suppose this thing that people reading this is if you work out how long it will take you to code, multiply that by five.

22 June 2012

Bromium Man

The whiteboard scribble I did to celebrate getting out of stealth.


21 June 2012

Bromium - Out Of Stealth

Yesterday Bromium (the company, not what the product will be) came out of stealth mode.  This was a very exciting time around the office because we can give a bit more context on what we've been secretly working away on.

During the proceeding months before this release of information a huge amount of work was going on behind the scenes briefing analysts and news outlets by our top team of Gaurav, Simon, Ian, Tal, and many more.  The avalanche of coverage actually happened to have been prepared a while in advance.

You can see Simon Crosby's presentation "Secure Everything" from GigaOM Structure here and also the Q&A interview afterwards here.  This lays the groundwork for the approach to security that we have pursued at Bromium.  The new website also went live with a couple of videos explaining what we are building.  Lots of new terminology like Microvisors.

A few good articles about Bromium are on Wired and also BloombergBusinessWeek.  The Register has an excellent piece here.

In the meantime I'm going to be plugging away writing code as we hurtle towards our next announcements that will reveal the actual product!

16 June 2012

Bromium - T-Minus Four Days

At the beginning of this year I started my new role at a stealth start-up called Bromium.  What can be said about it is we are building a next generation virtualisation product for security.  The exciting thing is that we are now only four days away from the public announcement about what we have been working on.  This will be at the at the GigaOM Structure conference in San Francisco where our CTO Simon Crosby will be doing a talk called "Secure Everything" (schedule here).

It's been an interesting and hectic time here at Bromium.  Lots of good news in the past month.  We should be in a new office by the end of this month with lots more space.  A new Director Of OS X Products going to start soon in the Cambridge office, which now means two-thirds of the Shed dwellers from Camvine have ended up at Bromium.  We also have several new people starting over the next month, which is great news because they are all extremely talented.

I've spent a lot of time over the past few months doing job fairs and interviews as we try to build a strong Cambridge team.  The Silicon Milkround in Cambridge was not as successful as we would have hoped, but we got a few interesting things out of it.  The Silicon Milkround in London was a roaring success with some crazy amount of CVs and contacts (I think the figure was 40-50) and if you were there you might have seen me do a two minute pitch for the company.  These events I swear were some of the hardest work I've ever done, but really fun and satisfying, I can guarantee I slept well afterwards.  We spent a lot of time sorting through all, this has meant I've done about 20 interviews over the past month which is a good way to fry your brain.

At Bromium it feels like we can't use the term brogrammers which has been co-opted by others.  And to prove that I am not I got minus 165 on Are You A Brogrammer?

If you are a developer, test engineer, or an OS X developer and fancy the challenge of your career drop me a line or mail jobs.uk@bromium.com - we are still hiring.

After the 20th June it will be an exciting prospect to actually tell people what I've been working on!

09 April 2012

The real important feature of C++

I've been having to write a lot of C code recently, and it makes me pine for some features of C++ because of the verbosity of some portions of code. It made me consider what really changes the way you code in C++ on comparison to C.

 

There are features like inheritance, but that really just manages the function overriding that is implicit. There is templates whilst discounting template meta programming is just a glorified way of avoiding code duplication. There is obviously the more aggressive type checking which is handy but doesn't really change that much about your coding style.

 

So what is this feature? Destructors. Immediately by adding that feature you change the way you can code and also by avoiding some C verbosity you avoid memory leaks. This opens up the whole realm of RAII and the code scoping rules start to take on a different life because you can execute code when objects begin to fall out of scope and are freed. This code then kind of discourages gotos (I really hate those) because you are relying more on the stack unwinding than remembering to malloc/free.

 

I reckon if I had destructors for the C code I have written in the context of structs then I could have had a lot less code.

 

03 April 2012

Allwinner A10 - open-source ARM

Developing on ARM is a minefield.  Each platform is different (even if they look the same) and you are lucky if any of the software is not covered by NDA or some prohibitive SDK cost for experimentation.

People have obviously heard of the Raspberry Pi but mid-to-lat 2011 there was anotehr project starting with a company called Rhombus-Tech where they wanted to create a proper CPU card and also have all the software GPL compliant.  This is exciting because you can then get the hardware and with the software available create what you want to.  They went around trying to find the hardware that could potentially have GPL compliant software.  The project has become a CPU card that uses the legacy PCMCIA form factor but re-tasks it entirely for a pluggable processor.  The spec is called the EOMA-68.

Enter the Allwinner A10, a seriously cheap piece of kit ($7 per chip in volume) which is a Cortex A8 with MALI 400 graphics, DDR3 memory and has a BSP capable of decoding 2160p video.  The company behind it Allwinner want to be GPL compliant and have been providing source code for the kernel and the other bits.  The A10 has started to appear in tablets and set-top boxes and the fully integrated platforms are very cheap.  I picked a capacitive touch 7 inch tablet off eBay for £86 including next day postage.  The Mele A1000 set top box is the platform of choice that people have started to pre-emptively use whilst waiting for the Rhombus Tech project to take off and it costs about £80.

The Rhombus Tech spec for the Allwinner A10 is here and you can sign up for preorders for the alpha/beta/release boards here.

So the real exciting bit about this is potentially all the software is out there to construct a decent Linux distribution to build toys and products on.

The main kernel development and tidying up is occurring here and is even being rebased on the 3.3 kenel as well as the Android one.  The u-boot repository for getting it booting is here.  There is a partial code dump for the hardware video decoding here although it looks like it is some of the user space code.  There are some repositories of core Android libraries here and there are some board files for the Allwinner here.  It looks like in the board files there is the OpenGL ES 2.0 and EGL shared objects here.

There are also a couple of good articles about building a custom Android for Allwinner A10 based appliances - part 1 and part 2.  There are also lots of resources for board hacking at the Rhombus Tech website and also some on the Embedded Linux wiki.

So if you fancy some cheap ARM development its worth signing up on the orders page.

Now I need to get hacking on that tablet....

28 March 2012

Camvine - The Highs And Lows

These are some other random highs and lows from the Camvine experience, lifting the veil on some bits and pieces.

What would I have rather have done with those three-and-a-half years?  Nothing else.  It'll be in my blood for a while longer.

HIGH

We did the Cambridge Tech Demo Night.  Quentin had prepared lots of slides to cram into 7 minutes, and I was going to simultaneously do a full unboxing from scratch (with no cheating) simultaneously to how how easy it was to set up.  Unfortunately projector connection failure caused us to throw out the slides and Quentin fired up a web browser on some Windows machine that was part of the University network that by default was controlling the projector.  He talked, fired up the browser and I started unboxing and connecting cables, within about three minutes we were connected, registered and uploading content to the display.  The projector failure was the best thing to happen to us because we showed the product end-to-end in such a short period, we even got a standing ovation - scary and heartwarming stuff.

LOW

I was always asked to evaluate a new hardware platform, which I would get working.  Then I was always expected to support it without any hardware because £200 was too much to keep a small PC around.  Unfortunately for other things like Windows licences the company was able to pour money down the drain.

HIGH

We had a crazy test rig.  It involved a full server in a Xen VM with two NICs so there was a fake Internet hanging off the back of it.  This was always our saviour, we insstantly knew if software updates or networking were broken because we had the physical hardware dangling off the back, not some mock representation.  This obviously couldn't save all problems but it meant the customers rarely saw a broken release.  It was horrible and verbose, but I saw the power in it being an integral part of the software development.

LOW

Not only did we have the major SSD failure of 2011, we had them replaced.  Then they died again, but this was a quick hotswap because we did not trust the management services we were paying for any more.  What I also discovered was the server grade Intel drives were actually cheap'n'nasty Crucials for consumers.  Because of the work I did reducing the database load we were completely fine back on spinny disks.

HIGH

Working on Linux full-time.  Working on Mac full-time.  Going nowhere near Windows.  And making sure my Macs and Linux machines had awesome decals.

LOW

Stopping working on the Mac.  I realised I was using a VM with Linux the whole time on my Mac so I got myself an Asus U36JC which was an awesome laptop that I had configured perfectly for Linux.

HIGH

Rewriting the network stack to be pure Python.  The code is all here https://github.com/garrybodsworth/coda_network - this means NTLM proxies worked, streaming worked, and all the intricacies of networking worked as they were supposed to.  Although I never got an official document to say I could open-source it, I did it anyway.

LOW

The Windows player.  As alluded to we had to have a Windows player and the whole feature creep and ignoring my initial specification documents helped make it a disaster.  To be honest, I have no problem with a windows player, but it took two developers for over a year and it never got released in the end.  If two developers had been working full time on the thing we were actually selling then I think it would have been a huge win all round for the customers because all the features needed would have been done.

HIGH

Having an iOS player.  This did get released.  Michael Dales did excellent work on one day a week to get this done in a realistic timescale.  I think technically it was a damn good idea and something to promote, unfortunately the comany itself didn't know what to do with it so it languished in marketing purgatory.

LOW

Battling with ATi binary drivers, that was massively hard work, but I eventually had it so it could stay up for a significant period of time before the ASIC locked.  The main culprit was the XVBA decoding seemed to cause memory to eventually go mad.  The nvidia drivers had their fair share of issues like breaking vsync rendering for rotated displays which meant I ended having to make everything a surface in clutter and rotate that(!)  Adobe Flash support was similarly problematic mainly because of the underpowered CPUs we were using.

HIGH

Programming the initial 35 batch of CODApods on my own in the shed on a Friday with Tron playing on the TV above my desk.  Tron was being played through a CODApod of course!

LOW

Being forced to evaluate hardware from people that were basically chancing it and were realistically years away from having a fully working system.  I ended up sinking way too much time into that.

HIGH

The people who I feel embodied Camvine, Quentin, Sarah, Michael, and Thomas, without them there would have been no company to try and build.  John Naughton was also integral to the initial spirit.

And all the other people in no order and if I forgot anyone slap me round the head in the pub - Brian Boakes, Martin Waddell, Joseph Newman, Jonathan Thackray, Richard Morrison, Tomasz Wojciechowski, William Baer, Rob Berwick, Nick Brook, Dan Clemens, Allison Holloway, Steve Hales, John Martyn, John Naughton, Diego Barrow, saafad Khan, Emily Poulton, Breton Saunders, Justin Drake, Glenn Moodie, and Adam Brunning.

27 March 2012

Camvine - An Obituary - From Start-Up To Stop In Four Years

Maybe I could also call this "what was i doing for the last three-and-a-half years?"  Today could really be described as the final day of Camvine's existence, even though I left at the end of December (something I would rather not have done, I'll explain later) this is something immensely dear to my heart, so I thought I would weave my thread through it's patchwork.  So, what was Camvine?  It was an Internet connected digital signage system where you could control your displays anywhere around the world through your web browser.

I was working at DisplayLink, a company set up by Quentin Stafford-Fraser, but wasn't enjoying my work, so I was looking at startups for a good challenge and found a startup founded by Quentin Stafford-Fraser...  The weird thing was I had never met him, but thought it would be an interesting challenge.  I still have no idea what made them take a chance on me, although my profile on the website called me a veteran, which definitely made me feel old.  I think one of the major factors was that I at least had an idea of how the hardware worked, you see the first product of CodaBox + CodaLinks was based on the ethernet nivos from DisplayLink.  They were an interesting proposition, very low power and low cost, but also not massively capable pieces of hardware, so one small low powered Intel Atom powered central machine could run up to 20 displays.

So I joined Camvine mid-2008 and I ended up working in "The Shed" which is a studio at the bottom of a garden in Cambridge, where ndiyo and DisplayLink also started.  It was my first chance to do commercial Python development and it was an exciting free-flowing environment.  We even had Fun Fridays where we could work on stuff not on the roadmap, quite a few cool features came out of that single day of the week, unfortunately some did not get integrated like transitions code I had working quite sweetly on the CodaLinks.  The website was really awesome evolving into a drag and drop user interface for controlling your displays and content modelled on the iTunes style UI.

Around that time my wife fell pregnant with our first child, you would think that startup life and children do not mix, but I went head-first into it all and it was immensely rewarding.

Soon, we are heading towards a "The Pivot"...  The team needs introductions, we have Quentin Stafford-Fraser, CEO, ideas, maverick, like Hannibal in the A-Team.  We have Michael Dales, CTO, uber-hacker.  Thomas Hunger, developer, the brains.  Sarah McKeon, the office manager, the glue that holds the company together and keeping it running (every company needs a Sarah).  And me, lowest rung on the ladder.

Not long after me a sales and marketing director joined the company whose expertise was in the advertising arena, so we started to target advertisers.  Whilst CodaLinks were in interesting esoteric proposition for communicating they were not what the advertising industry craved which is tons of rich content (read: Flash) and high definition videos.  The advertising industry and digital signage are not a good place to be, but I'll come onto that.  So now we hit the pivot point around April 2009, which incidentally is the month my daughter was borrn.  We grab some Asus EeeBoxes and decide to use an Intel Atom N270 based system and start writing code for it (partially based on porting our playback code for the CodaLinks).  I start hitting the base platform with coding hammers (we were based on Ubuntu so we stauck with that) and we ended up with a Jaunty franken-distro set up mainly by me and Thomas.  By September we are shipping EeeBoxes to a potential customer needing to get rid of an unmanageable legacy system, the weird thing was I had this Acer Revo sitting on my desk and I couldn't get the damn thing to boot Linux on a USB key, then somehow I worked out how to make it work by altering the heads/sectors on the key (I'm sure I blogged about it at the time).  It had an nvidia ION as the GPU which was very interesting, for a start thhe binary drivers were much better than the Intel ones at the time (I was one of the original users of the KMS drivers trying to get it all working), also it had the ability to do video decoding on the GPU.  Because the driver quality was so good it blew the Eeeboxes out of the water with the same CPU, so this became our hardware platform in what seemed like overnight, unfortunately we did have to write off about 20 Eeebox units at the expense of the company, embarressing on my part, but totally the right decision.

Sales and marketing decided the new product should be called the CODApod, urgh, horrible name!  Internally I called it the incredicoda which not only sounding quite unhinged is really easy to grep for in source code, as a developer this is important.

The advertising industry is rubbish for digital signage, they want everything perfect, which is surprisingly difficult to achieve.  You end up having to trade things on multiple axes which are robustness, speed, quality of output, and accuracy.  You can throw more staff at solving these issues but that is not an option at some startups like this running on a shoestring.  One of the things you can't explain no matter how hard you try is that displaying a web page is not a no=op, you have to download resources, start rendering, get some more resources, render some more, then you might be complete ready for display.  Even with everything cached in memory that is not a no-op.

The most important thing to get the software shipped just 5 months after deciding on this was a robust software update, then things could be fixed in the field remotely.  I had made software update more robust since joining Camvine, and continually strived to make it unbreakable.  We evolved the software over the next six months into something quite impressive with hardware accelerated codecs (we paid Fluendo to finish development of VDPAU), newer, better web browsers and robust PDf viewers.  Unfortunately a bombshell hit in December 2009 when Thomas Hunger decided to leave for the bright lights of London.  The months that followed were a blur, me and Michael were support and development, answering the phone and support requests as well as me developing the client hardware and him on the website.  It was simultaneously awesome and horrible, horrible volume of work, but I think we were a well oiled machine with just two people in the shed.  We stuck our headphones in and got our heads down and moved forward as well as maintaining all that was there.

Some more people joined, Quentin stepped down as CEO, and we got a new CEO.  The next bombshell for me was losing Michael as CTO.  I was profoundly gutted, not only was it amazing to work with him, I consider him a good friend.  His reasons for leaving, I will leave up to him, but losing him as CTO and protector of the Camvine faith is the worst thing to ever happen to the company.  It took me a while to realise that he should have become CEO (although obviously he would not want that job), but that would have been what was right and proper for the company.  I had a crazy idea that I should take over website development and hand CODApod development over to someone else, since I knew the most about the system and Camvien was proving to be very poor at hiring, even not allowing me to advertise on some of the places I wanted to.  Michael left October 2010, and the handover was an unmitigated disaster.  I was forced to work on attempting to make a playlist perfect for an advertising potential customer, who was never going to put their money where their mouth was.  I was doing this instead of doing a decent handover, luckily Michael was still available one day a week.

I got my head down and obsessed about the website.  Christmas was a nightmare because I was now the only thing between our website and total oblivion, especially with issues like the server going up to load 30 at 8:30am every morning.  I ended up by March 2011 restructuring pretty much all the deployment to use nginx, gunicorn, and Fabric, the tools of the modern Django coder.  The databases were split onto new machines, the load balancing worked pretty great.

Unfortunately, the CODApod was essentially dead.  The company had decided that they wanted a Windows CODA player, so this would now suck two man-years of effort into a black hole as it was never officially released.  During this time the only development on the CODApod was what I did in what i laughably called my "spare time", also the CODApod software had to be ported to work on as many hardware platforms as possible (ATi, nvidia, Intel Core) where there were no scheduled developer resources.  I was doing all this whilst also running and developing and being on-call for the website.  I felt really sorry for the customers knowing no client side developments were happening that could improve their hardware further.  Somehow I managed to have a release of CODAsoft available two months after starting work in my spare time, and I kept it up to date until the day I finished.  It worked flawlessly on Intel graphics (945 through to I think Ivy Bridge), ATI (open-source and binary drivers with hardware acceleration) and nvidia (everything the binary driver supports and with hardware acceleration).

We left the shed in 2011.  A very sad day for me.  The weird thing was the physical seperation matching the logical seperation of the company actually was a great thing that was not appreciated by some people.  By moving into one huge space things started to bleed into one another and there was an immense lack of self-control by certain roles.

The moment I knew I had to leave Camvine completely and utterly was Father's Day 2011.  We were out in the middle of nowhere with me, my wife, daughter and father-in-law, and there was basically no phone reception.  I got an odd phonecall and discovered the servers were down and attempted to debug what was going on through a phone call and intermittent data coverage.  Basically we were screwed.  For a bit of context, I was forced to outsource database management to someone else in order to alleviate my workload, and we ended up with two SSD based servers with master-slave, back-ups and database dumps every day.  Everything failed, but it had been failing for a month.  Hard drives?  Broken serving corrupted data silently.  Backups of the HDs?  Failing silently.  Database dumps?  Failing as well.  What happened was they decided to upgrade a backup program which caused the mysql process to restart, which is where all hell broke loose.  I managed to get in the office at 5pm Sunday and was there reconstructing enough of a business to open doors again the Monday morning otherwise we should have just shut up shop.  I have no idea how I did it, but enough was workking by 1am and Quentin was great moral support.  The next two weeks of 14 hour days involved recreating as much data as possible which was almost everything, which was something to at least be proud of.  The thought of this still makes me feel sick.  The write rate of the web service was blamed as it was doing something like 50 per second, so I rewrote portions and got it down to 0.5 per socond.  This was not the problem and it happened again but the recovery procedure was much more robust in that instance.

The real kicker - it happened after we have a Pingdom report with one month being 100% uptime.

I said I wanted to leave but I was convinced to stay by being told that things will be worked out to make my workload more manageable.  Needless to say this never happened, and I got a payrise which was definitely not what I wanted.  What I wanted was Camvine to succeed and I wanted to have a healthier relationship with my work which was continually getting more and more as time went on.  I was forced out not by a specific person but the system I was working under, with four developers being wasted on professional services and the Windows player, it required way too much heavy lifting.

So I started to look, and ended up at a great start-up called Bromium.  This all took longer than I would have liked.   When I handed in my notice, I found that period was made to be immensely uncomfortable.  At least I had Italian Spider-Man to brighten the days.  I finished December 2011 with my DNA all over the place, massive amounts of website and codapod source.

So what did I use to build the CODApod?  Ubuntu, GTK, WebkitGTK, poppler, clutter, connman, Python, and lots more.  I was really proud of the clutter Python based desktop compositor I wrote.  If people tell you Python is too slow they are wrong, 60 FPS every time.  The whole thing used process abstraction for robustness so bits could fail easily without taking the whole system down.  Everything failed gracefully.  People told me they could tell it was a Camvine system because of the smooth tickers (using OpenGL textures).

The website was all Python and Django.  I'm really glad I took that on, because I discovered I love website development and I think I did a really half decent job, apart from the database disaster of 2011.  The site was dealing with 1 million requests per day and didn't break sweat on very modest hardware.

One month after I left it was announced Camvine was going into administration.  This was a massive shock to me since I didn't know it was on the cards and thought there was 6 months money still in the bank.  Subsequently on seeing some financials the bloodletting had to happen and the company was never going to turn the corner to profitiability.  I find it annoying myself because I did not want to take out too much money from the company so made sure my wages were kept at the low-end whch most people see as idiotic, but I see as my commitment and investment in the company.  I made sure I helped people as much as I could to find more work and to help out in any way I could, after all it felt like a piece of me still.

In administration it feels like people are dividing up a piece of me.  I care immensely about the product and what the company could have been capable of.  The codapod is the almost as old as my daughter and the time I have sepnt working on Camvine beyond what is contractually obligated is time I haven't spent with my daughter.  I still wouldn't trade it (apart from the horrible ending) as technically I feel I have done great work and I have grown immensely.

The pain I feel that people are trading my code without me with is huge and I look forward to not being near to it in the future because maybe the distance will let me move on.

02 March 2012

New Github Repo - polipo on Microsoft Visual Studio

Today I've just uploaded a fork of polipo to my Github account.  it is a caching proxy server which works really well with a tiny amount of resources. It is primarily driven by Unix development so it has a version that works on Cygwin and a version that is not considered complete or tested that can use the Mingw toolchain.

I wanted to get it working with Visual Studio and I stumbled across some work here at polipovc where someone got it working with Visual Studio.  It is a code dump rather than a series of patches, so I took the polipo git repository on Github and forked it.  I then applied the changes as a series of patches making minimal changes in each one and tried to normalise it with the project coding style.  I also created a solution for Visual Studio 2010.  It compiles for 32-bit and 64-bit (but just ignore the swatches of warnings).

The on-disk cache seems to not be working properly right now so I might look a bit more closely into why that is the case.  To be honest the directory checks need to be factored out.

You can see my polipo github repository here.