May 2019 – London Bus Pal Blog

My back-end API’s are all running on a GoDaddy hosting account. Until yesterday, I was using one of their shared servers and the response times started to bother me.

The median response time for a call was about 760ms, which is bad in API terms. What is worse for me, is the disparity – the quickest results tended to be around 500 to 600ms, but then other calls would take up to 4 or 5 seconds to complete. I have some heartbeat checks which does nothing other than return a response, so these should be super-quick, but they were not.

I had a look around the internet and determined that part of my problem was that I had chosen one of the slowest frameworks out there (Slim v3) to build my API. To prove whether it was the framework or my server, I did two things: created a blank heartbeat which doesn’t use the framework (basically just “echo ‘ok’;”) and then also setup node.js to check the speeds. The results pointed to the server, not the choice of framework. Apart from requests within a few seconds of each other, “cold” requests would always take around 500ms.

Upgrading to business

The decision was made to upgrade to business hosting. Surprisingly little information is available on GoDaddy’s upgrade process other than a bunch of complaints about it not working. I was really unsure about how seamless it would be as I didn’t want to be spending the whole day troubleshooting. I half-did a bunch of backups and paid over the money.

About 20 or 30 minutes later, things were done. A quick check of my API’s showed that everything was running and everything appeared to have just worked…or so I thought.

The main two issues which happened to me were that my MySQL database was really slow after the transfer and my WordPress blog stopped working. The blog was fairly trivial to fix as some permissions had changed during the move (not sure why only on for my WordPress installation as everything else remained intact).

One other side issue which happened is that I didn’t disconnect one of my terminal connections. It became clear after a few small tweaks that I was on the old server, not the new.

Slow database

To be fair, most of yesterday is now just a bit of blur, but I noticed the slow database and tried to fix it. In my own inexperience, I ended up locking a bunch of tables (by running optimize on it repeatedly, renaming a 2GB table etc) and things just completely died on me.

A thorough search of the internet helped to kill of most of the processes which had locked up my tables. Stubbornly, two tables remained and I couldn’t see what was locking it, but generally everything else was resolved and my database was running ok apart from the two tables I couldn’t access.

The last suggestion is to just restart the MySQL service, but I do not have the access to do this, so I decided to phone GoDaddy support. I know people complain about them a lot, but I’ve previously had to deal with them (after they killed my SSH access) and it was a painless affair.

Just shoot me instead

The call wait time was short enough and I was soon answered by someone American sounding who seemed eager to help. I explained that I have two tables which are locked which means I cannot access them. This first got interpreted as me forgetting my password; no, I can access the database, I have two tables which are locked.

The things which frustrated me on the call were:

The technician clearly managed to access my database as he called out my table names (this becomes important in a minute).
The technician did not understand what a table lock meant or how to tell that a table was locked. Due to his own lack of knowledge, he kept trying to check with me if what I was saying was really the case or if I’m just another idiot user. Due to this, he had to escalate to his supervisor.
My total database size is around 2GB and I have 2GB of RAM available to me on the server. They repeatedly tried to insist that I simply had to buy more RAM as this was not sufficient due to my database size. I wasn’t having any of it and just kept on refusing.
I asked the technician if they were able to just restart the MySQL on that server and he then said that his supervisor just did this. I checked this and he did not restart the service. He kept explaining to me that it was the MySQL that was restarted, not the server. Yes, I was looking at it and I could see that MySQL was running for 18 hours already.
He then told me that his supervisor suggested that I upgrade to a package with more RAM as my database was inaccessible due to a lack of RAM. I can see the memory usage on my server and only around 300MB was used. He now claimed that he was unable to access my database at all, despite reading me back my own table names earlier. Only two tables were locked, everything else was fine (I checked this in several ways: directly on MySQL, through some API calls and then also checking my logs of external calls with perfectly good responses).

I had to just end the call after 44 minutes because this guy and his supervisor were completely useless. He was clearly out of his depth and instead of owning up to it, he started making up stuff like the fact that your entire database has to fit into the memory available on the server.

While I was on the call, I decided to just rework some of my database any way. The only annoying thing is that I’ve lost around 12 hours worth of data as the two tables in question, are used to store the collected data.

New day

I got up this morning and checked things out and the locks are gone and everything is going swimmingly again. Except for the fact that I redesigned half my database to work around the locked tables. It was a useful exercise any way since I was saving way too much data. (Yes, the one table in question makes up 1.7GB of the 2GB database).

I managed to do queries on it and it is all just fine. In fact…

Rechecking all of my heartbeats and health check, I noticed a few things:

The fastest responses times after I upgraded my package is now around 300ms – this compares to about 600ms for the fastest responses previously. This is a significant improvement.
The median response time is now down to around 350ms to 400ms down from 760ms. Given what I was doing on the server yesterday, I expect this to go down, as these measurements are all from the time while I was doing large numbers of database queries.
Checking this morning, some of my heartbeat responses are coming back below 100ms.
Most surprising is how fast the response is from Node.js – 22ms for a heartbeat – I might just migrate everything to use Node.js instead as this is the speed I am after.

Executive summary

Here is a summary on my feelings about GoDaddy:

Shared hosting is ok for absolute starters, but in hindsight, I should have just got started on business hosting.
GoDaddy is okay in terms of their offering and I feel much better about them after upgrading to the business package.
The company is being let down badly by their support staff. My call yesterday feels like a major run-in – it is also frustrating that both times I have called their support, they try to upsell to me. Support should be good at support, not sales.

Version 4.4.x introduces more bus information into London Bus Pal.

Firstly, I tried to keep the design of all of the functionality to ensure that it is unobtrusive to users not interested in the information, but still easily enough to access for those who are.

Changes which can be observed in the main functionality:

When searching for a specific bus, type-ahead is now available and the app will make suggestions when you start typing either a bus registration number or a fleet number. The type ahead information will also indicate the fleet number and operator.
Rather than just displaying a “no information or not running” message which is unclear, if you search for a bus and it is not in service, the app will show you this message quite clearly now.

New bus information screens:

Once you have found a bus, whether in service or not, you can access the information screens. For in-service buses, an information icon is now displayed in the top right-hand corner and for out of service buses, a clear button is available to show you bus information.
On the bus information screen, there are three tabs available:
- Assembly: this tab gives information about the fleet number and operator. If additional information is available about the bus assembly (such as engine, number of doors, chassis etc), it will be displayed here.
- Usage: this tab will show you which routes the bus has been used on over the last 7 days. Please note that these are estimates based on information interpreted from TfL. These numbers should be used as indicative rather than specific.
- Photos: the photos tab will show you publicly available photos from Flickr for this bus.

This functionality is in a pilot phase at the moment and subject to change. I have some rough plans on what I want to do with this functionality in future, but the purpose of this pilot is to see what the uptake is like and then maybe expand it out a bit further.

At the time of writing this, we have the information available for over 10,000 buses. Only around 750 of those have missing assembly details and I expect to start completing this over the coming months (of course, this will be an ongoing task).

Some plans for the future which I may look at are:

Bus lists of interest (rare workings, new buses, old buses and so on).
Personal bus lists – ability to put buses into your own personal groups; this can then later be used to see where all these buses are.
Route information – based on the information collected at the moment, it becomes possible to see how many buses are working any route at a point in time
Crowd-sourcing vehicle information – all fleet number and operator data is currently managed by myself; this is not sustainable in the long-run.
And many other ideas (some which I just don’t want to mention right now or others which might come from my users).

The roll out of this version is underway at the moment. I am expecting the Android roll out to be complete in a couple of days and iOS might take a little bit longer due to Apple’s requirements (and a technical issue which I experienced trying to create a build).