Entries this day: automated-testing-is-fun fixing-a-bug-and-setting-up-automagic-tests learning-a-lot

automated testing is fun

08:25 Tuesday 05 January 2016 JST

I woke up early (compared to when I went to sleep) because I had some ideas for doing automated testing. I guess there's no harm in adding some CSS classes to make finding things easier. I may even add a click all that are unselected option if I can't figure out how to do it with CasperJS.

But first I gotta repair a script which runs every 5 minutes via cron. It doesn't run on my local machine, so I missed it after updating the DB.

fixing a bug and setting up automagic tests

02:51 Tuesday 05 January 2016 JST

By far the large part of my time today was spent in a bit of a panic trying to figure out why the AB site was acting horribly wrong. Not even the most basic thing was working: logged in users couldn't sort nor even filter the properties. Yikes stripes!

I was really close to a major upgrade which was alllllmost working on my local system. I got it finished and then spent over an hour testing, and then push the whole thing to live site.

I checked the live site and the problem was still there! Holy crap crap crap crap what is going on?????

Well, I'll tell ya: I had wiped a record associated with my login account, so a JOIN was returning null. I recreated the record and the site worked fine.

Whew.

I don't actually know how the record was removed, but I didn't see any other users who were affected, so I'm going to ignore that pesky little detail for now.

But.

All of this inspired me to look again at testing systems. I have looked at Selenium more than three times and have never been able to properly install the thing, much less get started with tests. But today I found two awesome test systems: https://ghostinspector.com and https://casperjs.org/

Ghost Inspector is suuuuper easy to use. I was able to set up a few tests on the live site in about an hour. The most difficult part was figuring out how to compare two values on the page.

Do it like this:

Casper takes a bit longer to work out how to write the tests, but it has the awesome advantage of running locally, so it can access my websites running on my machine.

learning a lot

23:51 Tuesday 05 January 2016 JST

I spoke on the phone for 1.5 hours to a guy at AWS support. He was patient, understanding, helpful, and many other things that I appreciated. He summarized the convo for me, pasted below with some parts redacted:

Hi,

Thanks for contacting AWS Premium Support. It was a pleasure to speak
with you today in relation to the errors your clients encountered when
accessing your website CB.

I understand in the call that you have set up a Cloudfront
distribution point to serve the requests when users are requesting
images, and it is using ELB (redazzle) as its origin. Meanwhile, your
clients are uploading images to fileswarm.net which is pointing to
your ELB, and in turn images are stored on your backend instances. You
mentioned that in the past 3 days, between 5:00pm and 9:00pm GMT+9,
your clients were seeing some HTTP errors when accessing your website,
but you haven't capture the exact errors they were seeing.

You mentioned that the issues also appeared when uploading the images,
then I started troubleshooting on ELB because this traffic didn't go
through Cloudfront. I notice a few ELB HTTP 5XX errors, and some
spikes on the Max latency which almost reached to 60 seconds, which is
the default timeout value defined on the ELB.

You've enabled the ELB access log so we can have the exact ELB HTTP
5XX errors next time similar issue happens again. From the ELB access
log, you can also see the source IP, requested URL, HTTP response
code, etc. You also have enabled the parameter in Apache configuration
on the backends to record the time used to process the requests. As we
can see in the screenshare, with these information in hand, you can
cross check them to see why some requests are taking longer to be
processed, or not be able to be processed.

ELB HTTP 5XX errors
(link)

Max latency
(link)

Healthyhostcount
(link)

What I missed in the call is that you could also enable Access logs on
CloudFront, which will record the data of customers' requests as well
as the responses, and of course, any errors will be shown there as
well.

So next time when the issues happens, you can try to access the
image/upload the image to your website, and since they are traversing
through different paths, you will be able identify where the issue
resides, and use the logs above mentioned to further troubleshooting
the issue.

Lastly, below are the links we used to enable the %T parameter in
Apache, as well as how you could interpret the information recorded in
ELB access log.

Apache Logs: How Long Does It Take to Serve a Request?
https://www.ducea.com/2008/02/06/apache-logs-how-long-does-it-take-to-serve-a-request/

Monitor Your Load Balancer Using Elastic Load Balancing Access Logs
https://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/access-log-collection.html

I hope you will find above information helpful, if you need any
further assistance, please feel free to contact us again anytime, we
will be delighted to help you further.

Best regards,

Afterwards, I watched 2 or so hours of instruction on udemy about AWS. It seems like a great course, and well worth the current sales price!

https://www.udemy.com/aws-certified-solutions-architect-associate/

Thanks to https://acloud.guru for the course (and several others).