Web analytics and BI reporting services review: Google Analytics and SiSense Prism,

A Review of Services I’ve used in GameGround – Part III – Marketing tools

This is the 3^rd part in a series of blog posts reviewing several 3^rd pty products and services I’ve used in GameGround and my take on them. The basic approach I’m taking here is the applicability of the product for a lean-startup that wants to move fast. In the last post I wrote about Community engagement tools for the marketing team: sending emails and engaging customers in a conversation. This post is about Analytics and BI Reporting. Next up – OPS tools and of course, development infrastructure.

Google Analytics

This extremely popular free SAAS service by Google has become the de-facto standard in website traffic analysis. 10 Years ago I used to download my http server logs and run a simple analysis tool that gave me most of the basic analysis features I needed, but this SAAS has some excellent analytics features like measuring page view time, campaign origin tracking, goal tracking, integration with AdSense etc. There are a few BUTs here, which make me think twice before I choose this option again:

1. The service was built in the times when every website page had a unique URL. This is no longer how the internet works today – we use several widgets that load asynchronously into one page, we change the content of the page using Hash Navigation and AJAX. Especially with mobile content we don’t use the old website concepts anymore – it’s becoming ever more popular to have a single-page website that loads a JavaScript application that makes it all tick. An App, in short. Google knows this and allows you to report “manually” about any URL you are loading. So you can report on any action taken on the screen and on any widget loaded – call the service API with a different fake URL. The problem is that you lose most of the abilities to track the flows users have taken in your application, the count of page views and time spent on a page gets totally skewed. No remedy here.
2. You want analytics not only about page views. You want to know about downloads ppl have made, about clicks to the cancel button on the registration page etc. All that is not well supported with Google Analytics APIs. The Events API does not work with the goals feature and calling the APIs on page events (.g. clicks) reporting fake URLs is skewing everything and the numbers just don’t add up.
3.  The tool is mostly good for the product managers to see how the crowds are using the product/website. A few other causes are not well served: your sales manager would like to get information of sales progress from analytics on individual potential customer – integrate the analytics goals with SalesForce and alert sales staff to contact potential customers. Engineering would like to know of faulty pages (500s), broken links (404s) etc. Security officers would like to know of traffic spikes and login errors to track potential invaders and hackers. You’ll need other tools for those tasks. p.s. check out the friends at Totango for an excellent analytics tool specific for SAAS sales managers.
4. Much of the analytics data is delayed. You get much of your statistical views update daily. So you upload a change close to the end of your working day. You come in the morning – still no significant results. You have to wait another day before you get some results. And that is IF you’ve managed to write the fake URLs thing correctly. In many cases you’ll need to fix it a bit and repeat the test. This is way too slow for a startup…
5. GA is a basic website traffic analysis tool. Traffic alone is good for SEO tasks, understanding traffic sources, goal achievement etc. For Business Interlace you’ll need something much stronger with access to the business data stored in your database (see below).
   

In short – For modern websites and apps GA is almost useless – it will only give you the big picture. Forget about the details …Or check out a better service that was designed for it.

Two insights on the development management side of things: Plan the analytics of every feature as part of the design of the feature itself. Having a feature that one cannot analyze and understand user interaction with is usually worthless. Plan to spend more time than you initially thought to support Google Analytics efforts (probably true with any analytics.)

SiSense Prism

SiSense is a startup developing a very interesting reporting product that is based on unique Columnar Data Storage technology (as opposed to the “regular” OLAP-cubes or other in-memory solutions) that enables large-scale data-sets analysis. The product has an easy-to-use interface that allows creating of beautiful web-based reports for business intelligence, website analysis fort any where managers need a dashboard with stats. It can connect to multiple data sources including most common DBs and even cloud services like Google AdWords, Google Analytics and Amazon S3 logs. This means that the cost of creating and operating excellent reports is much lower than with some other popular products by IBM, Omniture, Microsoft, Oracle and so many others.

I liked using their product a lot. In GameGround the product was mostly operated by one of our QA guys (in addition to his QA roles) that had some basic knowledge in databases and SQL and assisted by our DBA occasionally.

A few notes for everyone thinking of building a BI suite using SiSense and the like:

1. Lean start-ups – abort here! The establishment of a BI product is lengthy, expensive and has a high learning curve. In most cases it would involve bring in an expensive contractor just to help you boot-start the thing. If your data includes a few thousands records you’re much better off with Excel. Excel can connect to most Databases and you can create filters and graphs and send them by email to the marketing/sales daily. It may sound “ugly”, but the time/cost it would take is a fraction of the time it takes to build a proper BI suite. BI reporting suites are not meant for lean start-ups! Starting thinking about a BI suite when you have some real customers and Excel’s abilities of crunching data are too low (over a couple of hundreds of data rows Excel starts slowing down to a crawl!)
2. In many (read: most) cases the information you want to investigate does not exist in the Database. To create a report that shows you how many ppl clicked on ‘like’ and how many ppl uploaded a picture every day you’ll need to add code to collect the data. In many cases this involves code on both the website, the back-end services and changes to the database.  No magic here – BI needs are met with development costs even if the BI person is part of the marketing team.
3. The most important thing with BI is knowing the right questions to ask. In most cases the basic question of “if I give you the data you requested what would you do with it” is never asked. Ask questions that gets you actionable data. The harsh reality is that in too many cases reports were requested and were never utilized.  Still producing those reports took a lot of effort.
4. As a manager, if you ask for very detailed reports you’ll find that you drown in details and cannot get the whole picture. The whole point of having BI dashboards is that you can get the interesting point in 5 seconds. So, start by asking for basic stats that can be visualized over time. Then ask to get details on specific actual things you see.
5. Building a good BI dashboard is, thus, an evolving process, not a project. The project would be to get to the point where you have the first 3 reports in place. Then you’ll want to continue develop more reports and improve on existing.

A few notes specific to SiSense Prism:

1. They are still a startup themselves. Investing in a BI suite that may not be there in a couple of years is risky. Still I give the team at SiSense a strong plus. I’ve seen the work they’ve done with wix is pretty impressive.
2. They prefer you to take a monthly paid subscription to use the product instead of paying a one-time. This is cool and allows you to pay-as-go and pay-as-you consume. It also reduces the cost of boot-starting a BI solution that mostly involves buying a strong server, paying for a contractor to help you out getting the thing to work etc.
3. Their pricing plans are a bit problematic. Their most basic feature – viewing the reports in a browser – is only available in the most pricy plan.
4. Their customer support had some serious issues at times. We got no response for a bug we had with their reports viewer software (we did not use the web-based version), and reverted to use a 500MB server software on each of the managers’ laptops…

Advertisement

Community Engagement services review: SendGrid, GetSatisfaction

A Review of Services I’ve used in GameGround – Part II – Marketing tools  

This is the 2^nd part in a series of blog posts reviewing several 3^rd pty products and services I’ve used in GameGround and my take on them. The basic approach I’m taking here is the applicability of the product for a lean-startup that wants to move fast. In the last post I wrote about A/B/Split testing tools for the marketing team. This post is about Community Mgmt. Next up – Web analytics and BI reporting, OPS tools and of course, development infrastructure.

SendGrid

One of the first features every service has is sending email to customers. There are 2 basic types of emails to send: transactional and mass-mailing. Transactional emails are those produced as a result of a user action, like registration, friend invites etc. Mass-mailing are those when you invite your registered users to an event, a sale etc. So why not use your own corporate SMTP server for those emails? Because you are likely to find yourself in one of the many black lists of spam servers at some point. If spam filters on several servers worldwide find your emails to be spam or If 2-3% of your users mark your email as spam you’ll be black listed and will not be able to send emails from your company at all… bad idea. Other issues you’ll have to manage yourself if you don’t use a SAAS for this is managing unsubscribe lists (<1% of users on social networks unsubscribe in average) and email bounce list (~12% of email address users give on social networks are miss-typed or bogus). Managing those lists is mandatory if you don’t want to get black-listed.

We started with using MailChimp, probably the largest of several competing services, but quickly found that they will not send our mass-mails as they are afraid their servers would get black-listed. We then had the same issue with Constant Contact and CampaginMonitor. It seems that most EMS vendors send all email from a set of about a dozen shared IP addresses. Thus, they have to minimize complaints across their entire portfolio. Most EMS vendors require that you give your users either opt-in (“I’m willing to get marketing materials” checkbox on registration) or double opt-in (+email verification). And if the complaints rate resulting from your service is above a very low rate they kick you out. On our first campaign to just 1200 registered users we had a complaint rate of 1.1% and their acceptable limit was 0.2%… For a young company with little history records that is running its first campaigns the demanded ratios we not acceptable. And- we wanted to have an opt-out on sign up, not an opt-in. We got stuck for a few days till we managed to resolve the mess.

Then entered SendGrid! SendGrid is a cloud-based SAAS with a technology that seems to be far more resilient to black-listing. Their white-label feature allows you to bind your domain MX records to one of their servers with an IP address in cloud. This means you do not share IPs with others and do not need to comply with such low complaints rates. If you get black listed you can change IP address and/or domain name and get back on business in a matter of minutes. So we set up 2 accounts – one for transactional emails, that are less likely to cause blacklisting, and bound it the company’s domain name. Then we bought another domain ‘mailer1 –mycompany.com’ and bound it to the second account. SendGrid system appends an ‘unsubscribe’ link to your emails if you don’t do it yourself and they manage the lists for you – they won’t send an email to someone who unsubscribed, even your service did send them. You get a dashboard where you can see stats of your sent mails, bounces, spam reports etc. and fix your email templates as needed.

The integration with SendGrid’s basic SMTP service took us 15 minutes. They also give you APIs to sync user lists, send using predefined templates etc. but we haven’t got to use those. Pricing is low for what you get. It’s highly recommended to work with them and utilize their APIs to save you the need to write email templates and change them every other days according with the product needs. Let the product guys edit the email templates on SendGrid control panel. No code changes involved unless a radical change is made and different parameters are needed to fill-in the template. So much simpler to operate this feature too. Our email system is working fine with a delivery rate of ~95% on the transactional emails, which is excellent.

Now, how about some tips on how to avoid getting your emails marked as spam? This is a bit out of scope here – maybe I’ll do another post on the quests I’ve had to work-around the spam filters mine-fields. Meanwhile, you may want to read here.

GetSatisfaction

This very successful SAAS product allows you to add a popup widget to your website where your customers can write their feedback, good or bad, and suggest you things. It allows your support and product ppl to engage in a conversation with your customers in a productive way. The Javascript integration is simple and with a bit of extra integration you can also OAuth your logged-in users, so that you can work on a common user-base and get back to the users who wrote a feedback. The control panel allows you to define your products, settings, admin the feedback etc. To have the OAuth feature (a must in my opinion) you need to buy [expensive!] the $99/mo plan. The integration went easy and all was working in a matter of hours. BUT!!! I don’t think this product is so great:

1. The product is a hybrid between an online feedback survey and a forums product. It has the disadvantages of both: you mostly get to hear only the ppl complaining about your product; irrelevant questions and remarks bloat the service with historical and irrelevant data.
2. The loading of their widget is slow and cause long delays in the page loading -time. On some pages the feedback widget was the first thing to show up on the page (WTF???). We ended up writing a script that delay-load their script to bypass their mentality of “We’re THE product and our users are the websites using it”…
3. The suggestion engine meant to prevent users from entering the same feedback over and over again is weak.
4. You already have a Facebook page, a blog and the product pages with comments. Why do you need another place where ppl would talk about your product?! I would give this service a pass next time I’m around. Instead just open a feedback page, place a Facebook comments widgets in it and you’re done.

Marketing tools review: Google Web optimizer, Visual Web optimizer and Unbounce

A Review of Services I’ve used in GameGround – Part I – A/B/Split testing and landing pages services 

GameGround.com is a service I’ve built during 2010 and was alive till mid-2011. I’ve managed this startup dev teams, developing a consumer facing social meta-game. This is a short review of several 3^rd pty products and services I’ve used and my take on them. The basic approach I’m taking here is the applicability of the product to a lean startup that wants to move fast. I started writing it and quickly found out that it’s actually too long for one post. So I’m going to make it a series of post covering Marketing tools, Community Mgmt. tools, OPS tools and of course, development infrastructure.

Google Website Optimizer

GWO this is a simple and free Google service that assists in A/B/Split testing. The JavaScript API makes the decision on which of the optional views to show and you get clear stats view on which of your options is better. I think this product is too simple and not very helpful as it misses the very basic idea of A/B testing: the whole point of A/B testing is that the designer/product mgmt. can run different views and phrasing to find what works. The problem is that for very small variations of pages you need to push code to your production servers and the marketing team cannot work without a developer that assists them in the process. This leads to too many people being involved and process being too slow. Another problem is that some of the stats are updated on a daily basis. In many cases you’d like to make a decision faster and move to the next test – why wait a day?

Visual Web Optimizer

VWO after giving up with GWO we moved to VWO. This new and relatively cheap SAAS-based product is great! After a simple JavaScript integration one can create page variations using a WYSIWYG HTML editor in VWO’s site. Set test goals, alter text, images and CSS and even replace whole pages in runtime. This means that the marketing team can create most A/B/Split variations by themselves and run the testing without a developer nearby (well… They will ask some Qs…). The testing stats results are displayed instantly in a very clear way (see images) and you can even tell it to automatically stop the testing and always show the winning variant. As a bonus you get heat-map views of your tested pages. The couple of issues we’ve had with them were (a) their inability to run tests on our logged-in pages. E.g. pass the login barrier with their WYSIWYG editor which. This required some extra work in the service integration phase. (b) at one point the testing stopped and we found out that a new version of their JS integration library was published without informing us (the customers). Anyways, I can tell that their support team was fast and gave us a quick remedy. VWO’s slogan is “World’s easiest A/B testing tool” and I think they are doing a wonderful job at it.

Unbounce

Unbounce is a landing pages SAAS. “… a self-serve hosted service that provides marketers doing paid search, banner ads, email or social media marketing, the easiest way to create, publish & test promotion specific landing pages without the need for IT or developers.” Yes! Landing pages for specific audiences and campaigns is an excellent way to drive traffic to your site. And Unbounce’s platform with its WYSIWYG HTML editor simplify the process even further allowing the marketing to create those pages and amange them as part of campaigns they are having without needing development involvement. They even give you multi-pages per landing page (e.g. a small website), a lead generation module, A/B/Split testing tools and other goodies. So far so good.

BUT! There’s a major but here: the SEO marks for those pages on Unbounce are extremely low. Search engines don’t like websites and landing pages that has only static content. They also don’t like it that the landing page in not under your own domain, but rather on Unbounce’s, and so they incorrectly see the landing page as a spam blog. This (among others, I’m sure) led us to get very few displays of our ads on Google Adwords and very few clicks coming from this major traffic source.

We ended up using some other desktop HTML editor to create a single-page site for each landing page. It was then uploaded to our live production servers, under the ‘/play’ folder, using a FTP we opened for it. This way the marketing team could create their landing pages according to the running campaigns and upload them to production with little or no dev/OPS involved. This is lean-thinking in its best – have as little ppl involved in each task. Ppl should mostly be able to complete their tasks end-to-end without needing to interface with others.