MarkedUp SDK v1.2 for Windows Store and Windows Phone Apps: Lets You Help Your Users Manage Their Privacy

People of MarkedUp,

Today we released MarkedUp Analytics SDK for .NET v1.2, and it’s the first of many planned updates to our SDK and reporting dashboard coming over the next couple of months.

This update is for Windows Phone 8 and Windows Store developers, and it has a number of important fixes and updates – I’ll start with the minor ones:

  1. Full support for logging nested inner exceptions and stack traces in C#;
  2. Fixed a bug with capturing the app version # from the wrong place in Windows Phone;
  3. Fixed some performance issues on Windows Phone, related to reinitializing tombstoned apps;
  4. And various other minor bug fixes and improvements.

Allow Your Users to Take Control of Their Privacy with Analytics Opt-Out

Within a day or two of our initial launch a year ago, the first feature request submitted to MarkedUp’s UserVoice page was a feature to enable MarkedUp customers to allow their users to opt-out of analytics.

Well today, we’re marking that issue as resolved. With MarkedUp v1.2, you can now use the built-in AnalyticClient.OptOut method to allow users to opt-out of tracking. Users can also opt-back-in at a later date using the same method if you wish.

tracking-opt-outs

We’ve also added a new chart that allows you to see how many users opt-out of MarkedUp Analytics, which you can find under Users –> Tracking Opt-Outs.

opt-out-chart

With this tool, you can now give your users control over whether or not you collect any information on them.

How to Use Opt-Out

If you’ve installed MarkedUp into your Windows Store or Windows Phone application, you just need to call one method to opt-out a user.

Users are tracked by default, so if you want to enable them to opt out you should add an “Opt Out of Tracking” option to your settings page.

Here’s the code you need to call to opt a user out of MarkedUp Analytics tracking:

WinRT and Windows Phone (C# / XAML)
MarkedUp.AnalyticClient.OptOut(true);

WinJS (HTML5/JavaScript)
MK.optOut(true);

That’s it! Click here for more detailed instructions.

Privacy is a growing concern among native app developers and app consumers, and we heard your requests and feedback and provided you with a dead simple tool for allowing your app’s users to control their privacy.

If you have questions, reach out to us at support@markedup.com or reach out to us on Twitter (@MarkedUpMobi.) We’re happy to help!

-Aaron

Track In-App Purchase and Trial Conversion Revenue like Never Before: New Commercial Reporting from MarkedUp

We’ve been working feverishly over the past few weeks to provide MarkedUp customers a radically better experience for tracking sales and revenue than before, and we’re proud to stand behind our latest release: introducing commercial reporting version 2.

commercial-reporting navigation

In our last major update to MarkedUp customers, we released verison 1.0.5 of the MarkedUp Analytics SDK and made some significant updates to how we track sales and revenue inside the app.

We made those changes in order to provide a better experience to customers who need to track sales and revenue from in-app purchases and trial conversions; this update is to make sure that you get excellent reporting to match.

So without further adieu, let’s show you what’s under the hood!

Break out Sales by Country, OS Version, App Version, Device, and More!

We’ve added over 24 new reporting views to MarkedUp in this update, and the biggest ones are the drill-down reporting for sales and revenue.

revenue-by-app-version

These reports help you measure important things like:

  1. Where your most frequent customers are located geographically;
  2. If one release of your app is more / less effective at converting users into customers than another;
  3. If there’s a disparity between your total installs on Device X and total customers on that same device; and
  4. How different operating systems and OS versions change the purchasing habits of your users.

Per-Product Sales Reporting and Top Performing Products

If you’re going to offer one in-app purchase, you might as well offer two – right?

For developers who are selling multiple products via the Windows Store or any other marketplace, MarkedUp offers the ability to break out all of the reports mentioned in the previous update by each product individually.

Moreover, we’ve added a new top-level report, “Top Performing Products,” that allows you to view the performance of all of your app’s products at a glance.

Revenue Reporting in Multiple Currenciescurrency-selector

Most companies standardize on a single currency, but since the Windows Store allows you to price your application in up in over 60+ currencies, we felt it necessary to make sure that you can record your sales in as many currencies as your business supports.

In all of your Excel® reports, we break out revenue by currency so you’ll always be able to keep them separate during accounting.

Microsoft Excel® Export for all Commercial Reports

excel-logo

As it commonplace throughout the rest of MarkedUp’s reports, all of our new commercial reports ship with the ability to export directly to Microsoft Excel® from day one.

Now you if want to perform any of your own analysis on revenue and transaction figures MarkedUp collects, you can download our data directly into Excel format on-demand.

Something New: “Users who bought” Report – Dynamic Search through Your Sales Transactions

The most powerful feature of them all is something special – the “Users who bought” report. This allows you to dynamically search over every sales transaction your app has ever recorded using MarkedUp version 1.0 and later.

Words don’t do “Users who bought” justice – you have to see it in action:

MarkedUp Analytics–Users Who Bought Report

If you sell in-app purchases or trial conversions inside your apps and aren’t using these features, take advantage of them now!

How to Get Started with MarkedUp’s Commerce API

Here’s how you can get started with MarkedUp’s Commerce API:

  1. Watch our tutorial video, “Using Commercial Transactions to Track In-App Revenue” [4:02]
  2. Read the MarkedUp Commerce API Overview in our documentation.
  3. Read our tutorial, “Tracking In-App Purchases and Trial Conversion Using MarkedUp Analytics.”

And as always, if you have questions, reach out to us at support@markedup.com or reach out to us on Twitter (@MarkedUpMobi.) We’re happy to help!

Best,

-Aaron

Simplifying the MarkedUp SDK

Earlier this evening we released MarkedUp 1.0.5 into the wild, and this SDK includes some major changes – namely, we removed about 80% of the public methods from our SDK and completely redid the entire commerce system. You can read the full MarkedUp v1.0.5 release notes here.

I wanted to share with our users why we made the changes we made and why we ultimately think that these changes will improve your experience using our analytics services.

The Commerce APIs

Not to tip our hand here, but we’re not too far away from shipping a comprehensive set of reporting features and improvements aimed at helping native app developers better understand where their sales and revenue are come from and who their target customers really are.

So naturally, making sure that our SDK’s commercial features are in tip-top shape was a huge concern when we started this development sprint.

We design all of our products to be as automated as possible in order to provide developers who depend on our services with the best experience possible. With that in mind, here’s how we designed our original commerce APIs:

  1. Accept a ProductId (official unique identifier used to track an in-app purchase or paid version of an app on the Windows Store) and a human-readable product name for each MarkedUp.InAppPurchase[Complete or any other commerce method] and MarkedUp.TrialConversion[Complete et al];
  2. Talk directly to CurrentApp or equivalent API and load the pricing data directly from the Windows Store for any given user making a purchase, including current market and currency;
  3. Cache Windows Store values locally to avoid repeat lookups;
  4. Post successful transactions to MarkedUp’s Commerce system using data retrieved from Windows Store.

With the earliest versions of our SDK (v0.6.7 and earlier) we went to market with this method, but strongly suspected that there were some issues with this methodology. Given that we can’t test CurrentApp without running inside a live app that actually has in-app purchases, it was difficult for us to get a feel for what was going on inside our own SDK under the circumstances.

So beginning in SDK 1.0, the MarkedUp Analytics SDK created an error on your log browser every time it failed to retrieve pricing information from the Windows Store. And sure enough: we’ve had a steady ticket volume from customers who’ve seen those errors.

The root cause: for whatever reason, the Windows Store price-lookup APIs have an extremely poor success rate – over half our calls to look up the price of an item fail! We’re not sure why, but that’s not our problem. Making sure our customers can successfully report on their sales is what we care about.

The “right” way to do it.

So, we introduced manual, but extremely reliable methods for recording trial conversion and in-app purchase sales in the v1.0.4 version of the MarkedUp SDK and had a dozen or so customers deploy it in their live applications, and we were met with tremendous success.

Here’s what the updated commerce methods look like in v1.0.4 and v1.0.5:

//All of this data can be mapped from a ProductListing via the Windows.ApplicationModel.Store namespace 
var iap = new MarkedUp.InAppPurchase() 
{ 
    ProductId = "SampleIAP1", 
    ProductName = "Sample In-App Purchase", 
    CurrentMarket = RegionInfo.CurrentRegion.TwoLetterISORegionName, 
    CommerceEngine = "Windows Store", //or a third-party engine, like PayPal
    Currency = RegionInfo.CurrentRegion.ISOCurrencySymbol, 
    Price = 2.59 
};
MarkedUp.AnalyticClient.InAppPurchaseComplete(iap);

Pretty straightforward – all of the information needed to populate a MarkedUp.InAppPurchase or MarkedUp.TrialConversion object can be derived directly from the Windows Store, or you can populate it yourself with your own values. We may even open-source some extension methods to populate these objects from ProductListing and ListingInformation objects.

MarkedUp no longer cares if the in-app purchase or trial conversion exists on the Windows Store or not – and that’s a feature; we want to be able to support commerce that happens outside the Windows Store and this gives us the flexibility to do both.

Say “goodbye” to [TC|IAP]Cancelled, Selected, Shown, and Dismissed, methods.

We originally created the InAppPurchaseShown, Dismissed, Selected, and Cancelled methods as a way to help developers set up simple conversion funnels for their in-app purchase and trial conversion sales efforts.

While we haven’t launched conversion funnels yet, we had a number of customers asking us about where they could see data for these methods and whether or not they were necessary for tracking in-app purchase revenue.

We decided to gut these methods from the SDK entirely, as of v1.0.5, and they will not be coming back for the following two reasons:

  1. Session events are a much better tool for tracking conversion funnels and in-app behavior than a dorky built-in event and
  2. It’s simpler to use and makes it obvious to new MarkedUp users how to record an in-app purchase or trial conversion.

The Session Event APIs

As of today, all of the input form and search + share charm events have been removed from MarkedUp. We strongly recommend using session events to track these types of events; they’re the right tool for the job.

So why’d we remove these? Because frankly, they added a lot of bloat and things to dig through while using our SDK on any platform – and the search + share charm functionality was extremely specific to Windows 8 in particular.

Closing

MarkedUp is ultimately going to support every native platform, and one of the things that consistently attracts great reviews and testimonials from our users is the simplicity of our products and services. Simplifying the SDK was part of the natural order of things, and it will make it easier for us to add support for iOS and Android in the near future.

As always, please let us know if you have any questions!

Best,

-Aaron

WinRT and Windows Phone 8 Code-Sharing Pitfalls

One of the things Microsoft promised and touted highly prior to and during BUILD 2012 was that it would be possible to share reams of code between your Windows Store and Windows Phone 8 applications.

From MSDN’s “Windows Phone API Reference” here’s what Microsoft made available, taken directly from the article:

 

The Relationship between WinRT and Windows Phone 8 APIs

The diagram has three distinct areas and these are described as follows:

  1. The set of Windows Runtime API not supported on Windows Phone 8. The API surface area of Windows Runtime is very large, with over 11,000 members. We’ve adopted a subset for Windows Phone 8 that allows you to build compelling phone scenarios. Area 1 in the diagram above represents the APIs that are not available on Windows Phone 8.

  2. The set of Windows Runtime API adopted for Windows Phone 8. This is represented by area 2 in the above diagram and consists of approximately 2,800 members. For some types, we have not implemented certain members. For others we have added additional members to support phone-only features. In both cases, these differences are noted in the API reference documentation.

  3. We’ve added key APIs needed to build great apps for the phone. These are represented by area 3 in the diagram and total about 600 members. For example, we have brand-new APIs for speech synthesis and recognition, VOIP, and other features. Creating these as Windows Runtime style APIs means you can use them regardless of the programming language you use for your app.

Naturally there are portions of code that can’t be shared easily, such as device-specific / sensor capabilities and user interface.

However, Microsoft promised that a significant number of WinRT APIs would be made available on WP8 and for a long time touted that “copy and paste” portability would be achievable.

MarkedUp’s SDK technology doesn’t interact directly with the UI or sensors on either WinRT or Windows Phone 8, so the majority of the APIs we use are ones you’d expect to work with both platforms, such as:

  • File system and storage;
  • Serialization;
  • Networking;
  • Globalization;
  • the Windows Store APIs;
  • Licensing;
  • Async and multi-threading; and
  • Unit testing with MSTest.

Our experience in porting our code from WinRT (C#) to Windows Phone 8 (C#) was largely good, although we found some nasty pitfalls along that way. In light of BUILD 2013 and all of the new features coming to Windows Phone and Windows 8.1, we thought it would be a good time to share these.

Networking

System.Net.HttpClient – By far the biggest and most vociferous WinRT & WP8 compatibility issue echoed from the development community is the fact that System.Net.HttpClient, Microsoft’s modern HTTP client that is built from the group up to work with async, advanced media types, RESTful APIs, and so forth. HttpClient is a huge boon to our productivity in the course of working with WinRT.

For reasons that I am not privy to, Microsoft did not include HttpClient in the core framework for Windows Phone 8, so MarkedUp uses System.Net.WebRequest wraps it with the appropriate async and error-handling decorators.

One of the major pitfalls of WebRequest is that it throws an exception whenever you receive an HTTP code that isn’t in the 200 (“Success) category. Trying to handle these exceptions asynchronously complicates things considerably, but it can be done.

It should be noted that HttpClient for Windows Phone 8 is available as a separate NuGet package. We can’t use any third party dependencies in our SDK but you are certainly free to use these inside your apps.

System.Net.WebRequest APIs are different across .NET 4.5 / Windows Phone 8 and WinRT – One strategy for dealing with the lack of HttpClient availability on Windows Phone 8 is just to use System.Net.WebRequest everywhere, including WinRT. This is something we tested since standardizing on a single HTTP layer helps reduce our QA overhead.

Our experience was that while vanilla .NET 4.5 and Windows Phone 8 have 100% reuse and compatibility when it comes to WebRequest, this is not the case with WinRT.

For instance, WebRequest does not have any directly settable ContentLength or UserAgent properties in WinRT; WebRequest has a different interface altogether for forming HTTP request bodies in WinRT. Additionally, the System.Net.WebException enum implementation in WinRT is missing a number of options that make it difficult to perform error-handling consistently across platforms.

We ultimately determined that it was less expensive to maintain two different network stack implementations for Windows Phone and WinRT than to try to work around all of the incompatibilities and different behaviors with WebRequest in WinRT.

File System

Windows.Storage.Search is not fully implemented on Windows Phone 8 – For most WinRT developers, Windows.Storage.Search is probably used for one specific purpose inside any given app: determining whether or not a file exists.

Since there isn’t an equivalent to File.Exists in the Windows.Storage namespace, most developers (us included) have to resort to something like this:

One issue that we ran into early on in our QA for Windows Phone is that not all search options which use a Windows.Storage.Search.CommonFileQuery enumeration are implemented in Windows Phone 8. For instance, CommonFileQuery.OrderByName will throw a NotImplemented exception.

However, you can work around most of these issues by just using CommonFileQuery.DefaultQuery if the ordering of the files returned isn’t particularly important.

Licensing

CurrentApp.LicenseInformation doesn’t actually tell you if your Windows Phone 8 app is running as a trial or not – WinRT developers rely on CurrentApp as the single source of truth for all things Windows Store and licensing on Windows 8, and the class is available on Windows Phone 8 and is what developers use for executing in-app purchases.

However, for reasons unknown, the data returned from CurrentApp.LicenseInformation doesn’t have the IsTrial flag set in accordance with the actual licensing status on Windows Phone. Instead, you have to use the legacy API from Windows Phone 7, new Microsoft.Phone.Marketplace.LicenseInformation().IsTrial(), to get that information.

In order to make things more tenable for our x-platform development with WP8 and WinRT, we wrapped CurrentApp into a class called CurrentAppProxy and have it call the appropriate WP8-specific method when inferring licensing information.

Commerce

CurrentAppSimulator does not exist for Windows Phone 8, thus there’s no built-in way to test in-app purchases – We depend on CurrentAppSimulator when testing anything that touches a trial conversion or in-app purchases, so we were a little shocked when we discovered that it isn’t included in Windows Phone 8.

Microsoft subsequently shipped their own mock in-app purchase library for Windows Phone 8, but since we have a shared testbed between WinRT and WP8 and can’t use any third party dependencies anywhere in our SDK we ended up writing our own. We will likely open source it on Github eventually.

No ListingInformation.CurrentMarket – One of the really handy features of ListingInformation in WinRT is the ability for it to tell you the current marketplace of any given user right off the bat using ListingInformation.CurrentMarket; this is important from an analytics point of view and really important if you manage your own commerce engine and want to localize prices in different currencies.

In Windows Phone 8, this method throws a NotImplemented exception. So how do you look up this information on WP8? Thanks to some smarties at StackOverflow, it turns out that System.Globalization.RegionInfo.CurrentRegion can’t be changed by the end user and is the culture reported by the phone to the store for any in-app purchases.

Globalization

RegionInfo uses different constructor arguments on WinRT and Windows Phone 8 – in WinRT, you can construct a new RegionInfo object by passing along the two-character ISO country code that you might receive from a marketplace method. This works great particularly when coupled with CurrentApp.

In Windows Phone 8, you’ll get an ArgumentException every time – in WP8 RegionInfo only accepts the culture name itself, i.e. “en-US,” as the constructor argument, not the region name.

GlobalizationPreferences.HomeGeographicRegion does not exist in Windows Phone 8
when try to establish the local time and culture for a user in WinRT, GlobalizationPreferences.HomeGeographicRegion makes this a breeze if you need to account for local time or any other globalization preference. This API does not exist in Windows Phone 8, but because a user can’t change their region settings you just just use RegionInfo.CurrentRegion.

Serialization

DataContractJsonSerializer doesn’t actually respond to EmitTypeInformation flags – one of the lesser-used features of DataContractJsonSerializer is its ability to control whether or not it includes type information in its outbound JSON payloads via the EmitTypeInformation property; this feature is really useful if you’re sending data to an ASP.NET MVC service and want to disable type hints so the model binder doesn’t fail.

Unfortunately, in Windows Phone 8 they never implemented the control flow that responds to this flag, so you get type information always whether you want it or not. Ultimately, we had to parse out type information using a regular expression upon serialization in order to resolve this issue for Windows Phone 8.

We hope this has been information – let us know if you have any feedback in the comments!

How to Install and Run the Windows 8.1 Preview in Hyper-V

The big announcement from //BUILD 2013 is the pending arrival of Windows 8.1, aka “Windows Blue,” and Microsoft made Windows 8.1 available for download immediately following the //BUILD keynote on Wednesday.

Like many developers, the MarkedUp Analytics team is naturally excited to try new things; that being said – we also don’t like having to wipe a dev machine and re-image it in the event that a beta release of Windows isn’t compatible with tools we use for doing our jobs every day.

So, we use Hyper-V or VirtualBox and run Windows 8.1 in a VM until it’s released to market.

Here’s how to get Windows 8.1 running in Hyper-V:

1. Download the Windows 8.1 Preview ISO

Download any of the Windows 8.1 Preview ISO images here. Pick whichever one best suits your needs. I’m going to use the English 64-bit .ISO.

2. Open Hyper-V and create a new virtual machine

If you’re running Windows 8 already, Hyper-V comes built into the operating system. Once the Windows 8.1 preview ISO is finished downloading you’ll want to create a new VM.

step1 - new VM

We’ll name our VM “Windows 8.1 Preview” and use the default storage location.

step2 - name VM

Windows 8 needs at least 2GB of RAM on a 64-bit system; I’m going to give this VM 4GB since I plan on running Visual Studio and some other RAM-intensive software on it later.

step3 - allocate RAM to VM

For now we’re going to leave the VM’s network as “Not Connected” – we’ll create a Virtual Network Adapter for it later.

If you already have a Hyper-V virtual network adapter that can share Internet connectivity with the host machine, use it here. Otherwise we’ll add one later after the OS installation on the VM is complete.

step4 - set VM network to not connected

Windows 8 needs at least 20GB of disk space on a 64-bit system. I’m going to give this VM 40GB and I’m going to use a dynamically expanding disk.

Dynamically expanding disks will be slow initially, but it saves room on the host machine in the event that you don’t occupy the entire VHD volume immediately.

If speed is an issue, you can create a fixed-size disk up front.

step5 - set VHD creation options for VM

3. Install Windows 8.1 Preview from ISO

Now we’re going to use the Windows 8.1 Preview .ISO file we downloaded earlier to install the operating system while we finalize our VM.

stp6 - install OS via Windows 8.1 ISO

This option will have you complete the Windows 8.1 Preview installation the first time the VM boots, using the .ISO file you downloaded earlier as the bootable media.

With all of those steps complete, you should now see the “Windows 8.1 Preview” VM on your Hyper-V list:

step7 - verify that Windows 8.1 VM is available in Hyper-V

Select the Windows 8.1 Preview VM and start it, and you should see the first “Install Windows” screen after a few seconds:

step8 - start the Windows 8.1 VM and install the OS

Click next.

You’ll need to enter in a product key for Windows 8.1 Preview during the first part of the installation process.

The product key for Windows 8.1 Preview is NTTX3-RV7VB-T7X7F-WQYYY-9Y92F, according to Microsoft’s official Windows 8.1 Preview installation instructions.

step9 - enter product key for Windows 8.1 Preview

If you see the following message and it asks you to choose between an upgrade and a custom installation, Select “Custom: Install Windows only.”

step10 - select custom install for Windows 8.1 Preview

Install Windows 8.1 on the VHD that we created earlier.

step11 - install Windows 8.1 preview on VHD created earlier

Let the Windows 8.1 Preview installation run to completion. And after 30 minutes or so you should be able to create a local Windows account and log in.

step12 - verify Windows 8.1 installation

4. If you don’t have one already, create a Virtual Network Adapter for Windows 8.1

If you didn’t have a Virtual Network Adapter ready during step 2, we’ll create one now.

First, shut down your Windows 8.1 Preview VM and turn it off – we’re about to make some changes to it.

step13 - shut off Windows 8.1 VM

Open up the Hyper-V manager and go to “Virtual Switch Manager.”

step14 - go Virtual Switch Manager in Hyper-V

Select “New virtual network switch.” Give the switch a name and make it an External Network. If you have multiple physical network adapters (i.e. ethernet and WiFi), use whichever one you use most often. Check the “allow management operating system to share this network adapter” box.

step15 - create new Virtual Adapter in HyperV

Apply changes.

Go back to your VMs and right click on the “Windows 8.1 Preview” VM you created – select “Settings,” then select “Network Adapter.”

step16 - Windows 8.1 VM network adapter settings

Change the Virtual switch to the new switch you just created; mine is called “Magical Switch.”

step17 - set virtual switch on Windows 8.1 Preview VM

Apply changes. Before we try starting the VM, let’s make sure that our switch was set up correctly – I often have trouble getting Hyper-V’s network adapters to behave properly the first time around.

Go to Control Panel and then to View Network Connections. Right click on the new switch you just created and select Properties.

step18 - verify virtual switch properties

Only the following properties should be set:

  • Client for Microsoft Networks
  • QoS Packet Scheduler
  • File and Printer Sharing for Microsoft Networks
  • Microsoft LLDP Protocol Driver
  • Link-layer Topology Discovery Mapper I/O Driver
  • Link-layer Topology Discovery Responder
  • Internet Protocol Version 6 (TCP/IPv6) and
  • Internet Protocol Version 4 (TCP/IPv4)

You will likely need to reboot your host machine in order to get the Internet working again. Go ahead and do that now.

5. Start Windows 8.1 Preview; Profit

You now should be able to start your Windows 8.1 VM in Hyper-V and connect to the Internet.

Install complete

Please give this guide a try and let us know if you run into any trouble!

Available Now: MarkedUp Analytics for Windows Phone 8

MarkedUp Analytics now available for Windows Phone

Windows Phone 8 support has been a long-requested feature for MarkedUp and it we’re pleased to announce that as of today, June 19th, MarkedUp Analytics fully supports Windows Phone 8!

If you install the latest MarkedUp Analytics SDK for .NET from NuGet, it will automatically work in any Windows Phone 8 application.

Here are some things you need to know about the MarkedUp Analytics SDK for Windows Phone 8:

1. The MarkedUp SDK for Windows Phone is virtually identical to the WinRT SDK for C# / XAML – you can port all of your MarkedUp SDK calls from any XAML-based Windows Store application to a Windows Phone 8 application with no changes. See the updated MarkedUp SDK Reference for more details.

2. The SDK maintains a slim file size (~200kb) so you don’t need to worry about it affecting your app’s download size from the Windows Store.

3. The SDK automatically enables the app capabilities it needs to run during your NuGet installation, so you don’t have to do anything. See the Windows Phone 8 App Requirements for more details.

4. You can now add Windows Phone project types to your MarkedUp dashboard. Click here to add a Windows Phone 8 app to your dashboard right now.

And most of all…

It only takes 3 minutes to install MarkedUp into your Windows Phone app and see live data.

Don’t believe us? Watch our tutorial video below.

How to Install MarkedUp Analytics for Windows Phone 8 Apps

 

We’re thrilled to make our services available on Windows Phone 8, and we’re always striving to do an even better job serving the needs of app developers everywhere. So if you have any suggestions for what we can do better, please leave us a note on our UserVoice. Thanks!

Introduction to Parameterized Session Events

One of the most commonly requested features for MarkedUp has been the ability to support parameterized session events, and as of the release of MarkedUp Analytics SDK for .NET 1.0 this feature is now publicly available to our customers.

Parameterized session events help you group and categorize related user events that occur inside your applications; this gives you the ability to maintain a high-level view of how people are using your application while also allowing you to expose more details if needed.

For example: I have a Hacker News Windows Store application that I’ve been working on in my spare time and I want to be able to track how many articles people actually view when they use it.

So, I embedded a standard MK.SessionEvent call inside my WinJS application to track how many people viewed an article:

And this allows me to see the total number of times someone has viewed an article using my app:

Hacker News application with a simple, unparameterized event

But what if I want more information – what if I want to know which articles users are viewing? Parameterized session events can help us with that.

I’m going to modify the call to MK.SessionEvent and include a new parameter: “ArticleUrl.”

“ArticleUrl” is the name of my parameter, and the value of this is the URL of whatever article a user is viewing. In C# / XAML you can just pass in an IDictionary<string, string> object for all of your parameters and values, but in WinJS you need to use a Windows.Foundation.Collections.PropertySet collection.

Once I’ve run the app with this custom parameter included, I can drill down into the ViewArticle event and see a new parameter:

Hacker News app with an ArticleUrl parameter for the ViewArticle session event

And if I click on the “ArticleUrl” parameter, guess what we see next? A list of all of the URLs for each event!

hacker-news-viewarticle-articleurl-parameter-values

With that simple change, our ViewArticle event becomes much more informative and valuable – we can even use the date picker to see which articles were most popular amongst our userbase during each month or week!

Advanced Example: Distinct Session Event Parameters

Let’s take our usage of parameterized session events a step further. One of the features of my Hacker News application is the ability for it cache articles to the local filesystem in order to conserve bandwidth. I’m considering extending this capability to enable the app to run entirely offline – so my question is: how often do people really look at cached articles?

Parameterized session events can help us answer this question.

I’m going to move my MK.SessionEvent call from viewItem.js and into my caching layer – that way I can tell if the article I am serving is cached or not. Here’s my code looks like for that:

I have to different methods that are called by my app when we need to view an article – one that checks if the content is cached and serves it if it is, and another that downloads the content from the Internet.

Whenever we serve the content from cache, I pass in a ps[“Cached”] = true parameter and a ps[“NotCached”] = true whenever we don’t.

So after running the app a few times, it looks like most users don’t view cached articles very often:

Hacker News WinJS app - cached vs. uncached articles

I’m doing it this way so I can see directly at the ViewArticle view how many articles are served from cache versus those who aren’t without having to drill down any further.

But the choice is yours – parameterized session events are designed to be flexible and allow you to use them however you wish.

Make sure you check out our tutorial on using parameterized events with MarkedUp Analytics, and please let us know if you have feedback!

Using Cassandra for Real-time Analytics: Part 2

In the part 1 of this series, we talked about the speed-consistency-volume trade-offs that come along with implementation choices you make in data analytics and why Cassandra is a great choice for real-time analytics. In this post, we’re going to dive a little deeper on the basics of the Cassandra data model and illustrate with the help of MarkedUp’s own model, followed by a short discussion about our read and write strategies.

Once again, lets start off of our LA Cassandra User group meetup’s presentation deck on slideshare. Slides 12-18 are relevant for this post.

Cassandra Data Model

The Cassandra data model consists of a keyspace (analogous to a database), column families (analogous to tables in the relational model), keys and columns. Here’s what the basic Cassandra table (also known as a column family) structure looks like:

Cassandra Column Family structure

  Figure 1. Structure of a super column family in Cassandra

Cassandra’s API also refers to this structure as a map within a map. where the outer map key is the row key and inner map key is the column key. In reality, a typical Cassandra keyspace for, say, an analytics platform, might also contain what’s known as a super column family.

 

Cassandra Super column family structure

                                              Figure 2. Structure of a super column family in Cassandra

 

Evan weaver’s blogpost has a good illustration of the twitter keyspace as a real world example.

MarkedUp’s keyspace has column families such as DailyAppLogs (that count the number of times a particular log or event triggered per app) and Logs (that capture information about each log entry). These are also illustrated in figure 1 below.

The Datastax post about modeling a time series with Cassandra is particularly helpful in deciding upon the schema design. We index our columns on the basis of dates.

Note that since we use a randomPartitioner  where rows are ordered by the MD5 of their keys, using dates as column keys helps in storing data points in a sorted manner within each row. Other analytics applications might prefer indexing by hours or even minutes, if, for example, the exact time of day when the app activity peaks needs to be measured and reported. The only drawback would be more data points and more columns in the keyspace. With a limit of about 2 billion column families in Cassandra though, its almost impossible to exceed the limit. Thus, the fact that Cassandra offers really wide column families leaves us with enough leg room.

The row key in a Cassandra column family is also the “shard” key, which implies that columns for a particular row key are always stored contiguously and in the same node. If you are worried that some of your shards will keep growing at a faster rate than others, resulting in “hotspot” nodes that store those shards, you can further shard your rows by means of composite keys. Eg: (App1, 1) and (App1, 2) can be two shards for App1.

The counter for all events of a particular type coming from apps using MarkedUp are recorded in the same shard. (“What about hotspots then?”, you might wonder! Well, Cassandra offers semi-automatic load balancing so we load balance if a node starts becoming a hotspot. Refer to the Cassandra wiki for more on load balancing)

MarkedUp’s Read/Write Strategy

Now that we have a better understanding of the Cassandra data model, lets look at how we handle writes in MarkedUp. Logs from the Windows 8 apps that use Markedup arrive randomly on a daily basis. For incoming logs, we leverage the batch mutate method.

As you might have probably guessed, a batch_mutate operation groups calls on several keys into a single call. Each incoming log, therefore, triggers updates or inserts in multiple column families, as shown in figure 3. For example, a RuntimeException in AppX on Jan1, 2013 will update the DailyAppLogs CF with key AppX by incrementing the counter stored in the column key corresponding to Jan1, 2013 as well as the Logs CF by inserting a new key LogId. 

MarkedUp write strategy

Figure 3. MarkedUp’s write strategy

 

MarkedUp’s read strategy leverages Cassandra’s get_slice query, which allows you to read a wide range of data focused on the intended query, reducing waste (A ‘slice’ indicates a range of columns within a row). A query to count a wide range of columns can be performed in minimal disk I/O operations. Setting up a get_slice query is as simple as specifying which keyspace and column family you want to use and then setting up the slice predicate by defining which columns within the row you need.

The slice predicate itself can be set up in two ways. You can either specify exactly which columns you need, or you can specify a range of ‘contiguous’ columns using a splice range. Using column keys that can be sorted meaningfully is thus critical.

Figure 4 below illustrates the query “Get all Crash and Error logs for App1 between Date1 and DateN”. The get_slice_range query can easily read the counters as a complete block from the AppLogsByLevel CF because the CF is sorted by dates.

MarkedUp read strategy

Figure 4. MarkedUp’s read strategy

 

If you’ve read our previous blog post closely, you might be wondering if the returned information is even correct, given the fact that Cassandra compromises on consistency in favor of speed and volume (remember the SCV triangle?). Cassandra guarantees what is known as eventual consistency, which means that at some given point (milliseconds away from the triggering of the write operation), some nodes may still have the stale value, although by the end of the operation, every node will have been updated.

Luckily, Cassandra offers tunable consistency levels for queries. So, depending on your appetite for consistent output vis-a-vis speed, you can configure the desired consistency level by chosing different levels of “quorum”. MarkedUp uses ONE for writes and TWO for reads, to keep the web front-end as fluid as possible.

In the part 3 of this series, we’ll talk about some best practices of working with Cassandra and choosing a schema that fits your needs. Stay tuned for more!

Using Cassandra for Real-time Analytics: Part 1

In a previous blog post titled “Cassandra, Hive, and Hadoop: How We Picked Our Analytics Stack” we talked about our process for selecting Cassandra as our data system of choice for supporting our real-time analytics platform.

We’ve been live with Cassandra in production for a couple of months now and shared some of the lessons and best practices for implementing it at the Los Angeles Cassandra Users Group on March 12, 2013. You can see the presentation on slideshare if you’d like to view our slides.

We wanted to expand on what we shared in the presentation itself and share some of our applied knowledge on how to put Cassandra to work in the field of real-time analytics.

Let’s start by helping you understand how an analytics system needs to be built.

Real-time Analytics and the CAP Theorem

For those of you who aren’t familiar with Brewer’s CAP theorem, it stipulates that it is impossible for any distributed computer system to simultaneously provide all three of the following guarantees:

  1. Consistency;

  2. Availability; and

  3. Partition tolerance.

In the real-world all distributed systems fall on a gradient with each of these three guarantees, but the kernel of truth is that there are trade offs. A system with high partition tolerance and availability (like Cassandra) will sacrifice some consistency in order do it.

When it comes to analytics, there’s a transitive application of the CAP theorem to analytic systems – we call it SCV:

analytics cap theorem

  1. Speed is how quickly you can return an appropriate analytic result from the time it was first observed – a “real-time” system will have an updated analytic result within a relatively short time of an observed event, whereas a non-real-time system might take hours or even days to process all of the observations into an analytic result.

  2. Consistency is how accurate or precise (two different things) the analytic outcome is. A totally consistent result accounts for 100% of observed data accounted for with complete accuracy and some tunable degree of precision. A less consistent system might use statistical sampling or approximations to produce a reasonably precise but less accurate result.

  3. Data Volume refers to the total amount of observed events and data that need to be analyzed. At the point when data starts to exceed the bounds of what can be fit into memory is when this starts to become a factor. Massive or rapidly growing data sets have to be analyzed by distributed systems.

If your working data set is never going to grow beyond 40-50GB over the course of its lifetime, then you can use an RDBMS like SQL Server or MySQL and have 100% consistent analytic results delivered to you in real-time – because your entire working set can fit into memory on a single machine and doesn’t need to be distributed.

Or if you’re building an application like MarkedUp Analytics, which has a rapidly growing data set and unpredictable burst loads, you’re going to need a system that sacrifices some speed or consistency in order to be distributed so it can handle the large volume of raw data.

Think about this trade off carefully before you go about building a real-time analytics system.

What Data Needs to be Real-time?

“Egads, ALL DATA SHOULD BE ALWAYS REPORTED IN REAL-TIME!” shouted every software developer ever.

Hold your horses! Real-time analytics forces a trade off between other important factors like accuracy / precision and data size. Therefore, real-time analytics isn’t inherently superior or better for every conceivable use case.

Real-time analysis is important for operational metrics and anything else you or your users need to respond to in real-time:

  • Error rates or health monitoring;

  • Dynamic prices, like stock prices or ticket fares;

  • On-the-spot personalizations and recommendations, like the product recommendations you might see when browsing Netflix or Ebay.

In these scenarios, the exact price or the exact error rate isn’t as important the rate of change or confidence interval, which can be done in real-time.

Retrospective or batch analysis is important for product / behavior analysis – these are metrics that tell you how you should or shouldn’t do something, and they are data that you can’t or shouldn’t respond to in real-time.

You don’t want to redesign your product based on fluctuations during day-to-day use – you want to redesign it based on long-term trends over all of your cohorts, and it naturally takes a long time for that data to accrue and be analyzed / studied.

In this type of analysis it’s more important for the data to be comprehensive (large) and accounted consistently.

Analytic speed is always a trade-off between data volume and consistency – you have to be concious of that when you design your system.

The one property you’re never going to want to intentionally sacrifice is data volume – data is a business asset. Data has inherent value. You want to design your analytic systems to consume and retain as much of it as possible.

At MarkedUp we use a blend of both real-time and retrospective analytics:

markedup analytics blend

In this post and the next we’re going to focus on how we use Cassandra for real-time analytics – we use Hive and Hadoop for our retrospective analysis.

Why Cassandra for Real-time Analytics?

Cassandra is an excellent choice for real-time analytic workloads, if you value speed and data volume over consistency (which we do for many of our metrics.)

So what makes Cassandra attractive?

    • Cassandra is highly available and distributed; it has high tolerance to individual node failures and makes it possible to add multi-data center support easily if data affinity or sovereignty is an issue. On top of that it’s easy to expand a Cassandra cluster with new nodes if necessary (although this shouldn’t be done frivolously since there is a high cost to rebalancing a cluster.)
    • It has amazing write performance; we’ve clocked Cassandra writes taking up to  200µs on average for us, and that’s doubly impressive considering that most of our writes are big, heavily denormalized batch mutations.
    • Batch mutations give us the ability to denormalize data heavily and update lots of counters at once – in Cassandra it’s generally a good idea to write your data to make it easy to read back out, even if that means writing it multiple times. Batch mutations make this really easy and inexpensive for our front-end data collection servers.
    • Distributed counters were added to Cassandra due at Twitter’s insistence, and they’re a major boon to anyone trying to build real-time analytic systems. Most of MarkedUp’s real time analytics are implemented using counters – they provide a simple, inexpensive, and remarkably consistent mechanism to update metrics and statistics at write time. There are some trade offs (namely the loss of idempotency) but they make up for it in simplicity and speed.
    • Physically sorted columns are one of the Cassandra database implementation details worth learning, because with it you can create easily predictable and pre-sorted slices of data. This makes for really efficient storage of time-series data and other common types of analytic output. When you combined physically sorted columns with dynamic columns and slice predicates you can create lookup systems which retrieve large data sets in constant time.
    • Dynamic columns are a Cassandra feature that takes getting used to, but they are enormously powerful for analytic workloads when coupled with sorted columns – they allow you to create flexible, predictable data structures that are easy to read and extend.

We’re going to publish a series of posts about working with Cassandra for real-time analytics. Make sure you read part 2 where we go into detail on our read / write strategy with Cassandra for analytics!

MetroAppSite: Free, Open Source Metro-Style Website Templates for Your Windows Store Apps

Getting customers to notice and discover your Windows Store apps is hard, but you can reach users who aren’t inside the Windows Store using simple websites designed to promote your apps.

In addition, if your Windows Store app requires access to the Internet you are required by Windows Store policy to publish and link to a privacy policy hosted online (section 4.1.1.)

We decided to make life a little easier for Windows Store developers and built MetroAppSite – a fully responsive Metro-style website that uses Twitter Bootstrap and other standard frameworks to help developers promote their Windows Store apps.

And like most of our customers, we’re a .NET shop, so we built an ASP.NET MVC4 version of MetroAppSite too!

Features

Here are some of the great features that you get with MetroAppSite:

Metro theming and branding

Give your promotional website the same Metro look-and-feel that your users experience when they download your app from the Windows Store.

We even include a Microsoft Surface screenshot carousel for you to use to show off your Windows Store app’s look-and-feel.

metro-branding-metroappsite

MetroAppSite uses BootMetro and Twitter Bootstrap to give Windows Store developers an easy-to-modify, brandable template they can use to their own ends.

Fully responsive and touch/mobile-friendly

MetroAppSite’s CSS and design is fully responsive and touch-optimized out of the box. It looks great in full-sized web browsers and on mobile devices too!

metroappsite-mobile
Integrates seamlessly with third party services like Google Analytics and UserVoice

Unfortunately there isn’t a MarkedUp Analytics for websites yet, but in the meantime we made it dead-simple to integrate MetroAppSite with Google Analytics so you can measure your pageviews and visitors.

uservoice-logo

Additionally, we added hooks to integrate UserVoice directly into your app’s site so you can collect feedback and support tickets from users easily and seamlessly. UserVoice is what we used for our customer support at MarkedUp and we’ve had a great experience with it!

Templated privacy policy in order to make it easy for you to satisfy Windows Store certification requirements

Writing privacy policies can be a pain, so we made it easy for you to generate a privacy policy for your app using PrivacyChoice.org. You can paste these right into MetroAppSite and meet Windows Store certification requirements easily and thoroughly.

Demo Sites

We created some simple MetroAppSite deployments for you so can see what they look like in production:

Download

MetroAppSite is licensed under the Apache 2.0 license and is free for you to use in commercial or non-commercial projects.

Contribution

We happily accept pull requests via Github.