Building a Real-time Push App with Silverlight: Part 3

In this part we’re going to fire up Expression Blend (the trial for version 5 can be found here) and do some UI work.

In part 2, I created a simple Twitter client which connected to the streaming API, and connected to the sampling request which brings back random tweets.  Here is the data template:

<DataTemplate x:Key="TweetDataTemplate">

    <Grid DataContext="{Binding}">

        <Grid.RowDefinitions>

            <RowDefinition />

            <RowDefinition Height="Auto" />

        </Grid.RowDefinitions>

        <TextBlock FontFamily="{StaticResource FontFamily}" FontSize="12" Text="{Binding Text}" TextWrapping="Wrap" />

        <TextBlock Grid.Row="1"

                   HorizontalAlignment="Right"

                   VerticalAlignment="Bottom"

                   FontFamily="{StaticResource FontFamily}"

                   FontSize="13.333"

                   Foreground="BlueViolet"

                   Text="{Binding ScreenName}" />

        <TextBlock Grid.Row="1"

                   HorizontalAlignment="Left"

                   VerticalAlignment="Bottom"

                   FontFamily="{StaticResource FontFamily}"

                   FontSize="9.333"

                   Foreground="DarkCyan"

                   Text="{Binding CreatedAt}" />

    </Grid>

</DataTemplate>

This renders into something like this:

The text is randomly generated from Blend’s sample data capability, which is totally awesome as it allows designers to see what they’re working with, and keeps the sample data separate from the real data.

While design is a matter of personal taste, and you’re bound to get disagreements between different people, if you follow some basic rules you’ll satisfy a greater audience.

• Subtle gradients and small shadows
  • If you take a look at all the nice interfaces, they tend to use very slight gradients and small shadows.  Most of the time you don’t even notice unless you look closely.
  • I think Microsoft’s Metro design is beautiful.  Reason?  It emphasizes text over decorations (like gradients and shadows).  This tends to lead to very clean design because there’s very little opportunity to abuse gradients and shadows.
• Realism and light sources
  • Continuing on with gradients and shadows, they should be realistic.  Look at your design from a 3D point of view.  Apply a light source from a certain angle, and then apply your shadows relative to that light source.
  • Convey distance properly
  • Darker shadows imply being closer to the background, whereas lighter shadows imply being further away.  Use blurring to add emphasis to the distance.
    
  • If you overlap planes you should apply these rules to each individual plane.  Don’t use the same border for everything.  Think about how it would look like in real life if you laid it out like that with pieces of paper.  The shadow sizes for that will be different, so you should do the same.
  • Also keep in mind that the shadows used above are way too much for any application.  Be subtle!
• Consistent theme
  • This one seems obvious but nothing is worse than having a nice looking application bring up an unskinned dialog.
• Usability
  • If the design doesn’t serve a purpose to make it more usable, it shouldn’t be there.  Even something as simple as black on white follows this – you do that so you can read text.  However, even something as simple as that can be improved.  Take a look at why the Kindle is so successful.  The readability is better because of the lower contrast between the black and light-brown background.

With these starting points, let’s redesign the data template.

<DataTemplate x:Key="TweetDataTemplate">

   <Grid>

       <Grid.Background>

           <LinearGradientBrush StartPoint="0.5,0" EndPoint="0.5,1">

               <GradientStop Color="#FFDADADA" />

               <GradientStop Offset="1" Color="#FFC8C8C8" />

           </LinearGradientBrush>

       </Grid.Background>

       <Grid.RowDefinitions>

           <RowDefinition />

           <RowDefinition Height="Auto" />

       </Grid.RowDefinitions>

       <TextBlock FontFamily="{StaticResource FontFamily}" FontSize="12" Text="{Binding Text}" TextWrapping="Wrap" />

       <TextBlock Grid.Row="1"

                  HorizontalAlignment="Right"

                  VerticalAlignment="Bottom"

                  FontFamily="{StaticResource FontFamily}"

                  FontSize="13.333"

                  Foreground="BlueViolet"

                  Text="{Binding ScreenName}" />

       <TextBlock Grid.Row="1"

                  HorizontalAlignment="Left"

                  VerticalAlignment="Bottom"

                  FontFamily="{StaticResource FontFamily}"

                  FontSize="9.333"

                  Foreground="#FF003D8F"

                  Text="{Binding CreatedAt}" />

       <Border Grid.RowSpan="2" BorderBrush="#FF999999" BorderThickness="0,0,0,1" />

       <Border Grid.RowSpan="2" BorderBrush="White" BorderThickness="0,1,0,0" />

   </Grid>

</DataTemplate>

After these changes, it looks like this:

Did you notice the gradient?  You might think after seeing it here to adjust the gradients more so you can see it.  That would be a mistake.  See below.

 

 

To the right is the exact same thing, but stacked vertically three times.  When this happens the subtle difference between the top and bottom of the control is more pronounced, so it looks like multiple panels are aligned together.

 

 

 

 

However, there’s still a little touch you can add.  The white and gray borders are only 1 pixel high, but that’s the little touch needed to make it look crisp.

 

 

 

 

Finally, let’s see the before and after (or eh…rather after and before, because I took the screenshot backwards :P):

Building a Real-time Push App with Silverlight: Part 2

Let’s review the main Rx code from last time:

public IObservable<string> GetJsonStreams()

{

  var request = GetRequest();

  return Observable.FromAsyncPattern<WebResponse>(request.BeginGetResponse, request.EndGetResponse)()

    .Select(wr => wr.GetResponseStream())

    .Select(str => Observable.FromAsyncPattern<byte[], int, int, int>(str.BeginRead, str.EndRead))

    .SelectMany(ParseJson);

}

One thing I didn’t like about this was that the web request object was created regardless of whether the Observable gets a subscription or not.  This is potentially wasted resources, and I wanted to refactor this to be completely lazy.

And with this I started to run into my first “huh?” moments with Rx: I blocked the UI thread.  How did I do that?  I started down the path of exploring some more of the Rx methods, which lead me to Create, which lets you manually call OnNext.  With this train of thought, I came up with something like this:

return Observable.Create<string>(obs =>

{

  var request = GetRequest();

  var response = Observable.FromAsyncPattern<WebResponse>(request.BeginGetResponse, request.EndGetResponse)().First();

  var str = response.GetResponseStream();

  var reader = Observable.FromAsyncPattern<byte[], int, int, int>(str.BeginRead, str.EndRead);

  foreach (var json in ParseJson(reader))

      obs.OnNext(json);

  obs.OnCompleted();
  return str;

});

Great!  The initialization of the web request only occurs when subscribed!  And it will even dispose the stream (by returning str) upon unsubscription.  I ran the app and the UI thread immediately blocked.  What happened?

Rx has the concept of subscription and observation, and provides a way to subscribe and observe on different threads.  Here is the original code that subscribed:

s.GetJsonStreams()

   .ObserveOnDispatcher()

   .Subscribe(x => Text = x);

Can you spot the error?  I explicitly told Rx to observe on the dispatcher thread, because I want the action inside Subscribe to be invoked on the UI thread, but I didn’t specify where I want to set up the subscription.  Since I left it out, it uses the current thread, which happens to be the UI thread.  To solve this, it’s as simple as doing this:

s.GetJsonStreams()

  .SubscribeOn(Scheduler.ThreadPool)

  .ObserveOnDispatcher()

  .Subscribe(x => Text = x);

That’s it!  Easy!  This also follows one of the most important guidelines when using Rx: Subscription and Observation should be done as late as possible, typically just before the Subscribe.  Anything more and you’ll likely make Rx spawn more threads than are necessary or some other nasty bugs.  KISS!

Now with that out of the way, let’s replace the boring TextBlock with something more usable.  First, I need to parse all the JSON streams I’m getting into bindable models.  To do that, I upgraded my StreamReader component and threw in System.Json for some basic parsing:

public class TweetParser

{

    private int _stack;

    private readonly StringBuilder _sb = new StringBuilder();

    public IEnumerable<Tweet> Parse(byte[] buffer, int count)

    {

        for (int i = 0; i < count; i++)

        {

            var current = (char)buffer[i];

            _sb.Append(current);

            if (current == '{') _stack++;

            else if (current == '}') _stack--;

            if (_stack == 0 && _sb.Length > 0)

            {

                Tweet tweet;

                var value = JsonValue.Parse(_sb.ToString());

                if (value is JsonObject && Tweet.TryParse((JsonObject)value, out tweet))

                    yield return tweet;

                _sb.Clear();

            }

        }

    }

}

Nothing overly complicated.  Next, the Tweet object:

public class Tweet

{

    private readonly JsonObject _json;

    public static bool TryParse(JsonObject value, out Tweet tweet)

    {

        if (value.ContainsKey("text") && value.ContainsKey("user"))

        {

            tweet = new Tweet(value);

            return true;

        }

        tweet = null;

        return false;

    }

    private Tweet(JsonObject json)

    {

        _json = json;

    }

    public string Text

    {

        get { return _json["text"].ToValueString(); }

    }

    public string ScreenName

    {

        get { return _json["user"]["screen_name"].ToValueString(); }

    }

}

internal static class TweetEx

{

    public static string ToValueString(this JsonValue s)

    {

        return s.ToString().Trim('"');

    }

}

To keep things simple I’m only extracting the screen name and text.  I won’t bore you setting up the views since it’s just simple ListBox bound to an ObservableCollection<Tweet>, and a DataTemplate for Tweet.  When it’s all said and done, we see something like this:

Performance is still good at 2-5% CPU, even though we’re scrolling through 1000 items in near real-time.

Stay tuned for part 3, when we introduce Expression Blend and go into basics of UI design.  Also, most of this will hit GitHub very soon.

Building a Real-time Push App with Silverlight: Part 1

This is the beginning of a multi-part series where I’ll be building an interactive application with Silverlight 5 (still beta as of this post).  It will be built from the ground up and designed predominantly from a “push data” point of view, where the application is reacting to events in real-time, rather than a more traditional “pulling” point of view.  This type of application has exploded with the popularity of social networking and has made a lot of the traditional methods of building applications obsolete.

The best example of this shift is from Twitter.  When it first came out, it was (and still is predominantly) a “pull” model.  You have 200 API requests an hour, and you pull Twitter whenever you want to check if there are tweets of people you follow.  That is changing with the Streaming API where data is “pushed” to you as it comes.  This changes the way you write your code and requires a mind shift much like from for loops to LINQ.

The primary purpose of this series is build a real-time push application from beginning to end, and to show any problems I run along the way, and how I managed to solve them.  This will be my first attempt at building a Silverlight application as well as learning Reactive Extensions (Rx), so I’m bound to run into newbie mistakes.  If you have any tips or pointers please let me know!

Also, rather than doing this from a purely technical point of view, I’m also going to put on my designer hat and talk about UI design, and make it look good with Expression Blend when I get to designing the UI in later parts of this series.  I feel that this is often overlooked and can result in a lot of wasted work and lead to frustration.

This first post will be to get up and running and connected to Twitter with the Streaming API. This was chosen primarily because it is so much data can be pushed through and it can be used to demonstrate how to write a high performance Silverlight application.  It’s one thing to write something maintainable, and another altogether to make it run fast as well.  I’ve worked on too many WPF projects where performance took a back seat, and in WPF particularly this tends to create some massive technical debt.  The Silverlight/WPF technology stack is an exception to the rule and you should definitely think about performance from the start.

Well let’s get started!  We will connect to Twitter, convert it to an Observable, and then display tweets on the UI.

First, let’s create our TwitterStream class:

   1: public class TwitterStream

   2:     {

   3:         static TwitterStream()

   4:         {

   5:             // this allows http authentication to work

   6:             WebRequest.RegisterPrefix("http://", System.Net.Browser.WebRequestCreator.ClientHttp);

   7:         }

   8:  

   9:         protected readonly StreamParser parser = new StreamParser();

  10:  

  11:         public string Username { get; set; }

  12:         public string Password { get; set; }

  13:  

  14:         protected HttpWebRequest GetRequest()

  15:         {

  16:             var request = WebRequest.CreateHttp("http://stream.twitter.com/1/statuses/sample.json?delimited=length");

  17:             request.UseDefaultCredentials = false;

  18:             request.Credentials = new NetworkCredential(Username, Password);

  19:             return request;

  20:         }

  21:  

  22:         public IObservable<string> GetJsonStreams()

  23:         {

  24:             var request = GetRequest();

  25:             return Observable.FromAsyncPattern<WebResponse>(request.BeginGetResponse, request.EndGetResponse)()

  26:                 .Select(wr => wr.GetResponseStream())

  27:                 .Select(str => Observable.FromAsyncPattern<byte[], int, int, int>(str.BeginRead, str.EndRead))

  28:                 .SelectMany(ParseJson);

  29:         }

  30:  

  31:         private IEnumerable<string> ParseJson(Func<byte[], int, int, IObservable<int>> reader)

  32:         {

  33:             var buffer = new byte[256];

  34:             int bytesRead;

  35:  

  36:             while ((bytesRead = reader(buffer, 0, buffer.Length).Single()) > 0)

  37:             {

  38:                 foreach (var json in parser.Parse(buffer, bytesRead))

  39:                     yield return json;

  40:             }

  41:         }

  42:     }

And that’s all there is to it!  The most complicated part is probably parsing of the JSON documents themselves, which I refactored into its own class.  The algorithm in the parser is pretty primitive, as it just looks for opening { and closing } and calls that a document – nothing more, nothing less.  Note that this uses the deprecated basic HTTP authentication.  Sooner or later it will be shut down and OAuth will be required, so I will upgrade when that happens as I want to keep the amount of written code to a minimum.

Perhaps the more interesting bit is this part with Rx:

Observable.FromAsyncPattern<WebResponse>(request.BeginGetResponse, request.EndGetResponse)()

  .Select(wr => wr.GetResponseStream())

  .Select(str => Observable.FromAsyncPattern<byte[], int, int, int>(str.BeginRead, str.EndRead))

  .SelectMany(ParseJson);

The API takes some getting used to, but once you “tune in” things start to make sense.  The FromAsyncPattern essentially wraps the APM pattern’s Begin/End calls into a single Func<>, which when invoked gets you the result as if you called End(), however without all the tedious AsyncCallback implementation.

The nice thing about this wrapper Func is that the code starts to look synchronous, even though it is using the ThreadPool behind the scenes.  Next, the response is projected into its response stream, which again is converted to another Observable for reading bytes from the stream until finally it goes into the ParseJson method which projects string results.  This is both good and bad.  Good in that it can make asynchronous code look short and succinct (in this example I’ve wrapped 2 APM pattern calls in 2 lines of code), bad in that anyone inexperienced with Rx will get lost pretty fast.

Rx thus far has been a fairly high learning curve.  Maybe it’s just me.  When I read about it from others, or from seminars, it all makes sense and I generally don’t have any questions.  Trying to use it directly is another story, as there are so many overloads it’s easy to get overwhelmed.  I feel like Intellisense is making things worse!

Carrying on, the code above actually doesn’t do anything yet.  First, you must Subscribe to an observable before it does something, similar to you must foreach on a IEnumerable before it starts pulling data.

To finish off, let’s create a simple Window with a TextBlock bound to a Text property.  This is the constructor:

public MainPage()

{

   InitializeComponent();

   DataContext = this;

   var s = new TwitterStream();

   s.Username = "blingcoder";

   s.Password = "1234567";

   s.GetJsonStreams()

       .ObserveOnDispatcher()

       .Subscribe(x => Text = x);

}

Perhaps one of the nicest features of Rx is automatic thread switching between background threads and UI threads.  Above, there is a call to ObserveOnDispatcher, which as the name implies it observes to events on the Dispatcher thread :-).  Rx automatically handles switching to the UI thread so when Subscribe occurs we’re on the UI thread.

Performance is also very good so far, only maxing out at 5% CPU for what appears to be a continuous flood of tweets from random people.

And there you have it!  A completely asynchronous streaming Twitter client in roughly 20 lines of code (minus the parsing).  Stay tuned for part 2….