SnoopShell: Evolution

It’s been a while since I last announced SnoopShell, where I took some PowerShell and injected that into Snoop.  Well, I didn’t stop there!  I decided to continue working on it and adding more useful features.

Well, a bunch of things have changed.  For one, it’s no longer targeted at .NET 4 and PS v3 anymore (and you’ll soon know why).  Second, there’s a bunch of new features!

Automatic Profile Loading

Upon startup, the shell will look for a couple well known locations and automatically dot-source them to load them into the current session.  This works the same as the standard $profile.  The filename needs to be SnoopProfile.ps1, and the search paths are %USERPROFILE%, the WindowsPowerShell, and the Scripts folder deployed with Snoop.exe.

This is incredibly useful since you can write your own custom functions and scripts and have them available to you all the time.  As an added bonus, because of the dynamic nature of PowerShell, you can make modifications to the SnoopProfile.ps1, save, and then invoke a “. $profile” to reload the profile and update the session with your changes (all without restarting the application).

That’s awesome sauce indeed ;-)

PowerShell Provider

This was more of a for-fun thing at first just to see if I could do it.  Writing a PS provider is not fun at all, since it’s not very well documented and I actually needed some help from ILSpy to figure out how things really worked.  Nonetheless, it’s got some basic functionality that is helpful to navigate around.

Yep, the selected grid actually has a path, like how you would navigate the file system.  Let’s see what happens with a cd.

Cool, you can cd into the child “directory”, and it’ll automatically select the item in the tree view as well.  What if you’re lazy and don’t want to type?

Wildcards are supported.  And because the visual tree doesn’t exactly require unique names, I needed to trick it by adding a number after each duplicate item.  So the above matches the third Rectangle child of the Grid.

Code Injection

One of the cool things about Javascript is that it’s so darn easy to test.  You make a change, save, reload, and you’ll immediately see if something worked or not.  This feedback loop is so fast it changes how you work and formulate ideas.

In the static world, we don’t really have this luxury, and especially not when you’re working on a large project, which at work, takes just under a minute for a full rebuild.  And this is on a monster machine.  Because of this, we had to employ tricks and workarounds to speed things up, like messing with build configurations and build output paths to minimize duplicate work.  Despite that, it’s still a pain to wait for the application to start and all that jazz.

What if we could do the super fast feedback loop development, in a static world?  Well, now you can!

It starts with a simple function:

function replace-command([string]$msg = 'hello world') {
    $action = { [system.windows.messagebox]::show($msg) }.GetNewClosure()
    $cmd = new-object galasoft.mvvmlight.command.relaycommand([system.action]$action)
    $selected.target.command = $cmd
}

The above function will replace anything that has a Command property on the target, like a Button or MenuItem, with a MessageBox showing a message.  For the curious, GetNewClosure is needed so that $msg is available within the inner script block.  Unlike C#, closures are not automatic.

Since PowerShell is dynamic, if you need to make a change, simply save the script, reload it with a dot-source, which will overwrite the existing function, and then set the target’s Command property again.  Awesome!

The only annoyance is converting PowerShell code back into C# code once you’re done.

Evolution

If you made it this far you didn’t forget about my comment about untargeting .NET 4 and PS v3.  Well, changes have been merged into the main branch!  Soon the masses will be able to experiment with supercharging their applications with PowerShell!

I’ll likely continue working on my fork as there’s still more goodies I’d like to add.  Stay tuned!

SnoopShell: The marriage of Snoop WPF and PowerShell

I was given the opportunity to review a couple chapters of the excellent book PowerShell for Developers, written by my colleague Doug Finke.  One of the concepts in the book was embedding a PowerShell console into your application.  This idea is ingenious and we added this feature to our client’s software, and so far it has increased our productivity and opened the doors to many possibilities.

So what’s so cool about embedding a shell into your application?  Well, for starters, one of the immediate advantages is that it gives you the opportunity to test your application at run time.  If you are implementing the MVVM pattern then basically anything you can see in the UI is bound to some property in your view model.  What if you could expose an instance of your view model to the PowerShell console?  Yes, you would be able to interact with it directly, change values, and property change notification will kick in and update the UI.

The possibilities start to open up from there.  You can start scripting out common tasks – write once, run many times.  Or you can write a full fledge test suite as a script, give it to a QA tester, and have them run through it as a special kind of integration testing, one that happens with live, real data.  Or how about being able to modify code, at runtime, to try out an implementation without need to recompile or restart the application?  Sounds pretty awesome to me!

With this, I started thinking why don’t I try and add this to Snoop?  It’s a staple tool for any WPF developer, and adding scripting capabilities to Snoop will make it even more useful.

So, I sat down for a weekend and took a shot at it.  And with that, SnoopShell was born!

My fork of Snoop can be found here: https://github.com/bling/snoopwpf

It’s still in super-duper alpha, so features/ideas are still getting formulated, but here’s a glimpse of what it can do now.

The $root variable points to the root of the tree.  As you can see, Snoop represents this as a ApplicationTreeItem, which has a bunch of properties, the important ones being IsSelected and IsExpanded.

Let’s try interacting with the object by setting the IsExpanded to true.

So far so good.  Now let’s find my username using Ctrl+Shift.  The $selected variable is automatically synchronized with the selected item in the tree.

Let’s do some black magic and change my name.

Finally, let’s find every ListBox in the application.  Find-Item is used to recursively find everything in the visual tree which is a ListBox.

And from here, it’s as simple as grabbing the DataContext of any control to get access to the view model.

By the way, this is targeting PowerShell V3, so you will need to have the RC installed.

Try it out and let me know what you think!

N2N: .NET 2 Node: Part 1

Let’s get some actual coding done.  If you want some basic prologue, check out the first part of my series here.
Anywho, let’s get to the meat of what we’re trying to accomplish.  I will start with the typical C# class you would write, do the same in Javascript, and then again in C#, but using the style of Javascript using closures.
Here’s the use case:

• Handle a web request
• Check the MongoDB database to see if an entry exists and return it if available.
• Otherwise, query an external API for data, save it into MongoDB, and then return that.

This is a pretty standard straight-forward approach in web applications.  If you’re not using MongoDB, you’ll be using memcache, Redis, or some other equivalent data store.

First, let’s implement an pseudo-code ASP.NET MVC controller which does the above:

public class QuoteController : Controller
{
    MongoCollection collection;
    IExternalApi external;

    public ActionResult Get(string symbol)
    {
        var quote = this.collection.FindOne(new { symbol });
        if (quote == null)
        {
            var json = this.external.GetQuote(symbol);
            this.collection.Save(json);
        }

        ViewBag.Quote = quote;
        return View();
    }
}

Error handling and initialization are omitted for brevity, but it’s an otherwise straight-forward synchronous send JSON to the browser response.  So how does something like this look in Node?  First, let’s throw in a web framework.  I picked Express because it seems to have the most traction right now.  Again, for brevity I will omit a bunch of boot strapping code, and skip to the core logic.

var QuoteService = function() {
    var mongoCollection; // new MongoCollection
    var external; // new External API

    // req/res are the request/response passed in by Node/Express
    this.get = function(symbol, req, res) {
        mongoCollection.findOne({ symbol: symbol }, function(error, item) {
            if (item === null) {
                external.getQuote(symbol, function(error, result) {
                    mongoCollection.save(result);
                    req.write(result);
                    req.end();
                });
            } else {
                req.write(item);
                req.end();
            }
        });
    };
};
module.exports = QuoteService;

OK, looks a lot different!  The first thing you’ll notice is that the class is actually a function.  Javascript doesn’t have real classes like C# or Java, but you can simulate things like private and public members by using closures.  In the code snippet above, mongoCollection and external are private variables.  However, the get function is public because it’s prefixed with this.  Finally, because Javascript has closures you can still access the private variables and retain their state.

Last but not least, as far as programming in Node is concerned, every function returns void.  Node is extremely fast – but it is still single threaded, which means you need to take extreme care not to block on any operation.  In summary, you need to get used to working with callbacks.  In the example above, the callback for the Mongo query invokes another function, which again uses a callback for the result.  Typically any errors are also passed through to the callback, so in place of try/catch blocks, you will be checking to see if the error parameter has a value.

To conclude, let’s write a C# version that mimics the Javascript style.

public static void Main(string[] args)
{
    var QuoteService = new Func<dynamic>(() => {
        var mongoCollection = new MongoCollection();
        var external = new IExternalApi();
        return new
        {
            get = new Action<string, Request, Response>((symbol, req, res) =>
            {
                mongoCollection.FindOne(new{symbol}, (error, item) => {
                    if (item == null)
                    {
                        external.GetSymbol(symbol, (e, result) => {
                            mongoCollection.Save(result);
                            req.Write(result);
                            req.End();
                        });
                    }
                    else
                    {
                        req.Write(item);
                        req.End();
                    }
                });
            })
        }});
}

100% valid C# syntax.

Would you ever want to do that in C#?  Probably not.  But if you need some mental conversion from C# to/from Javascript, I think it’s a good place to start.

In summary, Javascript is kind of like programming in C# using only anonymous classes, Action/Func, dynamic, declared all in the main method.

N2N: .NET 2 Node

Well, it’s been quite a while since I’ve blogged about…well…anything, and I figured it’s about time I get off my lazy butt and do something with my spare time on weekends.  What better option than to see what all the hype is about Node?  I had to do it sooner or later.

As any newbie would do, they go to Google and type “nodejs tutorial”.  The Node Beginner Book came up first, so I went with that.  It was an excellent tutorial.  Prior to this I also skimmed through the book JavaScript, The Good Parts, so I had a basic understanding of the language syntax.

One of the first oddities I noticed, was that NodeJS seems to have a convention of comma-first.  You notice this immediately because most examples start with require(‘module’), and if they require more than one module, the second line is prefixed with a comma (as opposed to the more traditional comma at the end of the line).  I apparently missed the discussion by 2 years!  It was still interesting nonetheless.

As someone with a strong .NET background, I definitely experienced all the usual ‘gotchas’:

• == vs ===
• falsey values
• variable hoisting

Once you understand all of these things, Javascript isn’t so bad.  Oh, and of course understanding closures will get you a long way in being effective with Javascript, because that’s what you need to use to do proper scoping.  If C# didn’t have lambdas and closures it would have been a much longer journey to “get it”.

Not too longer after, I deployed my first Heroku app running on NodeJS.

Anyways, enough with the prologue…I won’t bore you with anymore beginner/tutorial stuff.

Let’s get on with what I plan on doing over a multi-part blog series.  When I build something on my own time, I can’t build something just for the hell of it to learn something….that’s not enough.  If I build something it has to be useful – something that I (or someone else) will find valuable.

I won’t reveal what it is yet, but it’s going to involve Node/MongoDB on the backend, with Backbone on the front-end.  Should be fun :-)

My Thoughts on MEF

Ever since MEF was conceived, despite the authors saying that it is not an IoC container, it has since evolved to become one of the more popular IoC containers.  I’ve always avoided it because I disagree with using attributes, and I’ve had no reason to use it over Autofac or Windsor.

Recently, I found a reason to use it – Metro-style applications only support MEF so far.  My Twitter client ping.pong uses Autofac as the IoC container.  It uses some very basic functionality like factories and hooks.  To my surprise, MEF has no support for either of these.

Coming across these limitations solidifies my opinion that MEF is a plugin container, not an IoC container.

First let’s take a look at automated factories. What I mean is that by registering Foo, like so:

container.RegisterType<Foo>();

the container will automatically provide us a Func<Foo> without explicitly having to register it. This can be useful when you want to create an instance of Foo some time in the future rather than at constructor time.  You can do this with MEF via an ExportFactory<T>, but it’s limited because you cannot override dependencies at resolve time.

For example, let’s say Foo has a constructor of Foo(Bar1, Bar2, Bar3). With MEF, you have no control at resolution time what the Bars are. A container that has support for automated factories (like Autofac and Castle Windsor), will let you resolve a Func<Bar1, Foo>, which lets you override Bar1 at resolve time. Similarly, you can resolve a Func<Bar1, Bar2, Bar3, Foo> and override all dependencies. Any dependencies not overridden fall back to their configuration in the bootstrapper. This is a very useful feature, and coupled with the scoping features for automatic disposal it opens up many doors for elegant solutions for what otherwise are complicated problems.

On to the second point; MEF has limited extension points. This one sounds odd since MEF is all about designing decoupled plugins so surely it should have extension points! The problem here is that MEF is designed as an explicit API (attributes are required) rather than an implicit API. In Autofac, you can scan an assembly and register every type. In MEF, every class needs to have an [Export] on it.  It also baffles my mind why [ImportingConstructor] is required even when there’s only one constructor. All this explicitness means you lose a bunch of “free” extension points that typical IoC containers have, like this:

b.RegisterAssemblyTypes(GetType().Assembly)

  .OnActivated(x => x.Context.Resolve<IEventAggregator>().Subscribe(x.Instance));

What the code above is saying that every time any component is activated, it will subscribe to the event aggregator. If the component doesn’t IHandle<> any messages, it’s a no-op and continues on. If the instance does IHandle<> messages, this will ensure it’s hooked up.

The closest thing I could find in MEF was IPartImportsSatisfiedNotification (yes, an interface, more explicitness!).  It contains a single method OnImportsSatisfied() which gets called when the part is created.  Needless to say, the one line of code from Autofac would translate into a method for every implementation of IHandle<>, and since OnImportsSatisfied() contains no contextual information, every component will need IEventAggregator injected just to be able to call Subscribe.

To fully complete this example, Autofac has the following methods when registering a component: OnRegistered, OnPreparing, OnActivating, OnActivated, and OnRelease.  Each of these methods gives you complete contextual information at the time it is called like access to the current scope of the container, the instance (if applicable), which component which requested the dependency, etc.  This makes it almost too easy to extend the container.

For MEF, the only real extension point is an ExportProvider.  It is pretty low level (all it does is parse attributes for you) so to write anything similar for MEF requires a lot more code.  To further illustrate this point, compare the interception modules from AutofacContrib and MefContrib.  The Autofac implementation is a single file with a couple extension methods.  The MEF implementation is an entire namespace, over multiple classes, not the mention that it also relies on other infrastructure code in MefContrib.  Basically, the guys that wrote MefContrib had to write a mini-container within MEF.

MEF is great for building extremely loosely coupled applications.  I don’t think it has any business in an application where you know and own all of the dependencies; there are simply better libraries for that.

ping.pong Twitter Client

I have a whole series dedicated to blogging about how I wrote a Twitter client from scratch, but if I want anyone to actually use it, I better do some advertising :-D.

ping.pong is a fast and lightweight Twitter client written in Silverlight.  As of this moment it targets v4 but will likely target v5 whenever that is released.

Here are some highlights…

Visually Pleasing

The UI is based on your typical column-based design.  The column widths will automatically resize to take up all available horizontal space.  There is no horizontal scrolling.

Access to the Streaming API

No more rate limits!  Your timeline is connected directly with Twitter’s streaming API, which means whenever someone you follow tweets you will know about it in almost near real-time.  Searching is also done through the streaming API.

Fast & Lightweight

ping.pong was built to run fast with low CPU utilization.  Even when stream searching for “e” (yes, the letter E), the CPU usage stays under 15%.  The maximum number of tweets that Twitter sends appears to be 50 per second.

Conversations

ping.pong will quickly show an entire tweet conversation by navigating reply tags back to the original tweet that started it all.

Free & Open Source

The full source code for ping.pong can be found on GitHub.  You can compile it, make modifications as you please, and run it yourself.  The only thing missing is the consumer keys which uniquely identifies this client from another.  You can generate them through Twitter once you have a developer account.

Installer

The latest and greatest can be quickly installed from here.  Check back periodically for updates!

Push Driven Development with Reactive Extensions

This is going to be the last post that concludes my series on building a real-time push app with Silverlight.  Any additional posts would likely be outside the context of writing a push app and more about how I’m adding features to ping.pong, my Twitter app, so I think this is a good place to wrap up and talk generally from a top down overview of building a push-style application.

Here’s a recap of everything discussed so far:

Part 1:  Basics – Creating an Observable around a basic HTTP web stream against Twitter’s streaming API

Part 2:  Subscription and Observation of Observables

Part 3:  Basics of UX design with a look at shadows and gradients.

Part 4:  Integrating with 3rd party libraries, notably Caliburn Micro and Linq2Twitter and how to achieve polling with observables.

Part 5:  A minor hick up with Linq2Twitter.

Part 6:  Taking advantage of transparencies to improve the design and reusability of UX.

Part 7:  A summary of all things encountered so far, replacing Linq2Twitter with Hammock, first release of code to GitHub, and a binary released capable pulling and streaming tweets from Twitter.

Part 8:  Examples of using Caliburn Micro to easily resolve data bindings that otherwise would be much more effort.

And that leads us to this post…

PDD (Push Driven Development)

One of the main goals of this series is to create a performant Silverlight app based on push principles, as opposed to more traditional pull principles.  To that effect, ping.pong has performed remarkably well and is limited only by Twitter’s throttling, which currently appears to be maximum of 50 tweets per second via the streaming API.

Writing the application from a push-driven mindset was definitely unintuitive at first, and I had to refactor (actually rewrite is more accurate) the code many times to move closer to a world where the application is simply reacting to events thrown at it (as opposed to asking the world for events).

To be absolutely clear on what I mean on the differences between push and pull, here’s a comparison:

Pulling	Push
var e = tweets.GetEnumerator(); while (e.MoveNext()) // is there more? {   e.Current; // get current   DoSomething(e.Current); }	IObservable<Data> data = /* get source */ // whenever data comes, do something data.Subscribe(DoSometing);

On the pulling side, the caller is much more concerned with the logic on how to process each message.  It needs to repeatedly ask the world, “hey, is there more data?”.

On the push side, the caller merely asks the world, “hey, give me data when you get some”.

Twitter is a perfect example because their APIs have both a pulling and pushing models.  Traditional clients poll continuously all the time, and many had configurable options to try to stay under the 200 API calls per hour limit.  Most of Twitter’s API still consists of pulling, but the user’s home line and searching can be streamed in near real time via the streaming API, aka. push.  Streaming tweets effectively removes the API call limit.

Push and Pull with Reactive Extensions

The beauty of Rx is that regardless of whether it is actually pushing or pulling, the API will look same:

IObservable<Tweet> tweets = _client.GetHomeTimeline();

tweets.Subscribe(t => { /* do something with the tweet */ });

As far as the caller is concerned, it doesn’t care (or needs to know) whether the GetHomeTimeline method is polling Twitter or streaming from Twitter.  All it needs to know is when a tweet comes it will react and do something in the Subscribe action.

In fact, Subscribe simply means “when you have data, call this”, but that could also be immediately, which would be analogous to IEnumerable.

However, if that was the only thing Rx provided it wouldn’t be as popular as it is, because other pub/sub solutions like the EventAggregator already provide a viable asynchronous solution.

Unlocking Rx’s power comes with its multitude of operators.  Here’s an example:

public static IObservable<Tweet> GetStreamingStatuses(this TwitterClient client)

{

  return client.GetHomeTimeline()

      .Merge(client.GetMentions())

      .Concat(client.GetStreamingHomeline());

}

GetHomeTimeline and GetMentions initiate once-only pull style API calls, while GetStreamingHomeline will initiate a sticky connection and stream subsequent tweets down the pipe.

The Merge operator is defined as this: “Merges an observable sequence of observable sequences into an observable sequence.”

I think a better description would be “whenever there is data from any of the sources, push it through”.  In the example above, this would translate to whenever a tweet comes from either the home timeline or the mentions timeline, give me a Tweet (first-come-first-push), followed by anything from the streaming timeline.

And there lies one of the greatest beauties of Rx.  All of the complexity lies solely on setting up the stream and operators.  And that, also, is its disadvantage.

Rx Complexity

Let’s take a look at the Concat operator, defined as: “Concatenates two observable sequences.”  In the remarks sections it states this: “The Concat operator runs the first sequence to completion. Then, the second sequence is run to completion effectively concatenating the second sequence to the end of the first sequence.”

Let’s try it out:

var a = Observable.Interval(TimeSpan.FromSeconds(1)).Select(x => x.ToString());

var b = Observable.Interval(TimeSpan.FromSeconds(1)).Select(x => "s" + x);

a.Concat(b).Subscribe(Console.WriteLine);

// output: 0, 1, 2, 3, 4...

As expected, only numbers are printed because the first sequence never ends, so it won’t concatenate the second one.  Let’s modify it so that it does finish:

int count = 0;

var a = Observable.Interval(TimeSpan.FromSeconds(1))

    .TakeWhile(_ => ++count < 5)

    .Select(x => x.ToString());

Note, that using Observable.Generate is preferred because it doesn’t introduce an external variable, but I stuck with Interval so the code looks similar to the second observable.  As expected again, it will print “0, 1, 2, s0, s1, s2”.

OK, let’s spice things up.  Let’s make b a ConnectableObservable by using the Publish operator, and immediately call Connect.

int count = 0;

var a = Observable.Interval(TimeSpan.FromSeconds(1))

    .TakeWhile(_ => ++count < 5)

    .Select(x => x.ToString());

var b = Observable.Interval(TimeSpan.FromSeconds(1)).Select(_ => "s" + _).Publish();

b.Connect();

a.Concat(b).Subscribe(Console.WriteLine);

What do you think the output of this will be?  The answer is “0, 1, 2, 3, s5, s6, s7, …”

Despite using the same Concat operator, the result can be very different depending on the source observables.  If you use the Replay operator, it would have printed “0, 1, 2, 3, s0, s1, s2, …”

Years and years of working in synchronous programming models have trained us to think in synchronous ways, and I picked Concat specifically because Concat also exists in the enumerable world.  Observable sequences are asynchronous, so we never know exactly when data comes at us, only what to do when it does.  And because streams occur at different times, when you combine them together there are many many ways of doing so (CombineLatest, Merge, Zip, are just a few).

The greatest hurdle to working in Rx is to know what the different combinations do.  This takes time and practice.  RxTools is a great learning tool to test out what all the operators do.

Unit Testing

Last but not least, Rx can make it easier to write unit tests.  The concept is easy: take some inputs and test the output.  In practice this is complicated because applications typically carry a lot of state with them.  Even with dependency injection and mocking frameworks I’ve seen a lot of code where for every assert there is 10 lines of mock setup code.

So how does it make it easier to test?  It reduces what you need to test to a single method, Subscribe, which takes one input, an IObservable<T>.

Conclusion

Rx is a library unlike any other you will use.  With other libraries, you will add them to your solution, use a method here or there, and go on with your life.  With Rx, it will radically change the way you code and think in general.  It’s awesome.