MetaTrader Indicators

One of my hobbies is forex trading, and a sub-hobby of that is writing custom indicators. MetaTrader is a trading platform that is very popular in the forex community because it allows traders to write custom indicators. It's pretty easy to get started if you have programming experience because the language is very similar to C. And yes, you can also write your own bot if you wanted, called an Expert Advisor. Personally I don't spend too much time trying to write a winning bot simply because there are way too many variables to take into account, and eventually the market's going to yell "in your face" and your winning bot will end up losing. I just write simple scripts that help me identify entry points, and filter out all the non-interesting price movements. Recently I've been looking at breakout strategies because they are simple, yet effective. The result is a custom indicator I wrote which simply put calculates the high and low for a time period, and then draws pretty rectangles. Like so: The above is simply the high/low of 2 hours before the opening London session, and the opening US session. Isn't it puurrrrrrty?

I'm doing the VI challenge!

One of my time-wasting hobbies is running Linux on virtual machines. I play games from time to time so Windows is still my main OS. But otherwise, I do use my Linux virtual machines for many things like my personal Mercurial repository is running on VirtualBox. For the curious, my distro of choice is Archlinux. Anyone who's needed to tinker with a Linux distro knows that you spend a lot of time in a text editor changing configuration files. Typically, the choices are pico/nano for its simplicity, or vi because it's on any POSIX compliant operating system. I decided, ah what the heck, I'm gonna learn vi and use that for editing files instead of nano. I got decent with vi. I could move around (albeit I still relied on home/end and arrow keys), save, search, etc., but on Windows I'd still use Notepad2. In an attempt to improve my productivity in text editing, I'm doing the VIM challenge! All my text editing I will be using WinVI (I even replaced the notepad.exe with it), and I even got the demo of ViEmu running on Visual Studio. For the next 2 weeks I will be using vi for anything text related. I've been going through tutorials and trying to remember all the new things I'm learning. Most importantly, I'll be trying to avoid the home/end/arrow keys as well. BTW, '*' is an awesome feature in vi. It highlights all instances of the current word in the document, and subsequently you can use n or N to back or to the next instance. Like, take a look at the following screen shot: Boom! All instances of "total" highlighted with just a Shift+8. Let's see you do the same thing with Ctrl+F, mouse-click "Find next" (or worse, reverse your search).

A Trick Question With Closures

Given the following code, what do you expect the output to be?

for (int i = 0; i < 100; ++i)
{
    ThreadPool.QueueUserWorkItem(delegate
    {
        Console.WriteLine(i);
    });
}

Keep that answer in your head! Now...what do you expect the output of the following?

int i;
for (i = 0; i < 100; ++i)
{
    ThreadPool.QueueUserWorkItem(delegate
    {
        Console.WriteLine(i);
    });
}

Were your answers the same? Different? Why?

A Simple Thread Pool Implementation (Part 2)

I said last time I'd compare the performance of my super duper simple implementation against the .NET ThreadPool, so here it is! Here's the test code:

ManualResetEvent evt = new ManualResetEvent(false);
int total = 100000;
int count = 0;
DateTime start = DateTime.Now;
for (int i = 0; i < total; ++i)
{
//  ThreadQueue.QueueUserWorkItem(delegate(object obj)
    ThreadPool.QueueUserWorkItem(delegate(object obj)
    {
        Console.WriteLine(obj);
        if (Interlocked.Increment(ref count) == total)
            evt.Set();
    }, i);
}
evt.WaitOne();

Console.WriteLine("Time: {0}ms", (DateTime.Now - start).TotalMilliseconds);
Console.ReadLine();

Here are the initial tests. My CPU is a Core2Duo clocked at 3.6GHz. I also ran a release build outside of the debugger. I set my ThreadQueue to have 25 worker threads. The results were pretty surprising:
ThreadPool: 1515.6444ms ThreadQueue: 5093.8152ms Well, that's interesting. The .NET ThreadPool whooped my butt! How can that be? Let's dig deeper and find out how many threads actually got used. I used ThreadPool.GetMinThreads and ThreadPool.GetMaxThreads, and I got 2-500 for worker threads, and 2-1000 for completion IO threads. Then, I tracked how many threads were actually used, like this:

Dictionary threads = new Dictionary();
// ...snip
    ThreadPool.QueueUserWorkItem(delegate(object obj)
    {
        Console.WriteLine(obj);
        threads[Thread.CurrentThread.ManagedThreadId] = 0;
        if (Interlocked.Increment(ref count) == total)
            evt.Set();
    }, i);
// ...snip
Console.WriteLine("Threads used: {0}", threads.Count);

Surprisingly, only 2 threads were used. So, I set the number of worker threads of my ThreadQueue to 2. Viola! 1437.5184ms, which is just a little under 100ms faster than the ThreadPool. I guess this shows that more threads does not mean better or faster! For fun I set the number of worker threads to 200 and it took 27906.6072ms! There was clearly a lot of locking overhead here...

A Simple Thread Pool Implementation

I've always wanted to do this, and finally I've gotten around to doing it. Here's a super duper simple implementation of a working thread pool. I named in ThreadQueue just so it's clearly discernible from the one provided by the framework.

public static class ThreadQueue
{
    static Queue _queue = new Queue();

    struct WorkItem
    {
        public WaitCallback Worker;
        public object State;
    }

    static ThreadQueue()
    {
        for (int i = 0; i < 25; ++i)
        {
            Thread t = new Thread(ThreadWorker);
            t.IsBackground = true;
            t.Start();
        }
    }

    static void ThreadWorker()
    {
        while (true)
        {
            WorkItem wi;
            lock (_queue)
            {
                while (_queue.Count == 0)
                {
                    Monitor.Wait(_queue);
                }
                wi = _queue.Dequeue();
            }
            wi.Worker(wi.State);
        }
    }

    public static void QueueUserWorkItem(WaitCallback callBack, object state)
    {
        WorkItem wi = new WorkItem();
        wi.Worker = callBack;
        wi.State = state;
        lock (_queue)
        {
            _queue.Enqueue(wi);
            Monitor.Pulse(_queue);
        }
    }
}

As you can see, it's very short and very simple. Basically, if it's not used, no threads are started, so in a sense it is lazy. When you use it for the first time, the static initializer will start up 25 threads, and put them all into waiting state (because the queue will initially be empty). When something needs to be done, it is added to the queue, and then it pulses a waiting thread to perform some work. And just to warn you, if the worker delegate throws an exception it will crash the pool....so if you want to avoid that you will need to wrap the wi.Worker(wi.State) with a try/catch. I guess at this point you may wonder why one should even bother writing a thread pool. For one, it's a great exercise and will probably strengthen your understanding of how to write multithreaded applications. The above thread pool is probably one of the simplest use-cases for Monitor.Pulse and Monitor.Wait, which are crucial for writing high-performance threaded applications. Another reason is that the .NET ThreadPool is optimized for many quick ending tasks. All asynchronous operations are done with the ThreadPool (think BeginInvoke, EndInvoke, BeginRead, EndRead, etc.). It is not well-suited for any operation that takes a long time to complete. MSDN recommends that you use a full-blown thread to do that. Unfortunately, there's a relatively big cost of creating threads, which is why we have thread pools in the first place! Hence, to solve this problem, we can write our own thread pool which contains some alive and waiting threads to performing longer operations, without clogging the .NET ThreadPool. In my next post I'll compare the above implementation's performance against the built-in ThreadPool.

SourceGear Vault, Part 4, Conclusion

I suppose I should give a disclaimer since I am *not* an expert with Vault, and my opinions may be completely due to my lack of understanding of the system. With that in mind, here's what my experience of using Vault has been so far. In general, it is not as fast as TortoiseSVN. There are many operations in SVN that are instant, where the comparable operation in Vault is met with 10 seconds of "beginning transaction" and "ending transaction", sometimes more. Feature-wise, Vault has much more to offer than Subversion does. Basically, it can do most of what Subversion can do, plus features from SourceSafe (like sharing, pinning, labeling, etc.). However, like I mentioned before, Vault has no offline support whatsoever, and you cannot generate patches, so it effectively cuts off any kind of outside collaboration. You could say that this is fine because SourceGear's target audience is small organizations where everyone will have access to the network anyway, but that doesn't mean that you won't be sent off to the middle of nowhere with no internet access and you need to fix bugs now! Not to say that Subversion is much better in that scenario, but at least you can still revert back to the last updated version.

SourceGear Vault, Part 3 (Performance vs Subversion)

I downloaded the kernel 2.6.29.1, and extracted it to the working folder. I figured this was an easy way to have a real-world scenario of changes (albeit a little big since it is all changes that happened between 29 and 29.1). Anywho, I was pretty surprised to find out that Vault does not have any feature whatsoever to allow collaboration between programmers other than via the client. You cannot create a .patch file and email it to someone. Everyone is assumed to have access to the server. This thwarted my plans to test the performance of diffing the changeset, because I simply planned on comparing how long it would take to create the patch. Ah well, I guess I'll just have to compare the performance of committing the changes between 29 and 29.1, which is a common use case as well so I don't mind. Time it took to a) start up the Vault client, b) bring up the Vault commit dialog, or c) bring up to TortoiseSVN commit dialog: Vault startup: ~1m02s Vault commit: ~7m55s Subversion commit: ~4m33s Time it took to commit: Vault: ~13m45s, and at 14m34s the HD stopped spinning Subversion: ~1m42s Hmmmmm, it doesn't look like Vault did too well in this comparison. Getting a status of all changed files took almost twice as long compared with Subversion, and what's worse, committing with Valut took 12 minutes longer than it did with Subversion. The extra minute with the hard drive spinning was attributed to SQL Server completing the transaction, which is why I separated that part, because as far as the client was concerned the operation was complete after 13m45s. One more quick test...branching. Subversion is famous for its O(1) time to branch anything, from the smallest to the biggest. SourceGear's page of Vault vs Subversion mentions that both offer cheap branches. Let's see that in action! I used TortoiseSVN's repo-browser and branched the kernel. It was practically instant with no wait time for the operation to finish. Vault, on the other hand, took a total of 47 seconds from the time it took me to commit the branch, to when the status said "ready" again.