IronPython and the elmcity project: Together again

In the first installment of this elmcity+azure series my plan was to build an Azure-based calendar aggregator using IronPython. That turned out not to be possible at the time, because IronPython couldn’t run at full strength in Azure’s medium-trust environment. So I switched to C#, and have spent the past few months working in that language.

It’s been a long while since I’ve worked intensively in a compiled and statically-typed language. But I love being contrarian. At a time when low ceremony languages are surging in popularity, I’m revisiting the realm of high ceremony. It’s been an enjoyable challenge, I’ve gotten good results, and it’s given me a chance to reflect in a more balanced way on the “ceremony vs. essence” dialogue.

Meanwhile, Azure has moved forward. It now provides a full-trust environment. That means you can run PHP, which is interesting to a lot of folks, but it also means you can run IronPython, which is interesting to me.

In this entry I’ll show you how I’m starting to integrate IronPython in the two main components of my Azure project: the web role that provides the (currently minimal) user interface, and the worker role that does calendar aggregation.

Using IronPython in an ASP.NET MVC Azure web role

The elmcity service writes a lot of log data to an Azure table. I’ll want curators to be able to query the slices of that log that pertain to the cities whose calendars they are curating. For Providence, RI, which uses the elmcity (and delicious) id mashablecity, the URLs for those queries might look something like this:

/services/mashablecity/log_info (log entries of type “info”)

/services/mashablecity/log_exception (log entries of type “exception”)

Here’s an URL route to carve out a namespace shaped like that:

routes.MapRoute(
 "services",
 "services/{id}/{what}",
 new { controller="LogServices", action="QueryLog", id="", what="" },
 );

Here’s a simplified version of the corresponding LogServicesController.cs:

[HandleError]
public class ServicesController : Controller
  {
  public ActionResult QueryLog(string id, string what)
    {
    return new ObjectResult(id, what);
    }
  }

public class ObjectResult: ActionResult
  {
  string id;
  string what;

  public ObjectResult( string id, string what)
    {
    this.id = id;
    this.what = what;
    }

  public override void ExecuteResult(ControllerContext context)
    {
    switch (this.what)
      {
      case "log_info":
         var script_url = make_script_url(this.id, this.what);
         var args = new List() { this.id, this.what };
         var result = new ContentResult
          {
          ContentType = "text/plain",
          Content = Utils.run_ironpython(script_url, args),
          ContentEncoding = UTF8
          };
        result.ExecuteResult(context);
        break;
      case  "log_exception":
      // etc
      }
    }

This fragment takes in the URL parameters, forms the URL that IronPython will use to fetch the script that it runs, packages the parameters into a list, calls a method to invoke IronPython, and dumps the script’s output into the outgoing HTTP response.

Here’s the code to invoke IronPython:

public static ScriptEngine python = Python.CreateEngine();

public static string run_ironpython(string script_url, List args)
  {
  var ipy_args = new IronPython.Runtime.List();
  foreach (var item in args)
    ipy_args.Add(item);
  var result = "";
  try
    {
    var s = Utils.FetchUrl(script_url).data_as_string; 
    var source = python.CreateScriptSourceFromString(s, 
      SourceCodeKind.Statements);
    var scope = python.CreateScope();
    var sys = python.GetSysModule();
    sys.SetVariable("argv", args);
    source.Execute(scope);
    result = scope.GetVariable("result").ToString();
    }
  catch (Exception e)
    {
    result = e.Message.ToString() + e.StackTrace.ToString();
    }
  return result;
  }

Whatever the script deposits in a Python variable called result winds up as the content of the HTTP response.

Using IronPython in an Azure worker role

Until recently I’ve been running some IronPython maintenance scripts from a standalone client machine. Now I’ve pushed them to the cloud. Here’s the scheduler that sets a timer to invoke a handler on a periodic basis:

public static void scheduler (ElapsedEventHandler handler, int minutes)
  {
  var timer = new Timer();
  timer.Elapsed += handler;
  timer.AutoReset = true;
  timer.Interval = 1000 * 60 * minutes;
  timer.Start();
  }

And here’s the handler:

public static void IronPythonHandler(object o, ElapsedEventArgs e)
  {
  try
    {
    var s = Utils.FetchUrl(Configurator.ADMIN_SCRIPT).data_as_string;
    var source = python.CreateScriptSourceFromString(s, 
       SourceCodeKind.Statements);
    var scope = python.CreateScope();
    source.Execute(scope);
    ts.write_log_message("info", "IronPythonHandler");
    }
  catch (Exception ex)
    {
    ts.write_log_message("exception", "IronPythonHandler", 
      ex.Message.ToString() + ex.StackTrace.ToString());
    }
  }

Best of both worlds

I’m still sorting out how I want to combine these two worlds, and I’m having a blast doing it. Could I have written the whole system in IronPython, had the option been available when I started? Undoubtedly. But high ceremony, coupled with a sophisticated tool like Visual Studio, has its charms. So does low ceremony and emacs. Using both together, and leveraging all their strengths, is really productive. And it’s loads of fun too.

6 Comments

  1. Jon, what do you mean by “IronPython cannot run at full strenght in ASP.Net medium trust environment”? IronPython runs well on Silverlight where the security settings are more strict than ASP.Net medium trust.

  2. I found (https://blog.jonudell.net/2008/11/25/ironpythonazure-status-report/) that I could not import pure Python modules in medium trust.

    Actually, even though I can do so now using full trust, I’m still looking for a solution that doesn’t require uploading the whole standard Python library every time I redeploy my Azure service.

    I’ve looked into customizing the import system (http://blog.dowski.com/2008/07/31/customizing-the-python-import-system/) and I think this kind of approach could become more interesting and more important in an Azure world.

  3. IronPython should be able import pure Python modules from folders the web app has read access to. However, by default, an ASP.NET medium trust app can only access the app folder (I think). So you either need to include the Python modules in your app folder, or you (or the web host) needs to change the security permissions to allow the web app access to the Python modules folder on the machine. This issue would apply to any resource your app depends on.

    If you run into more issues, do send details to users@lists.ironpython.com

  4. It’s been a while since I tried this, I’ll recheck. Of course at this point, I’ve been seduced into running with full trust for other reasons, notably binary serialization/deserialization. But we should clarify the original point.

    I’d love to know your thoughts on the custom import question. It may or may not be practical to do that. But Azure does prompt us to rethink our relationship to the filesystem, which I think is probably a good thing going forward.

  5. IronPython does support import hooks. IronClad (http://www.voidspace.org.uk/python/weblog/arch_d7_2008_05_31.shtml) uses it to bootstrap IronClad before loading native pyd extensions. Azure (which is just a specialized case of ASP.Net medium trust, right?) does have a file system and I think the solution is just to get the configuration and security settings right, but yes, in general, import hooks do give you a lot of flexibility.

    PS: Btw, I had written more details but forgot to add my name. Hitting Submit ends up throwing away all the typed content, and going Back does not keep the typed content. FYI FWIW :(

  6. > the solution is just to get the
    > configuration and security settings right

    That should work, but it would be painful to have to redeploy a great big Python lib every time you rev a tiny little Azure app.

    As Christian Wyglendowski notes in the import-over-the-web example cited above:

    “I can think of a number of features would be needed for a serious implementation of something like this (caching, HTTP-AUTH, signatures, remote package support, etc).”

    No such serious implementation seems to have been done yet, and I’m probably not qualified to do it, but I do find myself wishing for it.

    Funny how some languages (like JavaScript) have to be arm-twisted to talk to the file system, and others have to be arm-twisted not to.

    > Hitting Submit ends up throwing away all
    > the typed content, and going Back does not
    > keep the typed content.

    Ouch. Sorry for that.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s