Last month in an item about working with crime data I asked:
Will there be a role for IronPython (or IronRuby) here, someday, such that you could use these languages inside Excel? That’d be very cool.
Several folks suggested that I should take a look at Resolver, an IronPython-based spreadsheet that deeply unifies Pythonic object-oriented programming with the sort of direct manipulation that makes the spreadsheet so useful. Resolver was and still is in private beta, but today’s screencast (Flash, Silverlight) will give you a good sense of what it’s all about.
The presenters are Giles Thomas, managing director and CTO of Resolver Systems (and creator of his own Resolver screencast), and Michael Foord, who blogs about Python, contributes to the IronPython cookbook, and is also working on the forthcoming book IronPython in Action.
If you are (or would like to be) using Python to wrangle business data, Resolver will make sense immediately. You’ll love the idea of wielding Python’s powerful data manipulation features in that context. You’ll appreciate what it would mean to harness not only the Python standard libraries but, because Resolver is IronPython-based, also the .NET Framework and the universe of third-party .NET assemblies. And you’ll be intrigued by the way in which the IronPython code that represents and animates a Resolver spreadsheet can be reused elsewhere — for example, in web applications.
But there’s more to the story. Because a cell in a Resolver spreadsheet can contain a reference to any .NET object, Resolver creates, as Giles Thomas says, “a somewhat pathological but entirely new way of programming using a spreadsheet.” You can, for example, define an anonymous function — say, a function that returns the square of its argument — and store it in cell B4. Then you can place a value — say, 5 — in cell A2. Then you can store this formula in cell B6:
=B4(A2)
That says: “Apply the squaring function in B4 to the value in A2.” The result in B6 will be 25.
I’ve long argued that the interactive and exploratory style of dynamic object-oriented languages is an important but underappreciated benefit. As I may have mentioned before, IronPython’s creator Jim Hugunin told me that when he first showed IronPython to folks at Microsoft, he was surprised by their reaction. He thought the big wow would be IronPython’s ability to streamline and accelerate use of the .NET Framework. But while people did appreciate that, they were truly wowed by something that’s second nature to every Python programmer — the read/eval/print loop which traces all the way back to the earliest Lisp systems.
It is a magical and powerful thing to be able to explore and modify a running program’s code and data. From those early Lisp systems to today’s Python and Ruby implementations, we have been doing that exploration and modification using a command line.1 We can trick it out with recall, name completion, and search, but it’s still a command line with all the limitations that entails. If I’ve defined an object A and stored some code or data there, my definition and invocations of A will scroll out of view as I continue to work. They won’t be visually persistent.
In a Resolver spreadsheet, these objects are visually persistent. I haven’t yet got my hands on Resolver, but here’s an example of what I think that will mean. Suppose that I have a data set I want to transform, against which I’m testing five different versions of a transformation function. I’d put the data in cell A1, the functions in cells B1..B5, and the results in C1..C5. Now I’ll see everything at a glance. The spreadsheet that would conventionally have been the results viewer at the end of a series of tests becomes the environment in which the tests are written, performed, and evaluated.
The spreadsheet is also an important bridge between programmers and their business sponsors. It’s no accident that Ward Cunningham’s FIT (Framework for Integrated Test) was originally inspired by Ward’s experience of inviting business analysts to write test cases in spreadsheets. In its current form, FIT uses HTML tables in a wiki as the bridge between analysts who write tests and developers who write the code that must pass those tests. I think Resolver and FIT may prove to be a marriage made in heaven.
While Resolver will initially appeal to business programmers who appreciate Python as a language, and IronPython as a way of leveraging the .NET Framework and .NET-based business logic, the ideas it embodies transcend Python and .NET. I’ll be fascinated to see how this “pathological but entirely new way of programming using a spreadsheet” will evolve.
1 Smalltalkers will note that they have been using a three-pane browser all along, and that’s true. However the spreadsheet metaphor, in this context, is something else again.
Genius. Especially if, when you change the data or a function, the effects ripple through all the dependencies in realtime (like a spreadsheet) – that’s true functional reactive programming.
Is the grid pattern a good one here – I doubt it’ll usually fit naturally into tables? Perhaps something like the draggable icons on a desktop might work better, with arrows between them indicating dependencies.
Fuzzyman has done an outstanding job!
Btw, if you want to see a IronPython Smalltalk-like browser:
http://softwareindustrialization.com/AWebbasedIDEForDistributedProgrammingUsingIronPython.aspx
Yes Chris – as soon as you make changes in the grid (or in the code to objects referenced in the grid) the document is recalculated and cells that depend on those values / objects will be updated.
By the way, if you are interested in trying Resolver you can sign up for the beta program on the website.
Michael
Well there’s one previous case of a visual system that’s changeable at runtime…Smalltalk!
I was playing around with Squeak a while back, and from within the IDE I added a few fields to the definition of the Character object, the very object displaying my code in the IDE. And it worked…from the moment I redefined it, no restarting, every character had my extra attributes.
Someone showed me an extension to Microsoft Excel that added first-class functions a few months ago. I think that is an excellent idea and I’m surprised nobody has done it before. People do manage to write a lot of working code in dynamically typed functional languages like JavaScript so I’ve no doubt that this functionality would be used if it were available.
However, Lisp’s EVAL is a red herring here.
Thanks, John, this has given me a few ideas for using Resolver that I hadn’t picked up on before. You are right, I don’t think anyone is making more effective use of IronPython than the Resolver team – they were building GUIs with it before most people had run their first line of code!
I’m the author of a similar program, nip2, see http://www.vips.ecs.soton.ac.uk . It’s a spreadsheet-like interface to an image processing library. The programming language is an object-oriented Haskell (pure functional). Cells are objects and can contain images, matricies, functions, whatever.
Sadly, although this is great for a techie, the interface is so impenetrable that I’ve not had all the success I hoped for. Ah well.
Your note about Smalltalker’s using a three paned browser all along are true enough, but the idea of putting it into a spreadsheet belongs to Smalltalk as well. ASP, a Smalltalk-based spreadsheet, was remarkably similar to this spreadsheet. It didn’t have .NET or web access of course, because it was built by the PARC Smalltalk folks in the late 1970’s.
The system was built for the CIA, but I think there was a paper published about it in an early OOPSLA conference.
It’s nice to see this idea reappearing. I recall The Analyst from Xerox XSIS that had a spreadsheet whose cells contained arbitrary objects and code. It was written in Smalltalk. Not many references on the web (the early 80s were pre-web-history), but I found the following paragraph on a forum: http://www.groupsrv.com/computers/about103270.html
“The Analyst Spreadsheet (which was also sold as a separate package) was simply the best. Cells could contain arbitrary Smalltalk objects, and formula were arbitrary Smalltalk code. When we showed people things like image manipulation within spreadsheet cells or computing inverses of matrices containing fractions and/or complex numbers, they often could not believe what they saw :-)”
“ASP, a Smalltalk-based spreadsheet, was remarkably similar to this spreadsheet.”
“I recall The Analyst from Xerox XSIS that had a spreadsheet whose cells contained arbitrary objects and code.”
Fascinating.
One of the factors that’s in play now, and wasn’t then, is not just the web, obviously, but the emergence of social programming environments. It’ll be interesting to see how this story plays out, this time around, given that new opportunity.
AnalyRim (initial version written 1982, updated irregularly until about 1990 and published on DECUS sigtapes in source) supports programs or database access in each cell, separate mappings of display cells to underlying cells, lots of matrix math and other stat functions etc. etc. It was written in Fortran mostly (the “universal assembler” of its day) and versions for MSDOS, VMS, RSX, AmigaDos and SunOS were written. A zip of the old code can be grabbed off http://www.gce.com for the interested. This kind of thing is not all that unique, but there are available language processor cores now that were not ready to hand back in 1982. The notion of having programs in each cell seems like a natural kind of thing to do though. Among other things it can mean one computation produces several outputs. For example the max function in Analyrim returns not only the maximum value of a range, but the address within the range where it was found; so do several other selection type functions.
I used a shareware spreasheet in the early 90’s that allowed you to use named blocks for “user-defined formulas”. It’s was a life-saver in numerical analysis classes. I haven’t seen this functionality since then.
If you’re interested in something way more forward looking than this, that was done in the 80s, look at Dan Ingalls’ Fabrik project.
I think this is only half the story to fix the horror that spreadsheets are; the other half that needs to be fixed is to remove the tables and the empty cells (what’s up with all those empty cells?). I’m working on a project to do this (pipsqueak). I have the Python evaluator written out, it’s a single file of Python. This, combined with a table object similar to the one in Apple’s “Numbers” could change the way a lot of people prototype programs.
It’s a spreadsheet, it’s a GUI builder (and GUI), it’s a database interface, it’s a programming environment…
I meant to mention the name of the program…It was called ALITE from Trius.
“what’s up with all those empty cells?”
Unfortunately I think that any ‘spreadsheet-like-application’ that presents users with an empty document and no cells is not likely to go too far. If you don’t like the cells you can always turn the grid off.
I am intrigued by some of the things apple have done with numbers though – focussing on the document rather than the data. I think that as we evolve the document generation capabilities of Resolver this is something that will get attention (I have an interesting project for my Dad where we will be generating Word docs from Resolver for example).
Something that allows you to combine Resolver worksheets and cell ranges into a document (with a good UI which is most of the battle of course) sounds like a good idea.
Very cool. It tickled my memory… some time ago DARPA funded some
research into so-called “deductive spreadsheets”. Rather than using Python,
they used Prolog or some other logic programming language. See
http://news-service.stanford.edu/news/2007/april25/logic-042507.html
It’d seem like this idea would be perfect to wed with the concept of scripts and pipes used so much in Unix. Each cell should be able to define either a rule by which cells interact or a script in any language. Text gets piped in and out of each cell and can be used by any other cell as input. You could even design a way of entering the entire spreadsheet at the command-line or a script of it’s own and then spreadsheets could act as cells themselves.
An Open Office/Symphoy version that doesn’t tie into Dot.Net would be useful for those of us who run BSD/Linux/Solaris.
Fascinating stuff! Speaking of the Smalltalk and other pre-Web implementations of the same general idea, does anyone have any copies? further details? I’m a member of the rather amorphous TUHS – The Unix Heritage Society – and they sound just like the sort of thing TUHS was set up to cover. See them at tuhs.org.
Isn’t this sort of what Siag offers?
http://siag.nu/siag/
I wrote something very similar to this for Java: Bean Sheet.
About the comment:
“Smalltalkers will note that they have been using a three-pane browser all along, and that’s true. However the spreadsheet metaphor, in this context, is something else again.”
Well Smalltalkers will also say that the programmer’s spreadsheet idea is not new, take a look at this :) :
http://wiki.squeak.org/squeak/1227
Our industry is almost unique in its ability to forget institutional knowledge. A quick trip to the ACM’s digital library and a 1986 paper on not only an object oriented spreadsheet, but a *collaborative* one, pops up to the top of queries of object oriented spreadsheets. Yet an entire project was implemented without any reference to our history… like 99.9% of projects implemented are built without any reference to history.
The design of an object-oriented collaborative spreadsheet with version control and history management:
http://portal.acm.org/citation.cfm?id=162950&jmp=cit&coll=ACM&dl=ACM&CFID=37158309&CFTOKEN=35992933#CIT
Object-oriented spreadsheets: the analytic spreadsheet package:
http://portal.acm.org/citation.cfm?id=28737&jmp=cit&coll=ACM&dl=ACM&CFID=37158309&CFTOKEN=35992933#CIT
Yes, this is 20 years ago. The good news, that means any patents for these technologies have inspired, so your new creation should be safe.
Grrrr. That would be *expired* not *inspired*.
Sorry for the rant, but it is just so disappointing that I see this about once a month: an announcement of something so new that it couldn’t possibly have been done before, yet when I ask if they have done a literature search they look at me like I am speaking in an alien language. Organizations like the ACM and IEEE have a vast troves of information and knowledge, yet membership continues to decline in the traditional professional societies in favor of vendor specific groups (that lack an *interest* in developing institutional knowledge because that doesn’t sell new products).
Much effort is lost duplicating the past.
“Much effort is lost duplicating the past.”
Agreed. As we go forward, we can hope and expect that what is known will be more fully and easily discoverable than it once was.
Case in point, regarding the ACM article you cite:
“Much effort is lost duplicating the past.”
John, you make a good point that if we’d set out to build an object-oriented spreadsheet, we’d have found a lot of prior art. You’re quite right, and a Google search shows up quite a lot of other products in that space (albeit not the ACM papers, as Jon points out in his comment above). The thing is, we set out to produce something quite different, and the OO/functional nature just dropped out – it simply happened to be easier for us to write Resolver in that manner than in any other, because of the constraints we had set for ourselves in our earlier coding.
The most confusing, and most fun, thing about doing a software startup is that the idea you originally have – in our case, converting spreadsheets to Python code and back again in realtime – never turns out to be the thing that ultimately piques people’s interest.
“The most confusing, and most fun, thing about doing a software startup is that the idea you originally have – in our case, converting spreadsheets to Python code and back again in realtime – never turns out to be the thing that ultimately piques people’s interest.”
Great point. Flickr, for example, began its life as an online game. I guess it still is one, but of a very different kind :-)
Resolver does look very interesting. i am not a coder, and feel myself firmly business side (but one who camps out with IT). i have written a moderate bit of VBA over the years (full in memory calc with arrays, stochastic algorithms etc), but have never seen a line of python. that as background, i’d like to emphasise that to me it appears very important to understand what this tool is and how it can be used. in addition to the technical merit, i see a lot of ‘business side’ potential, but that will be killed off if the devs and community only focus on the wonders of some wonderful yet forgotten OO approaches. business purpose is the key here – get this as a bridge to the business power users and you really kick Excel. I love excel, i really do, but only because its the only thing on my desktop IT will let me do my job with… Resolver could make a new world in some sectors. i see a lot of banks being interested, and as this was born from their thats hardly surprising. get this right with business users, and python, OSI, and the rest could really get a toe hold on the corporate desktop. That is really worth fighting for!
“business purpose is the key here”
Agreed. That’s why I make the connection to FIT, and hope that others will.
Yes, thanks guys for taking care of this. Much appreciated :)
An Open Symphoy version that doesn’t tie into Dot. Net would be useful for those of us who run Linux and Solaris.
I usually get bored after a while playing only 1 table, but thats me, dont know if others doing better or worse with multi-tabling