08 5 / 2014

On Purpose

Every day we humans wake up and do something. Those of us in poverty or at war do only one thing: try to survive until tomorrow. The rest of us work, learn, and play. For us, tomorrow is not a goal, it is a gift.

Even though tomorrow is a gift, we manage to keep extremely busy. We want to be busier but we can’t, because there isn’t enough time, which is something we frequently discuss.

We’re so busy, that looking at us, you might be curious to know what goal we’ve set for ourselves that’s so grand to so consume us. Our long lives assured, with what axiom does our desire to do anything begin?

I would be sorry to tell you, because it’s extremely weird, that we don’t actually know. The idea of existing without needing to put effort into survival is new to us as a species. Having the time to think about such things is almost as new.

Our instincts and intellect, sharpened so long by the stone of survival, idle uncomfortably. Our ancestors, only acting on the impulse to provide and secure for us, accidentally banished us to an existential purgatory.

So, we vigorously pace our holding cell and focus our energies on distractions. This is the activity that you might so easily mistake for movement with a higher purpose.

Fortunately, our holding cell – the Universe – is infinitely detailed, and we’ve had success managing our discomfort by occupying ourselves with its inspection from various angles and at various magnifications. Climbing Mount Everest is one such exercise.

Looking for patterns and relationships in the Universe, and explaining one’s findings to other humans, is another satisfying diversion. We kindly convey our discoveries through story, song, and Seinfeld, such that those of us less adept at pattern-finding may share in the diversion.

We less kindly codify distractions we’ve found into laws, governments, and other authorities. These are efforts to enforce distraction and criminalize acknowledgement of our scary existential limbo.

Like our ancestors, we act on impulse. Except, instead of survival, from which our activity was once derived, our impulse is to act for its own sake. We glorify, record, and amplify activity – pointless as we’re terrified it might be – and expand our technological and cognitive ability to separate ourselves from that Universe which so painfully reminds us what a paradox we have become.

27 7 / 2013

Leadership

I ran across a book the other day called The Senior Software Engineer. I haven’t read it yet, but it caused me to reflect on what my own definition of “Senior Engineer” is.

Obviously, a senior engineer has technology skills and experience. So do many “Junior Engineers”, though. So, what’s the difference? I think it’s leadership ability. To me, leadership is that skill that allows an individual to thrive when handed additional responsibility.

Leader = Manager = Asshole?

I once thought that leadership implies subordinates. It can, and often does, but it doesn’t always. The skill of leadership can be applied to dealing with other kinds of responsibility, like the outcome of a particular project or the resolution of a particular problem.

Unfortunately, our lives and world are filled with terrible leaders. This is because assholes seek power over their peers for the sake of it, and leadership roles promise exactly that. Instead of using power to achieve common objectives, asshole leaders abuse power to further their own. Leadership can be a difficult skill to learn not because it’s especially hard, but because so much of what’s said about it comes from these assholes.

The proliferation of assholes in leadership positions has made “leadership”, and particularly “management”, dirty words in software development. The movie Office Space parodies the kind of work environment bad leaders cultivate and personally excel in.

This is unfortunate, because if I’ve learned anything about the software profession, it’s that successful efforts require teamwork, and that successful teamwork requires skilled leadership. The skill of leadership is no less important in organizations with “flat” management structures: in such environments, everyone should be a leader.

Leadership: The Skill

So then, what is this skill, and how can one learn it? I don’t really know. I don’t claim to be a good leader, or a leader at all. All I know is that I think I’ve seen good leaders up close, and that I don’t think I’m an asshole.

Leadership (n): the skill of efficiently applying one’s abilities to one’s responsibilities in order to achieve some common goal.


That’s the definition I’d come up with if pressed. I think anyone can learn this skill. I don’t think a leader needs to be a man, a woman, or even an adult. Leaders needn’t be social butterflies nor square-jawed silent-types. I’ve seen great leaders of every age, gender, and disposition.

Since 1775, the US Army has done a lot of thinking on how to teach the skill of leadership. As an Army veteran and former NCO, I’ve found the Army’s conceptual framework practical and applicable to professional settings.

If you are interested in cultivating your own leadership ability, or that of your team, the Army’s field manual on leadership - FM 6-22 - is not a bad resource to start with.

FM 6-22 is filled with jargon and ideas specific to the military, but taken as a whole it represents a literally battle-tested framework around the idea of leadership that I’ve found widely applicable.

Of particular interest to software professionals and managers may be the following ideas from FM 6-22:

  • Counseling:Professional development is a core focus of the military. The military does everything it can to continuously train and retain its personnel. In my career as a programmer, I’ve found the general idea of professional development is poorly served in many organizations. Counseling, and counseling sessions, are part the Army’s process for keeping leaders aware of their soldiers’ individual situations, and for soldiers to receive continual performance feedback. Counseling, and the paper trail a series of counseling sessions establish, is also part of the promotion process. Appendix B of FM 6-22 covers the kinds of counseling and how they are implemented.
  • Developing:In my experience, the best leaders groom every subordinate as if that subordinate is to be their replacement. To do this, leaders must exercise some authority to create a positive, nurturing work environment. If a leader establishes a positive environment, the effects on the team are manyfold, including at least: a shared sense of purpose, a willingness among team members to share with and assist one another, and generally a higher volume and quality of intra-team communication. Chapter 8 of FM 6-22, “Developing”, contains much practical information about how to create and participate in a positive work environment.

Me, personally.

My personal perspective on leadership is influenced less by a particular doctrine or framework and more by how I have seen individual leaders operate across my military and software careers. Like any doctrine, FM 6-22 is ripe for abuse, and the worst leaders I’ve seen are those who cleverly use such doctrines to further personal objectives - the careerists and the assholes.

As a result, I know it’s possible to be a good by-the-book leader - and to be commended in an organization for “leadership” - and also be a shitbag. So, I see leadership more generally as a combination of skill and positive intent. This aspect of the idea of leadership is what the Army tries to capture with its “Be” component of the BE-KNOW-DO concept as forwarded in FM 6-22, Chapter 1.

For the good-hearted and aspiring leader, I humbly offer the following assortment of ideas and anecdotes from my own experience. Good luck!

  • Lead by example: You’ve heard this before; hear it again. I think this is the best general-purpose advice for anyone who finds themselves in a leadership role. When you lead by example, you demonstrate commitment to the common goal, which earns you respect. You can then use the respect you’ve earned to influence those around you to do things like build a positive work environment. However, like the golden rule, this approach can be tricky. Your time may not always be most effectively spent setting examples. For instance, you may hold a title like “architect” or “chief scientist”, and the nature of your work may be inherently different than that of those around you, precluding your work as exemplary. There might be different kinds of example you can set, such as: punctuality, work ethic, humility or level of communication.
  • Take care of people: This is also widely-parroted but very important. It’s rare for any kind of valuable or profitable software to be created by an individual. Most of the time, software is created by teams. The functionality of this team is critical to the functionality of the software. Thus, people are more important than programs. Good leaders don’t need to be “people persons”; they just need to continuously assess and consider the personal situations of those around them, and act appropriately and accordingly. The right way to take care of a person professionally is situational, but generally: give people responsibilities and individual goals. Provide meaning and purpose to people by trusting them, helping them, and giving them responsibility. Ensure on a regular basis that their basic needs to work effectively are met. Regular counseling sessions are one way to achieve this.
  • sua sponte: This is the motto of the 75th Ranger Regiment, and it means roughly “of your own accord”. My drill sergeant in basic training, a former Ranger School instructor, taught it to me, and I’ve never forgotten it. The gist of the motto is: act without prompting when you think it’s right to. Leaders have to do this because acting without specific instruction is often one of the responsibilities they are given. It’s even better for leaders to encourage those around and below them to do the same, because doing so and being accountable is how one may practice leadership. Train everyone around you to be you. Then, empower them to act as you. In the small teams I’ve often found myself on as a software developer, this practice has contributed greatly to collective productivity and happiness.
  • Be Prepared:This one comes from the Boy Scouts, but I came across it in the Army as “get your shit ready the night before”. Leaders think ahead on multiple dimensions because they know that whatever the future holds, they are responsible for the outcome. Generally, it behooves one to continuously consider the future and how to be better prepared for it. If you are in charge of a team, it behooves you to think ahead on behalf of them so that they are free to focus on the tasks at hand.

Oh, you’re still reading? For more on leadership from someone who knows way more about it than I do, check out this interview with General Martin Dempsey, the 18th and current Chairman of the Joint Chiefs.

13 7 / 2013

Faster Clojure Startup with Class Data Sharing

Java 1.5 introduced Class Data Sharing (CDS), a mechanism for quickly loading classes from an image of the classpath at JVM boot time. This is what the JVM uses to load its own runtime. It’s possible to generate your own image and instruct the JVM to boot with it, improving your own application’s startup time.

On my machine, I’m now able to start a Clojure REPL and run Clojure scripts almost twice as quickly as before. The following instructions worked for me on Ubuntu 12.10 with the openjdk-7-jdk package, but they can probably be modified to work on different systems or in concert with Leiningen/Maven.

Instructions

First, grab a recent clojure.jar.

It’s not a bad idea to then benchmark your current setup:

alan@cheezit:~$ time java -cp clojure.jar clojure.main -e '(System/exit 0)'

real    0m0.835s
user	0m1.088s
sys	0m0.064s

Now, create and install a new shared class archive file for your JVM that appends clojure.jar to the native runtime. This requires superuser:

alan@cheezit:~$ sudo java -Xshare:dump -Xbootclasspath/a:clojure.jar
[sudo] password for alan: 
Loading classes to share ... done. 
Rewriting and unlinking classes ... done. 
Calculating hash values for String objects .. done.
...

Finally, run a JVM with instructions to use the shared archive that bundles clojure.jar:

alan@cheezit:~$ time java -Xshare:on -Xbootclasspath/a:clojure.jar clojure.main -e '(System/exit 0)'

real    0m0.437s
user	0m0.564s
sys	0m0.040s

Permalink 3 notes

08 4 / 2013

Structural Editing Apocrypha

Here are some things from the Internet to whet your structural editing appetite.

The Interlisp Experience

The 1984 book Interactive Programming Environments, a collection of surveys of programming environments of the time, features this piece by Erik Sandewall detailing the development experience in an Interlisp environment.

UNIX and MIT Lisp Machines are also featured and discussed by their respective creators, but IMO pale in comparison.

Intentional Programming

Charles Simonyi has a vision for computing he calls “Intentional Programming” that’s pretty awesome.  Check out this database-backed structural editor for an Algol-like language.

Other Advancements

  • PEDIT was a structural editor for Prolog (which is homoiconic, btw)
  • Nokolisp was a Lisp environment written for minicomputers in the late 70s that features an impressive structural editor.

Are you aware of any structural program editing software or tools I don’t list here?

08 4 / 2013

Structural Editing Revisited

It’s been a little over three years since I first stumbled on the idea of a structural Lisp editor, and over a year since I last blogged about the idea.

Well, I have good news: Micha Niskin and I have overcome the technical challenges that I was unable to solo, and work toward a realtime structural Clojure editor suitable for our own professional use is underway.  We call it “stoke”, and I posted a short demo of our prototype on YouTube:

The editor consists of these main pieces:

  • reader: For code reading purposes, Clojure’s native reader does too much.  For instance, scalars like “+1.0e3” are read as a Double and later print as “100.0”.  Unordered collection types like sets and maps behave similarly, and the result is a mismatch between read and printed representations.  Our reader deals in only two types, symbols and sequences, and preserves enough information (using metadata) to print appropriately.
  • printer: We were able to extend Brandon Bloom's excellent Fast Idiomatic Pretty Printer to print Clojure code idiomatically.  FIPP’s intermediate representation, the “document”, gives us a data representation of the formatted code that doesn’t necessarily need to be printed character-wise, and opens the door to interesting alternative views of code.
  • meta-zip: This is less a piece and more an abstraction that we have found crucial, and zippers will be the editor’s primary extension point.  Zipper navigation and manipulation of our symbol/sequence representation is trivial.  Meta-zip is a “zipper of zippers” that lets us cleanly model tree undo/redo.

Our immediate goal was to have a terminal-based structural editor with paredit commands, REPL, and file interaction, and as of this writing we are just about done with it.  Next up are a Swing frontend, network/multi-user editing, API cleanup, and possibly a public release.

However, if we stopped here we’d fail to take advantage of one of the greatest things about having our own structural editor: files are no longer essential.  What does this mean?  I think the implications are vast, incredible, and since the death of Interlisp have been largely unrealized.

We’re used to working in and around files and filesystems, and much of our programming tooling is file-oriented - package systems, revision control systems, editors, versioning idioms.  I think that collectively these systems represent an attempt to turn filesystems into the databases we wished they were, and I think that by storing Clojure code in Datomic we can throw most of them out.

I want to live in a world where code is stored in a temporal database and I build new programs with the combination of query and structural edit, and this is where we ultimately want to take stoke.  You can follow our progress at http://tailrecursion.com.

image

*update 4/8/13*

Beyond a code database, other implications of the symbols+sequences intermediate representation (and zipper navigation model) include:

*update 4/11/13*

Posted Clojure Structural Editor, Take 2 to YouTube

26 2 / 2013

A bad year, a good year. A little retrospective.

This past year has been simultaneously my most difficult and my most exciting, and I include previous years spent deployed to Iraq in that calculation.

The end of my marriage was the event that sent me into a tail spin just over a year ago.  Having now experienced both, I can testify that what they say is true: the end of a marriage is no less painful than the death of a loved one, and its effects at least as lasting and treacherous.

Fortunately the world is not complete shit, all the time.  With the help - for which I am forever grateful - of my family and friends, I kept on professionally, and completed an amazing two-year tenure with Relevance to join the also-incredible crew at LonoCloud.  My affiliation with Splat Space and its outreach initiatives continued to be a source of stability, inspiration, and happiness.

Somewhere in the middle I rekindled a connection with the physical and in November completed a GORUCK Challenge.  I encourage anyone with the means and a need to prove something to themselves to do the same.

I now look to the future with renewed and expanded senses.  Of how horrifying and dark life can be.  Of how important and meaningful family and friendship are.  Of how lucky I am.  Of my own ability to dropkick adversity in the face and kick it while its down.

I begin this next and enlightened chapter at The Fresh Diet, a company with an ambitious plan to revolutionize their business using Clojure, ClojureScript, and Datomic.  I invite you to join me and The Fresh Diet team at tailrecursion.com, where we hope to catalog our open source contributions as we go.  We have big plans, and it’s going to be an amazing year.

image

Permalink 1 note

08 3 / 2012

A Sigh of Datomic

Over a year ago I had the great pleasure of working with the Datomic team through my wonderful employer, Relevance, while the project was still secret.  Datomic, which was released on Monday, is a new kind of database with a unique and powerful set of properties.

I believe Datomic makes new kinds of applications possible and old kinds of applications easier, and I encourage you to try it out.

My experience working on and around Datomic was incredibly formative technically and professionally.  The only downside has been the agony of waiting to talk with people about it and to apply it to public projects and products.  I’m excited and relieved that Datomic is now in the open.

As a technical influence, Datomic solidified a perspective on computing that I was first exposed to through Clojure, also designed by Rich Hickey.  I have found this perspective - which might be loosely characterized as emphasizing simplicity, immutability, and the importance of data - a powerful tool by itself, irrespective of either Clojure or Datomic.  It is this perspective that Rich elaborates on in his Strange Loop 2011 keynote, Simple Made Easy.

Simplicity, immutability, and data are powerful ideas to think about software in terms of, whether you are creating something new or assessing something that already exists.

As a software consultant at Relevance, I divide my time between creating solutions and choosing existing solutions in order to solve problems.  Combined with deliberate thought, research, and discussion with colleagues and clients, these ideas have helped me do both things better, to the extent that it sometimes feels like cheating.

Professionally, my involvement with the Datomic team taught me many things, the most influential of which was perhaps the importance of questions to problem solving.

I work in a totally intangible world of software, trade-offs, conceptions, and domains that has technical, social, and financial dimensions.  This world disappears when I try to imagine it all at once, which is my first instinct when I try to solve a problem within it.

Specific questions and their answers are a tool for considering this intangible world one piece at a time in the context of a particular problem.  If you ask and answer enough of them for yourself, you can build a practical mental model of the intangible.  This model allows you to make sensible decisions and form grounded opinions.  Questions are a big piece of the problem solving approach Rich discusses in his "Hammock-driven development" keynote at Clojure Conj 2010.

It’s not always easy to ask questions, especially if you know it will be hard to find the answer, or if the answer might be unfavorable.  But, unless you value your ego over your ability to solve problems, you should never hesitate.

Rich Hickey, Stu Halloway, Justin Gehtland, and the rest of the Datomic team did not hesitate, and now the world has a new database of singular vision and capability.

Permalink 1 note

10 2 / 2012

Structural Editing Redux

I gave a quick lightning talk at the most recent TriClojure about structure editing.  Structural editing is an idea I can’t seem to shake, and my efforts to wrap my head around its implications continue.

After the meeting, I followed up with a few interested parties about my discoveries, and thought to post them here for the benefit of the wider Internets:

The blog post that originally got me thinking caused me to write my own blog post, and a little while later I published a gist of a simple structural editor for Clojure using zippers.

The code in the gist has a lot of bugs.  In particular, the zipper function I’m using doesn’t handle vectors or hash maps gracefully.  I’ve thought about tightening the REPL loop so that every key press instantly triggers zipper movement.

The whole structure editor thing has also turned me on to the idea of touch/tablet IDE for Lisps - form bubbles that can be nested, similar to the Scratch project (but data).  I think this might ultimately be a better way to represent structure and sequence than characters and parentheses.

Once I found out about Interlisp (pdf), I went nuts, but information and screenshots are hard to come by.  Rich Hickey told me once that there is a video floating around somewhere of the Interlisp-D UI in action, but I’ve never been able to find it.

I’ve investigated what it would take for a real-time structure editor, and I think it’s a very hard problem.  Synchronizing data structures with their pretty-printed representation on a character display is no small feat.  Bernard Greenberg (pictured below) documents some approaches in his history of the implementation of Emacs.

More recently, Michael Fogus pointed me at Nokolisp, an old Lisp for DOS that features an interesting-looking structure editor, among other features (like “uncompiling” functions!)

My current thinking on the implications of a structural editor goes beyond convenience of representation and into the idea of programs constructed by query.  I imagine a world where code is stored and shared structurally instead of with text.  Programming with available libraries then becomes querying instead of explicitly programming.

As the programmer types, their editor might search these libraries in real-time and do things like suggest improvements using probability heuristics, or suggest substitution of their code with available library code that is a structural match.  This is an extension into a networked world of some of the ideas built into Interlisp.

Code as truly data also abolishes the need for git or other text-based source revision tools, as source would be stored at the granularity of expression - not the character or line.

Bernard Greenberg teaching Lisp

03 7 / 2011

Discover Lisp in Your Web Browser with Javascript

This post is available as a PDF

Many ideas introduced originally in Lisp, such as conditional execution (the if-then-else construct), garbage collection, and first-class functions1, can be found in modern non-Lisp programming languages.

Other ideas pioneered by Lisp, like the idea that programs can be composed solely of expressions, and homoiconicity, or the idea that a program can be written in its own data structures, are rarer.

Learning a Lisp dialect, whether it’s Clojure, Scheme, or Common Lisp, is a great way to see for yourself how powerful all of these ideas can be when combined.

An even better way to understand the ideas is to implement your own Lisp23.

After we explore some fundamental concepts, we’ll look at how a language like Javascript works.

Then, we’ll build our own small Lisp to Javascript compiler on top of Javascript in about 32 lines of Javascript code.

Expression vs. Statement

Programs in languages like Javascript are composed of two things:

  • statements
  • expressions

Lisp programs are composed only of expressions.

Both statements and expressions are, in some sense, instructions for the computer to do some piece of work. The difference between them is that a statement does not return a value, but an expression does.

Consider Javascript’s if-then-else statement:

if (x > 10) {
  alert("x was bigger than 10!");
} else {
  alert("x was not bigger than 10!")
}

We call alert inside both the then and else bodies of the expression because if does not return a value. If it did, we could pass it directly to a single alert call, as this wishful pseudo-Javascript demonstrates:

alert(if (x > 10) {
  return "x was bigger than 10!"
} else {
  return "x was not bigger than 10!"
});

If we eliminate statements from our wishful pseudo-Javascript entirely, there’s no longer a need for the return operator. It’s assumed that values “return” themselves:

alert(if (x > 10) {
   "x was bigger than 10!"
} else {
   "x was not bigger than 10!"
});

Javascript actually provides an if-then-else construct that returns a value, the ternary operator4, but it is seldom used and often discouraged5.

With expressions, it’s possible to write code in the imperative style the first example demonstrated. The opposite, writing expressions in terms of statements, is not.

If we agree that “power” means “possibility”, then languages like Lisp, which are based on expressions instead of statements, might be considered powerful.

Homoiconicity

Homoiconicity is a property of a programming language’s syntax, and means that programs in a language are expressed using that language’s own data structures. The property of homoiconicity is key to the way Lisp empowers the programmer to fabricate and manipulate syntax.

Homoiconicity might be understood by exploring the properties of source code and interpretation of that source code in a non-homoiconic language first.

Javascript, while influenced by Lisp in other ways6, is not homoiconic, and its popularity, availability, and conventional syntax make it a candidate for exploration.

Implementing Javascript

How a Javascript Interpreter Works

Consider this fragment of Javascript code, which passes the sum of 1 and the product of 2 and 3 to a web browser’s alert function:

alert(1 + 2 * 3);

When a browser passes this code, or data, as a string to its Javascript interpreter, something like the following happens:

  • The interpreter lexically analyzes7 the string, turning it into a stream of tokens. Tokens are objects in source code upon which other constructs are defined, such as function names, operators, and literals like numbers or strings. The tokens in the above fragment might be:
    • alert
    • (
    • 1
    • +
    • 2
    • *
    • 3
    • )
    • ;
  • The interpreter parses the token stream, attempting, recursively, to match the pattern of tokens to known language constructs. For instance, when the interpreter sees the function name alert followed by (, it knows to expect zero or more tokens that must be succeeded by a closing ).
  • The interpreter evaluates the constructs it found from the inside out:
    • 2 * 3 happens first, because, * has higher precedence than + and because the interpreter knows it needs a value or identifier to pass to alert.
    • The product, 6, is returned.
    • 1 + 6 is evaluated, returning 7.
    • Finally, alert(7) is evaluated, and a message is displayed.

If you were to write your own Javascript interpreter or compiler, you would need to implement a lexical analyzer, a parser, and an evaluator in order to complete the above steps.

Even if you were to implement your interpreter in Javascript, you’d have a long way to go, because none of these functions are provided by Javascript interpreters to Javascript programmers.

Javascript provides the eval function, but it accepts only strings, and you need to create source code strings yourself. Creating source code strings to pass to eval is effectively compilation, and done correctly could be as difficult as writing a compiler.

The task is not as daunting if we take the Lisp approach and build our programs in a data format richer than strings. Let’s meet the problem half way, and invent a Javascript syntax based on Javascript data.

A Homoiconic Javascript Subset: JSONScript

S-expressions

Consider this alternative representation of the code fragment, in a language we just invented, JSONScript:

['alert', ['+', 1, ['*', 2, 3]]]

Unlike the earlier example, which was a string that required multiple phases of analysis to understand and run, the above code is written using data structures native to Javascript: string, number, and array.

JSON, which stands for “Javascript Object Notation”, is a notation for expressing data in terms of Javascript data structures. JSON is expressive enough for us to use it to represent code for a simple programming language.

We could easily write a function called compile_jsonjs that converted this representation into a string that could be passed to eval, because:

  • We don’t need to write a lexical analyzer or parser: our code is already rich enough to express how it should be evaluated.
  • compile_jsonjs doesn’t need to know about operator precedence because we’re using s-expressions.

S-expressions, short for “symbolic expressions”, are a convention for writing code that allows us to represent a function call with an array. This convention is also known as prefix notation. The first element of the array is the function, and subsequent elements are arguments to that function that are evaluated first.

S-expressions leave no ambiguity about the order in which functions must be applied. In the original example, the expression 1 + 2 * 3 evaluated to 7 because it’s in the Javascript specificaton that * must evaluate before +.

In JSONScript, we express our intent to multiply first by making ['*', 2, 3] an argument to +.

S-expressions don’t have to be arrays. 42 and "hamster dance" are also s-expressions, because they also evaluate, but to themselves.

Lisp uses lists instead of arrays as s-expressions, but the idea is the same.

Some Implementation Details

There are at least two implementation details we face in implementing compile_jsonjs:

  • In JSONScript, as in Lisp, there is no distinction between function and operator. Functions, whether they are alert or *, must only appear as a string and as the first element of an array.
    • compile_jsonjs needs to recognize functions that Javascript considers to be operators, and return the correct syntax for them.
  • In Javascript, functions are identified by symbols, not strings. For instance, this syntax is not correct: 'alert'(7).
    • compile_jsonjs must return function names as symbols.
    • Conflating symbol and string is necessary but has unfortunate implications. For instance, it will be impossible to pass a function name as an argument to a function in JSONScript.

    • Adding the ability to pass function names as arguments equires us to go beyond JSON as our source data format, because JSON permits symbol literals only as an object field names.

Building the Compiler

We’re now equipped to reason about how compile_jsonjs will work.

  • First, we check if the input is an array.
  • If it is an array, we:
    • Compile the arguments
    • Check if the first element is an operator like * or +
      • If it is an operator, interpose the compiled arguments with the operator, and concatenate everything into a single string. compile_jsonjs(['+', 1, 2]) should return "1+2".
      • If it’s not an operator, like alert, we return a function call. compile_jsonjs(['alert', 'hello']) should return "alert('hello')"
  • If it isn’t an array, we return it as a string. compile_jsonjs(42) should return "42".

The Implementation

Next, we can translate our plan into code. We know that compile_jsonjs will need helper functions for:

  • Testing if an object is an array: This is how we’ll determine at the beginning of compile_jsonjs whether we’re compiling a function call or a literal. Literals are data that evaluate to themselves, like numbers or strings.
  • Testing if an array contains a particular object: We’ll need to make a list of functions that are Javascript operators, because we must emit different Javascript syntax for them.

The first helper function we need, isArray(), isn’t available on some browsers, and is actually difficult to write in a widely compatible way.

So, we can borrow it from the underscore.js8 library:

var isArray = function (obj) {
  return Object.prototype.toString.call(obj) === '[object Array]';
}

Next, we need a function for testing if an object is in an array. Such a function is also not available in some browsers, but we can implement it like so:

var inArray = function(array, item) {
  for(var i = 0; i < array.length; i++)
    if (array[i] === item) return true;
  return false;
}

With these functions in place, we can finally write our compiler:

var compile_jsonjs = function(expression) {

  var operators = ['*', '+'];

  if (isArray(expression)) {

    var func_name = expression[0];
    var func_args = expression.slice(1);

    var compiled_args = [];
    for (var i = 0; i < func_args.length; i++)
      compiled_args.push(compile_jsonjs(func_args[i]));

    if(inArray(operators, func_name)) {
      return compiled_args.join(func_name);
    } else {
      return func_name + "(" + compiled_args.join(',') + ")";
    }

  } else {
    return expression.toString()
  }
}

The full source listing, which you can copy and paste into a web browser’s Javascript console, follows:

var isArray = function (obj) {
  return Object.prototype.toString.call(obj) === '[object Array]';
}

var inArray = function(array, item) {
  for(var i = 0; i < array.length; i++)
    if (array[i] === item) return true;
  return false;
}

var compile_jsonjs = function(expression) {

  var operators = ['*', '+'];

  if (isArray(expression)) {

    var func_name = expression[0];
    var func_args = expression.slice(1);

    var compiled_args = [];
    for (var i = 0; i < func_args.length; i++)
      compiled_args.push(compile_jsonjs(func_args[i]));

    if(inArray(operators, func_name)) {
      return compiled_args.join(func_name);
    } else {
      return func_name + "(" + compiled_args.join(',') + ")";
    }

  } else {
    return expression.toString()
  }
}

Trying it Out

We can test our compiler by seeing what code it generates with our JSONScript fragment:

compile_jsonjs(['alert', ['+', 1, ['*', 2, 3]]]);

And we can evaluate our compiled code by passing that result to eval:

eval(compile_jsonjs(['alert', ['+', 1, ['*', 2, 3]]]));

Going Further

JSONScript probably isn’t a language you would want to use for your next programming project, but hopefully you now have a better understanding of what Lisp is and why it is different.

Further extending JSONScript will help you understand Lisp even more.

You can begin by adding support for other Javascript operators.

Next, you might implement lambda and quote. These will allow you to define functions in JSONScript, and to pass arrays to functions without evaluating them.

Once you have lambda and quote, you can implement other special forms, using Paul Graham’s The Roots of Lisp as a guide9.

If you’re feeling particularly ambitious, you might consider adding support for one of Lisp’s most famous and unique features - macros, or functions that:

  • are run before evaluation
  • do not evaluate their arguments
  • return JSONScript code

  1. What Made Lisp Different by Paul Graham

  2. The Roots of Lisp by Paul Graham

  3. Lisp in Small Pieces

  4. Wikipedia: Ternary operation (Javascript)

  5. jQuery Core Style Guide: Blocks

  6. Popularity by Brendan Eich

  7. Wikipedia: Lexical analysis

  8. _.isArray in underscore.js

  9. The Roots of Lisp by Paul Graham

Permalink 9 notes

17 11 / 2010

An XML Rant

This rant is brought to you by my reaction to Daniel Lemire’s post, "You probably misunderstand XML"

In school I took a class called “Data Interchange” or something, that taught primarily XML and the ecosystem around it.  We learned about XSD, RELAX NG, XSLT, XPath, JAXB, SOAP, and lots of other things I thought to be loads of horrendous bullshit at the time.  I hated that class.

I made a special effort to go, though.  I was on a personal mission to point out every counterexample to, failing of, and argument against XML that I could.  ”It’s the illegitimate half-brother of the holiest representation of all time, the S-expression, and it’s slower than shit to boot!” I cried.

I’ve learned a lot since then.  The biggest thing I’ve learned is that it’s easy to dismiss any technology with cultural baggage on religious terms.  It’s also easy to conflate your hatred of some technology, like SOAP, with some other technology, like XML, that is only loosely or incidentally related.  Without assessing pragmatically to yourself whether SOAP sucks because it uses XML, or SOAP just plain sucks, you have not adequately justified your hatred.

The more you work without asking yourself these questions, the more likely you are to commit the cardinal sin of anyone in the computing profession who gives a shit about what they do - use the wrong tool for the job.

What’s much harder, and much more useful, is to take a pragmatic approach to determining a technology’s deficiencies and capabilities.  I still don’t do this enough.  This was one of the themes of Rich Hickey’s “Hammock Time” talk at the first Clojure Conj.

But it’s not what led me to later realize the legitimate use cases for XML.  I digress.

What I later realized about XML as I programmed more is that the intrinsic structure, semi-structure, or non-structure of the data you are working with is only part of the question, and only leads you to a conclusion - well, opinion, mostly - on how to best represent it.  The more complicated the domain, the more complicated your opinion, and the hairier the ultimate representation.

As you bang out your implementation, the less intuitive but really insidious part of the question lurks: “Will people in the future working with this thing I created be able to use it and modify it for their own needs?  Will they know my intent well enough to improve upon it, or determine if some variant of it is “valid” in the sense that it is true to my original opinion, on which other consumers of this thing may depend?”

For domains that are simple in the ontological sense, it seems really dumb to use XML, because that question doesn’t seem worth addressing.

That’s because you’re confident other people could sit down, read a few your examples, and be comfortable adapting and extending your stuff.  JSON and YAML are great for applications that fit this scenario, especially when you don’t have to worry about transport to or from places outside of your control.  Like on a web app, when you’re doing Ajax stuff with your own backend.

Sometimes though, you need to transfer data between applications - applications that are coded by different people, who might have different perspectives of the underlying domain that the data being transported is derived from.  And that’s when you need to look into XML, because of all the formats out there, it has the most tooling and documentation around writing schemas.  XML schemas let you express your understanding of the domain, and how you chose to interpret pieces of it, declaratively, in an almost human readable format.  More readable than BNF in my opinion, anyway.

It’s easy to know when you’re doing it wrong.  Is your code littered with assertions about the structure of data coming from some outside source?  Are there failure cases for your application that can be triggered by malformed data sneaking in?  Do you have tons of unit tests around this?

If you do, you might be comfortable.  You might feel good.  But what about the other guys?  If you have multiple developers working multiple code bases, all of which depend on some common interchange format, then the data on Bob’s box running application Q actually depends on the unit tests running on Sally’s box running application Z.  You can commit another cardinal sin and duplicate the tests, version and integration-test all of these apps together, or you can suck it up and investigate a transfer format that supports schemas.  Depending on your transport and other requirements, this might be XML, or it might be something like Thrift.

Schemas are not the answer to every question, and, like any tool, they can be the wrong one for the job.  But in my limited experience, there’s a place for XML and it is easy to overlook because of your own cultural bias.