slidetocode.com

The Great Programming Language Continuum; or why C++ is dying

2013-01-20T00:00:00-08:00

The long term trends of TIOBE's programming language index tell a sad story for C++; it is on the decline. Once comparable to C in popularity, it has lost market share to a new breed of higher level languages. During the same period C's market share remained has retained its popularity. Why has such an old fashioned language thrived whilst C++, its more modern object oriented cousin, has begun to fade?

This is best explained by imagining that all programming languages live somewhere on a continuum between the lowest level language of all - Assembler - and what would be the highest level language of all if anyone could write an interpreter for it - Human speech. A small subset of the this Great Programming Language continuum ordered from low level to high level is

Assembler
C
C++
Objective-C, Go
Java, C#
Python, Perl, Php, Ruby
Lisp
?
Human Speech: Chinese, Spanish, English, . . .

Many languages are missing from this list, but it paints the correct picture. As you descend the list, direct commands to the processor are replaced by conceptual abstractions and the source code will reflect the problem being solved more than it will the instructions the computer must execute to solve the problem. Code written in a higher level language tends to be easier to read, shorter and quicker to write than code in a lower level language, but the resulting program almost invariably runs more slowly. When choosing a language for a problem a programmer must balance the performance of the resulting software against the time it will take to write that code.

Law of lazy Coders: Programmers will always code in the highest level language possible

The higher level the language you write your software in, the fewer lines of source code you will write, and in most cases, the faster and cheaper it will be to build that program. For these reasons, real world time and cost constraints force the majority of programmers to choose the highest level language possible when starting a project.

The highest level language possible is usually determined by the hardware. In the early nineties performance considerations mandated that desktop applications like spreadsheets and word processors could only be written in C. As CPUs sped up during the nineties, C++ and then Java became good alternatives for this class of application. With today's lightning fast batch of processors and high performance JIT compilers, even a dynamic language such as Javascript is an option.

C++ was popular for writing desktop software in the nineties, because the answer to the question what is the highest level language I can write a good spreadsheet in? was C++. During the noughties, advances in processor technology changed that answer to C# or Java, and programmers quickly migrated to friendlier, garbage collected languages with large standard libraries.

C has retained its popularity because the answer to the question what is the lowest level language I can code in without using assembly has always been C, and it is unlikely to change. For those who cannot afford any latency in their code, whatever the cost to the programmer, or who need to plug into the lowest levels of the operating system then C is, and will remain, their language of choice.

Haskell is interesting to me because it is tough to definitively place it on the continuum. Its native compilation and static typing make it low level when compared to its distant functional cousin, LISP. However, lazy evaluation, Algebraic data types are characteristics of a much higher level language. This is the magic of Haskell. Idiomatic Haskell knits together high level and low level aspects with Monads.

Take, for example, Parsec, a Haskell library for building parsers. Parsec itself handles low level string manipulation and wraps this in the high level Parsec monad to define a Domain Specific Language perfect for parsing. With this DSL defined, writing an actual parser is criminally easy. Take the following example of an if statement in BASIC. You can see how the ifstatement parser is constructed naturally out of parsers for smaller elements of text.

ifstatement = do
  wspaces
  string "IF"
  wspaces
  cond <- condition
  wspaces
  string "THEN"
  wspaces
  eol
  blk <- block
  wspaces
  string "ENDIF"
  wspaces
  eol
  return (EIf cond blk)

C++'s decline will continue. Hardware advances will continue to make its efficiency redundant and Domain Specific Languages are nudging the principles of functional programming into the mainstream. Like world peace and lower taxes, we have learnt to distrust the promise that functional programming is about to spring into the software industry's cube farms. Nevertheless, even conservative programmers have come to trust Linq, a dataset DSL for C#.

Five days in Egypt in September

2012-12-18T00:00:00-08:00

Saturday

My most indelible memory of Cairo airport is the tennis court sized poster of Obama that greeted us in arrivals. Alongside Obama's face was a quotation from him testifying that the Egyptians are America's friends in the middle east. This was the start of our five days in Egypt, but Cairo airport is neither more Egyptian than London Heathrow, nor less English. Our trip began for real when I opened the taxi door into the humidity, the noise, the lights and the chaos of Talaat Harb street at 11pm.

The taxi driver guided us through the food stands and the street sellers, past a jewellery shop into a tall courtyard off the street to where a bright "Bella Luna Hotel" sign pointed into a forbidding entrance hall. When built, the colonial era staircase in that entrance hall could have belonged to a palace, but today, with worn granite steps, no lights, few repairs and no paint or decoration other than a layer of dust this staircase could equally have led to hell. Fortunately it didn't, and five flights of steps later we arrived at the Bella Luna Hotel's bright reception hall, and our bed for the night.

Sunday

In the morning we left our hotelier to "make connections" regarding our travel to Luxor later that day and we set off for the pyramids. Two things stood out from this visit: the scarcity of tourists, and the intense hassle. Considering that the Great Pyramid of Giza has been on every traveller's must-see list since Herodotus immortalised it as one of his Seven Wonders I expected the Giza plateau to be awash with admirers, but it was empty. Empty, but not calm, for with few tourists the guides and hawkers were scrambling over those of us who were there.

I wish I could say that the pyramids had gained meaning for me as testament to the power of one of the world's greatest civilisations, or even as a monument to man's vanity, but on that day they were merely a big pile of rocks that provided shade as we swatted away panhandlers, "egyptologists" and even Egypt's tallest policeman. This crowd of hucksters is the only detail of the pyramids that cannot be seen from afar, and I would advise you not to approach closely enough that you ever experience it.

Tiring quickly of the hassle we returned to central Cairo to browse the Egyptian museum's vast collection of artefacts. The collection is so big that during three hours we walked past no more than a third of their displayed collection. The displayed artefacts are only small part of their full collection, most of which resides in the basement. When leaving I realised that whether I had spent another hour or even another week in the museum I would still have left having seen just a small fraction of their collection. In fact I am told that if one were to spend a minute on each piece on display it would take nine months to see them all, and one minute is not enough time in which to do justice to a five thousand year old, intricately hieroglyphed and highlighted sarcophagus.

Upsettingly, these artefacts are neither properly cared for nor curated. The few objects that are encased and marked carry peeling and browned labels that look as old as the artefacts they describe. Most objects, however, are unprotected from either the climate, light or the wandering hands of tourists. Undoubtedly the British Museum's possession of illegally expatriated artefacts is wrong, but so is such neglect of historical treasures. So much so, that after visiting the Cairo museum I was in doubt for the first time that it is best for western museums to return to Egypt the stolen historical treasures they contain.

That evening we returned to Giza to catch the sleeper train south to Luxor in order to see the lavish tombs in the famous Valley of the Kings. You can trade this ten hour train journey for a seventy minute hop in an airliner, but I have no idea why you would. You would miss the romance of being rocked to sleep in your cabin by the twists and turns of the Egypt Railways number 82 train as it charges through the night down the Nile valley. Sleeper trains provide a sense of adventure without the discomfort of a real adventure. Until someone builds a space rocket with a jacuzzi, an overnight train remains the classiest mode of travel, and this trip from Giza to Luxor is an unmissable experience if you ever find yourself in egypt.

Monday

Never was Egypt's problem with urban hassle more apparent than outside Luxor train station. We fought through a crowd of agressively helpful taxi drivers to leave the station and waved off at least one taxi driver a minute during the thirty minute walk to our hotel, the beautiful and peaceful Nur el Gurna.

Nur el Gurna's proprietor, Mahmoud, offered us tea and introduced us to his wife, daughter, sons, sons' wives, grandchildren, cats and ducks, all of whom live at Nur el Gurna and work variously as cook, cleaner, taxi driver and farmhands for the livestock he keeps. Life in Luxor, Mahmoud informed us, is not like life in Cairo. Families stay together in Luxor.

Mahmoud confirmed our suspicions about Egypt's tourist trade. Since 2011's Arab spring and the Egyptian revolution, tourists had stopped coming. When we visited there wasn't even a quarter the pre-revolution volume of tourists. Our pyramid guide had dropped from three tours a day to one tour at best, Mahmoud's luxurious rooms were empty, and many of the tombs we would visit in Luxor would have been empty if it weren't for our presence. Unfortunately with so few tourists, those dependant on the dollars they provide have to fight even harder. By the time we left Egypt we had a good idea what the rugby ball must feel like at the end of a match.

After an unforgettable lunch of tahina, kofta and spiced potatoes we set off to see some of the tombs that Luxor is famous for in the nearby Deir el Medina. Deir el Medina itself is not the resting place of the pharoahs, it translates to "worker's village" and it contains the tombs of the workers who constructed the Pharaoh's tombs. Having laboured fulltime on the Pharaoh's tomb, these workers would head to Deir el Medina on their rest day to construct their own. The tombs are minute in comparison to the Pharoah's palaces of death, but without the riches to attract grave robbers these tombs remained sealed until very recently and these were the best preserved tombs we saw, with murals as colourful as the day they were painted.

Archaeologically speaking, the worker's tombs in Deir el Medina, and the tombs of the Egyptian noblemen in the nearby Deir el Bahari, are of interest because the wall paintings show daily life in ancient Egypt rather than glorious battles inscribed upon the walls of the Pharaoh's tombs. Personally, I was fascinated to see how the ancient Egyptian obsession with death permeated the entire society.

Once we had explored sufficient tombs for the day, we started down the desert hillside towards the hotel. Almost immediately a heavily armed policeman appeared, welcomed us to Alaska, and walked down the hill with us. I'm not sure whether I should write "walked with us" or "escorted us", most likely the latter. Egypt's broader struggle between the progressive, Western elements of its society and the highly conservative Islamic elements is manifested in the tourist areas as a struggle between the small army of Tourist Police and fundamentalist terrorists. The Tourist Police are numerous, well armed and the occasional bullet hole in their shields and trucks shows that they are not for show. Thankfully, they are effective and there have been no serious attacks on tourists in the past few years.

Tuesday (September 11th)

The following day we caught a taxi to the world-famous Valley of the Kings - the preferred resting place for Egypt's great Pharaohs, most notably Tutankhamen and Rameses IX. Unlike the tombs of the nobles and the workers we had seen the day before, the Valley of the Kings is a mainstay of tourist trail. It is a professional operation set up to cater for 7000 tourists each day, with a visitor centre, an educational video and a fleet of road trains for ferrying visitors from the visitor centre up the valley to the tomb entrances.

The Pharoah's were buried with as much gold and jewellery as their tombs would contain. These treasures are tempting to graverobbers and most of the pharoah's tombs have been open for millenia, with the notable exception of Tutankhamen's tomb, which remained sealed until when 1922, giving Howard Carter a chance to rob it. Having been opened, the elements have stolen much of the richness and detail from the wallpaintings, but the cavernous hand-hewn chambers are spectacular monuments to their original occupants, and in many the massive, intricately carved granite sarcophagus has been left in place.

One tomb had a mirror set up on the floor to show the underside of the sarcophagus lid where the figure of a woman had been carved, presumably to ensure her husband was fully satisfied in his afterlife. Museums and books present history in such a dour and stuffy manner that one assumes the participants must have shared this formality. This five thousand year old dirty joke carved into the lid of a Pharoah's sarcophagus reminded me that the participants in history were not as humourless as the subject often makes them seem.

Soon our book of "Tomb tickets" was fully stamped and we hopped the empty roadtrain back to the visitor centre to meet Mahmoud's taxi driving son. The roadtrain and accompanying infrastructure lend the Valley of Kings the air of a well managed heritage site, but insufficient money and effort is directed at preservation of the tombs in the valley. These tombs, the pyramids, the temples and the objects in the Cairo Museum are gifts from the ancient Egyptian society to their modern descendants, and they are largely responsible for the 30% of the Egyptian economy based on tourism. Every time I saw a tourist brush his or her hand across a wall painting four millennia old, or a tomb attendant bump his hat against the painted ceiling, or witnessed the damage done by the rainwaters that are allowed to flood these tombs each winter I wondered how long this gift would last, and how healthy Egypt's economy will be if the tourist sector dries up.

Wednesday

After a second night in the return sleeper train we woke up wednesday morning in Cairo. By now we had seen enough of how the ancient Egyptians were buried, and we were keen to peek into the daily lives of their descendants. Leaving our hotel we turned south in the direction of the Islamic old town hoping to see the ancient Citadel.

The guidebook informed us that Cairenes treat the street as their living room. Cafes and shops spill out onto the pavement in any warm western city, but Cairo is the only city where I've had to duck a tray of croissants pulled from a hot oven left on the pavement by the baker. In Cairo machinists and bakers ply their trades on the pavement side by side with cafe owners, street sellers and passing pedestrians.

Amongst the combined smell of sweat, baked goods, spices and car grease on Cairo's streets, the insincere and avaricious friendliness of Egypt's tourist spots is replaced by genuine hospitality. We experienced this generosity firsthand when we stopped at a coffee shop to ask directions to the Citadel we were searching for. Almost immediately we were surrounded by a small crowd of men arguing amongst themselves as to how best we could get there, and inquiring about our backgrounds. They were friendly, engaging and interesting; a better advertisement for their country than the sea of hawkers surrounding the pyramids, even though we never made it to the Citadel.

Thursday

Our last day in Egypt was an early start and an uneventful ride to the airport. We were fortunate to leave when we did, for the Tuesday of our visit was the day when protests over the "Innocence of Muslims" film started and four Americans were killed in the US embassy in Benghazi. An Egyptian Coptic Christian was the source of the film and there were additional, peaceful, protests outside the US embassy in Cairo. The situation drew comments from Obama that he did not "consider them [The Egyptians] an ally...".

Having seen the scale of the Egyptian government's struggle with Islamic terrorism through the size and ubiquity of their Tourist Police I found this comment ill considered and crass. America's war on terror is neither an abstract, nor a distant, concept to the Egyptians. It is fought every day of their lives, all over their country, and as the bullet holes in the police shields show, it is not a bloodless conflict. If these people are not Obama's allies, I am forced to question what war the Americans are fighting in the Middle East?

Most recently Egypt has made the news due to protests over the emergency powers President Morsi awarded himself, a referendum over these powers, and then further protests over the suspicious referendum result that effectively installed Morsi as Egypt's dictator. It pains me that these news broadcasts are undoubtedly shrinking Egypt's tourist trade even further, undermining the moderate sectors of Egypt's society and probably closing Egypt's doors to tourists like ourselves in the future. I enjoyed my visit to this friendly nation, with its beautiful landscape, and its artefacts from its proud history. It is heartbreaking to think of Egypt developing into just one more hard line Islamic state.

Best before 2015. Should software come with expiry date?

2012-11-10T00:00:00-08:00

Buy a car and you would be lucky to have any original components other than the steering wheel and the stereo left after 200,000km, but software's digital nature lends it an air of immortality that leads many to believe it will last forever. Indeed there are BSD tools on my laptop that have been doing their job for 30 years. They will undoubtedly last another 30, but these tools are self contained, and unlike a great deal of modern software they don't interact with much less immortal third party APIs.

I considered this question when experimenting with cloud support for my iPad Clojure editor. Cloud evaluation of Clojure would improve the product, but I don't want to write and maintain the servers myself. Ideone offer superb, and free, cloud evaluation of Clojure. It would be the perfect solution right now. I could effortlessly add cloud evaluation to Lisping, but startups live fast and usually die young, so what happens if Ideone were to fail or to start charging? Customers would have paid me for an application with free cloud evaluation and they would no longer be getting it; I wouldn't be surprised if they demanded their money back.

Another example is Twitter's war with third party clients. Earlier this year, having built their service with the help of third party developers, they began to shut off access to their API. They have yet to use their nuclear option - an outright ban on third party clients - but it is not unlikely that they will do so one day. What then? Tweetbot and its competitors will no longer be fit for their advertised purpose. If a car failed like that you could get a new one.

In the case of a Twitter client you are paying for software to interact with Twitter's public API, so when that API ceases to exist, the developer cannot be held responsible, but it is a serious question with my Clojure editor. Ideone have a great service and they are doing well, but technology moves fast. It is almost inconceivable that any current 3rd party API will have survived in its current form in 10 years time, so if I were to base one of Lisping's core functionalities on a third party service then I am selling software with a built in expiry date.

There is plenty of precedent for "Best before" dates in the retail industry, but the difference between a cloud app that has lost its cloud and the mouldy block of Gouda in my fridge is that the Gouda came with a label clearly stating its expiry date. Software should come with the same guarantees. Please state how long you intend to support it for. Please state if you will update it when the

For me the solution was simple. I'm not a fan of the cloud, and I've always wanted to write an interpreter, so I've started writing an easily embeddable Clojure interpreter in C++. When it's mature I'll add it to Lisping. With a native interpreter, who knows, maybe Lisping will still be running on the 51st generation iPad. With luck I'll still be alive to add support for the lightsaber peripheral.

Marble mazes, edge detection and an iPhone 3g

2012-10-30T00:00:00-07:00

It started with a simple question. It was summer 2009 during the wild west days of the App Store when fart apps were cool and an android was still just that pale guy from star trek. My housemate pointed to my iPhone 3g and asked - "why don't we make some money out of that? lets write an app of our own.". Soon a page of scrap paper was covered in ideas; we crossed out all variations on the fart app and continued to whittle away until we had found our calling. Remember those marble run toys? we were going to turn my iPhone into a virtual version of that. Draw a marble run on paper, photograph it, allow our AR engine to work its magic. Now drop a marble onto the photograph of the course and watch it roll down the ramps and off the jumps you just drew as if they were real objects.

The next week I was on holiday in egypt and I started work on the physics engine. Ben's image processing library would transform the photograph into a boolean bitmap image where cells would be occupied if a line was drawn at the corresponding point in the photo. The hardware limitations of early iPhones meant that this collision bitmap would be a much coarser grid than the original photo. The question was how could I hide this coarseness by "smoothing" out these squares so that the ball would run the drawn lines exactly as the user intended.

A physicist by training, I started by transforming each "true" cell into a repulsive potential and integrating the equations of motion. Numerical integration of such a system is a nontrivial problem. I was unable to strike a balance between a method that conserved energy in collisions sufficiently well to fool the eye, ticking fast enough to stop the ball iterating through lines, and cramming all these calculations into the limited capabilities of the iPhone 3g. Throughout all this work I had never once had an issue with the simple vector bounce code I had implemented to make the ball bounce off the walls and keep it on the screen. I realised that the problem would become very simple if the drawing were in vectorised format.

The original problem of bouncing a ball off a bitmapped edges had morphed into the problem of how to vectorise that edge map and use the simpler vector bounce code. That is a hard problem in general, but if you know that your ball is near an edge (a simple occupancy check of the surrounding bitmap squares will tell you that) then it is straightforward to find a local straight line approximation to the drawing. To do this I returned to my inverse potentials, placing one on each occupied pixel, and I assumed that the grad of the resulting scalar field pointed normal to the tangent line at that point. The assumption turned out to be reasonable and it was not long before the ball was rolling smoothly down a coarsely bitmapped ramp.

Meanwhile, back at home Ben was grappling with how to efficiently implement Canny's edge detection algorithm. This all had to happen in the CPU because the programmable graphics pipeline would only make it to the iPhone ecosystem following year with the 3gs. If we earnt our money at any point in this development process it was the night he spent tweaking the edge detector to distinguish between the lines that come with lined paper, and those drawn on by the user. It took a few more evenings to polish the UI, but in no time Snap 'n Tap was in Apple's review queue.

The perfect conclusion to this story would be "we sold a million copies in a week, and by the end of the month we were slinging caviar at each other during a foodfight on my private yacht". That didn't happen, but we did make money. Certainly enough to recoup the cost of the iPhone, not enough enough for a helicopter, but more than enough to teach me that I was capable of making money. When I finished my PhD a year later I didn't thumb through the classifieds to find a job, I headed to canada to start a mining tech company. Although successful, spending my days down a mine and the evenings in -30C ruined the experience. I am now back in Europe building a software company with another old friend from PhD days. I'm proud to say I've never had a job, and thanks to Snap 'n Tap's lessons I don't think I ever will.

My iPhone 3g died quite soon after Snap 'n Tap's release, so there was only ever one update. It seems wrong to allow Snap 'n Tap to decay after the life changing lesson it taught me, so as soon as Apple get me my iPhone 5, it is update time. I will add support for retina displays, the new 4" screen and maybe even the camera iPads. Meanwhile, try it out for yourself in the App Store.

My DNA needs some refactoring

2012-10-08T00:00:00-07:00

Recently, I had a personal epiphany when I realised that clean code and good code are very different. This fact struck me when thinking about DNA. To a programmer's mind the proteins in a cell make up a virtual machine, and DNA is the code they execute. Furthermore natural selection guarantees that this is "good code" in the sense that it is proven to be effective in the real world. The problem is this code is crap - I'm assured by Biologists that junk DNA was a poorly chosen term and that recent research is proving that these base pairs are useful, but even so our bodies are contain an appendix, a tailbone and other vestigial structures - known to programmers as cruft. In short, despite being the recipe for the biological marvel that is the human body, our genomes could do with a refactoring.

I'm joking a little, but the parallel with ugly, but useful, code in "real-world" programming is good. You dream up a piece of software, write 1.0, and the code is beautiful. You release it and it is useless, its incompatible with Word 97 files, so you put in a fix, it crashes in the presence of a virus scan, so you put in a fix, it is . . . and before you know it your fixes are overrunning your code, so you get to work straight away on refactoring. Unfortunately the emails keep coming and this is a battle you are destined to lose. Constant refactoring is important to keep your code maintainable, but it is unlikely to keep it beautiful. Never mind though; writing useful software is an exercise in dealing with the special cases, and inevitably your codebase will reflect that lesson. These hacks and kludges are the code equivalents of your appendix and tailbone. Their existence is unjustifiable, in fact they are better out than in, but they go hand in hand with any well developed real world system.

I can already hear your complaints - this comparison is unfair, if our cells are a virtual machine, then DNA is more accurately compared to object code, not the source code it is compiled from - object code needn't to be tidy, it needs to be efficient. Good object code is efficient and good source code is beautiful. This leads me to my true, and admittedly obvious, epiphany. The sole job of a programming language is to bridge the gap between ugly, but efficent, object code, and beautiful source code.

I think this obvious point is often lost when discussing programming languages, especially when people are evangelising for functional languages. No one cares that a functional language makes it possible to write provably correct code, we do care about the flexibility that a functional flavour adds to a language, especially when a language is flexible enough for Domain Specific Languages.

Haskell's Parsec module was my lure into the world of functional programming, and it is still the best example I can give for the "why should I learn a functional language?" question. It is a parser library which allows you to rewrite the composition of grammatical elements as the composition of functions and build a parser directly from BNF without needing to rely on an external system to autogenerate code. The beauty is that it doesn't feel like a parser library, it feels like a programming language specifically for writing parsers.

The first parser you write with Parsec is the first parser you will write in which your code will reflect the structure of the data being parsed, not the highly efficient finite state machine it will compile to. If you want to know why Haskell is a "better" programming language than C/C++/C#/Java, this is why. In "curly braces and classes" family of languages you have the choice between writing the state machine from scratch, autogenerating your code with bison/lemon or if you are lucky a cunning but ugly compile time hack (sorry Boost::Spirit). In Haskell your code is a work of art. It reflects the grammar being parsed which makes it concise and self documenting. Nevertheless the compiler bridges the gap between this abstract elegance and the real world by churning out a blazingly fast executable.

How Clojure is to the JVM as Bash scripts are to Unix

2012-09-26T00:00:00-07:00

I'm undoubtedly one of the brainwashed hordes that believe Clojure is the best thing ever to happen to the Java ecosystem. Possibly this is because I am a longtime Lisper glad to see my favourite language as the centre of attention once more, but I do believe that the JVM was an ecosystem in need of a great dynamic language, and for me nothing fits that description better than a modern dialect of Lisp. Unfortunately I can't join in with the Clojure cheerleaders who suggest that it can be a complete replacement for Java. For me Clojure is to the JVM what Bash scripts are to Unix; it is an excellent choice for tying together JVM libraries into a web application, but a dynamically typed language without natural support for OOP is no use for writing those fundamental units of code without which Clojure wouldn't be nearly as useful.

Whatever your opinion on OOP as a programming paradigm (mine is "unconvinced") it is the foundation of code reuse and sharing in the JVM ecosystem and Clojure lacks natural support for writing JVM objects. I am well aware of gen-class, but is not idiomatic in Clojure, Lisp or functional programming in general. As such gen-class doesn't make clojure Object Oriented any more than C++11's lambda's make C++ a functional programming language. Another facet of Clojure that makes it a poor choice for writing JVM classes in is its dynamic typing, once again you can circumvent this with type hints, a necessary evil that is hardly idiomatic.

So if Clojure is a poor choice for OOP, it is a poor choice for writing JVM components, and it can only be useful as a language for tying together these JVM components - which after all is what makes it perfect for Web Applications. This isn't a criticism of the language, it is just a comment on its place in the ecosystem. I don't blame Bash for being a poor choice of language to write a Web server in, and I don't blame Clojure for being restricted to solely high level tasks.

However this incompleteness potentially explains the comparatively slow uptake of Clojure in enterprise when compared to other JVM languages such as Groovy and Scala. Given the choice of training a workforce in one language suitable for almost all tasks or training them in multiple languages, the tech-unaware boss is always going to choose the cheap option - one language for all situations. After all that was the secret of Java's meteoric rise in the first place.

A proposal for improving HN - it should cost Karma to comment

2012-09-07T00:00:00-07:00

I'm not alone in watching HN's downwards slide to the level of most other internet forums where posts and announcements are met by default with thoughtless criticism, complaints that the post is offtopic, and other obnoxious contributions. To an extent this is inevitable, but recently in some cases the noise has become loud enough to drown the thoughtful discussion the site is known for. I have a suggestion for improvement: it should cost Karma to comment, and when your Karma drops below a certain threshold you can no longer comment.

My reasoning is that the trash comments we all hate are kneejerk criticisms with little thought put into them, which I suspect are largely made for the sake of saying something and collecting Karma.

Assuming that Karma is increased as people comment on and upvote your comments/submissions, I propose altering the system such that making a comment should cost the equivalent quantity of Karma that would be gained by a single upvote or comment in reply. This way comments with so little value to the community that they are ignored completely or just downvoted would reduce the commentator's Karma, bringing it to the level at which he or she would no longer irritate us with his or her pointless submissions.

The rough edge with this suggestion is new accounts. I don't want it to become impossible for a new user to post, but if each new and unknown user gets "5 free comments", then the problem isn't going away. I am not suggesting suppressing anybody's ability to post however, so with this system new users would have to contribute a voteworthy or commentworthy article before being able to comment, and I don't think that would be a bad thing.

Lessons learnt whilst writing a development environment for iOS

2012-07-14T00:00:00-07:00

The past six months I have been working on a Lisp editor for iOS as a side project (Lisping). It has been great fun, and building IDEs for touch screen's is a project I believe in. Portability, ease of use, and low cost will guarantee a tablet's place (whether iOS, Android or windows based) as my travelling companion for many years to come, so being unable to code on a tablet is unthinkable for me.

Unfortunately the current generation of source code editors are slow and inefficient for coding. Lisping is my take on the solution, but it is a large market and I want to see a wider range of higher quality products. I'm not finished with Lisping, but I would prefer to writing projects on an iPad, not writing an iPad IDE on my desktop, so with a inflated opinion of my own wisdom I am sharing the biggest lessons I have learnt whilst building developer tools for iOS.

Don't fight the cloud

I learnt this one the hard way. I like to feel that my iPad is computer, and that it should be capable of everything a computer is capable of including onboard execution of an interpreted language. Technically speaking this is true, but Apple's terms of service have shackled the device for developers so much that it is unrealistic to build a good development environment that is also capable of executing code on iOS.

As an example, in the first version of Lisping I included an embedded Scheme editor rather than executing code offboard. This seemed a faster, more convenient approach than cloud execution, and in terms of execution speed and usability it is. However, to quote Apple Apps that download code in any way or form will be rejected, so I had to disable the code to import a file from an email or dropbox. Execution is immediate, but as a result the user has to copy/paste code from an email to import it, which overall has harmed the product. Needless to say I'm not repeating this mistake with my current series of updates to include Clojure, where all execution will be offboard. Most likely I will move Scheme execution offboard in the future as well.

It is frustrating to make as fundamental a design choice as this solely to circumvent cretinous review guidelines, but if you want to build developer tools rather than complain about Apple policies, you need to accept very early on that the cloud will be part of your product.

In the case of iOS IDEs you are lucky because your target market is composed of grownups (in the coding sense of the word anyhow). They know the situation as well as you do, and they will be grateful for a good product. Criticism of Apple imposed flaws in its design will be directed at Apple rather than you.

The iPad doesn't have a keyboard

This is the mistake almost every editor makes, whether it is designed for prose or code. The iPad doesn't have a keyboard, it has no up/down/left/right keys, no escape or tab keys, and the standard gestures for moving the cursor and selecting text are lame, so don't try to rewrite Visual Studio, Textmate, XCode or even Vim in iOS with the system supplied textviews and controls. It is a touchscreen platform, build a touchscreen editor.

The phrase "easier said than done" was coined for advice like that. In Lisping I went with the approach of editing the parse tree. It works great for Lisp, a language built around its parse tree, and I am keen to try it for Python, but I doubt it would be appropriate for the iOS C++ IDE I yearn to download (or if necessary write). Whether it is to your taste or not, it's approach is undeniably focussed on the iPad's touchscreen.

Aside from experimentation, writing and rewriting, I don't have any sound advice for exactly how to build the great touchscreen editor, but I can tell you that it will exploit the touchscreen the way that Vim exploits your keyboard.

Keyboard accessory views are a necessary evil and there is no easy way around syntax highlighting

There are two fundamental technical limitations with iOS's UIKit that any aspiring IDE developer must work around. The first is that you can't configure or subclass the system keyboard (why not Apple? seriously, what harm would it do?), and secondly that UITextView has no support for syntax highlighting.

The system keyboard does come with "keyboard accessory views", a thin strip along the top of the keyboard you can use to add extra keys. It is ugly and reduces the editable area, but unless you want your users to get RSI from swapping back and forth between the three keyboard pages, you must provide an accessory view.

Syntax highlighting is the real problem - UITextView will not display an attributed (syntax highlighted) string, and you cannot override its drawing code in a sensible manner. There is a nice workaround, for small snippets of text, a promising but incomplete drop in replacement for UITextView, and the sprawling OmniGroup framework that I couldn't bring myself to link into my code. Sorry, but if you want syntax highlighting, you will have to write your own text editor. It is a painful experience, but console yourself with the fact that Apple's UITextView is lame for editing code anyhow, and this is a chance for you to start from scratch and get it right.

Edit Clojure by its parse tree on your iPad - Lisping adds support for Clojure

2012-06-26T00:00:00-07:00

It has been three months since I released Lisping into the App Store. After smoothing out the mechanics of editing by the parse tree in the 1.0.x series of updates, I felt it was time to add support for Clojure, a great new JVM Lisp which is replacing Common Lisp as the "production" member of the Lisp family.

For those unfamiliar with Lisping, it is a touchscreen Lisp editor in which you directly edit the parse tree. Editing source code character by character is a concept wedded to the keyboard, and Lisping discards this in favour of selecting, creating and moving the syntax elements that reflect the true structure of code, an approach far better suited to the iPad's touchscreen than the traditional text editor.

Unfortunately Apple's strict prohibition against the JVM on iOS, and ClojureScript's lack of an (eval) function prevent Lisping from executing Clojure natively as it can Scheme, however next up on my TODO list is integration with ideone.com's superb API so that Clojure can be evaluated in the cloud from within Lisping.

Adding Clojure has upgraded what was once a personal coding experiment into a useful tool, nevertheless I am still working on Lisping, and I love to hear feedback, so feel free to get in contact.

Lisping 1.1 now supports Clojure, bugfix on the way

2012-06-16T00:00:00-07:00

Lisping 1.1 now supports Clojure, but unfortunately there is an irritating bug with the editor window so I am holding off on announcing it for now. I have submitted a bugfix to Apple along with a request for an expedited approval, so I hope 1.1.1 will be with you all very soon. In the meantime, apologies for any inconvenience caused.

The seven wonders of a hacker's world

2012-06-15T00:00:00-07:00

It's not an innovative title, but it's a great excuse to step back and review what hackers have done well, so, in no particular order, here are the projects that I think represent the seven wonders of the technology world.

CryEngine

I watched a stunning video showing Cry3's physics engine last week, then another yesterday on the upcoming Unreal 4 engine. More memorable than Unreal 4's visuals is the unremarkable comment by the level designer that to light your world, you place a sun in it.

Game lighting is no longer a case of balancing ambient, diffuse, and specular components to achieve a passable imitation of realistic lighting, you build your world and give it a sun, the engine simulates the rest. Game engines aren't the stylised facsimiles of the world I remember from my youth, they have become simulations of the world, and going by the soft body physics in Cry 3, they are pretty good ones.

Don't read too much into my choice of CryEngine over Unreal's lighting, or Id's sparse voxel octrees, but physics trumps lighting when it comes to simulating our world so I went with Cry.

Google's V8 Javascript engine

V8 hasn't made it onto the list because I appreciate javascript as a language, it is here because it is so damn fast. Surprisingly, unpredictably, and unreasonably fast. In fact it is so fast, that in at least one benchmark it is regexes more quickly than C, and this benchmark shows javascript to be a defensible choice for fastest statistical language, sometimes outperforming Matlab by a factor of a 1000. Javascript shouldn't be that fast; it was designed for minor scripting tasks and it is as dynamically typed as they come, so why doesn't it languish at the back of bechmarks with other languages in its class such as Perl, Python and Ruby?

The credit lies with V8. In general Javascript doesn't outperform anything, and historically it was as slow as other languages in its class; Google's V8 engine was the magic ingredient. I don't know how Google's engineers did it, but they beat the old cliche and made a silk purse out of a sow's ear when they wrote V8.

V8 has accelerated web apps just as Google hoped it would, but its blistering performance has promoted javascript into the realm of the backroom server. Node.js could never have existed without V8, so Node's existence is further proof of how the V8 design team achieved the impossible.

LaTeX

Technology's great legacy will be its democratising effect. Blogs made everyone an author, Expedia broke the hegemony of travel agents, and web cams have made everyone a porn star. LaTeX's gift to us all is the ability to write beautiful documents. It makes no guarantees that your content is worth reading, but you can be sure it will be beautifully laid out.

This isn't a simple trick to perform, and fourty years of performing it has left the LaTeX distribution slow, bloated, and temperamental, but when you need the best for your document, there is nothing like LaTeX.

Autopilots

I couldn't decide whether to include the recent 787, the pilotless global hawk or the software systems that guided the spirit and odyssey to softs landing on mars. Instead I'm taking their common thread - the autopilots. Flying is not a simple skill, and although autopilot software may not be at the cutting edge of artificial intelligence research like chess playing software is, it does perform a great service for the human race. Besides, any software package that can take 200 tonnes of plane, passengers and luggage from a runway in London, and guide it across the atlantic to a 15m x 50m patch of tarmac in New York is just plain cool.

Lisp

A programming language had to appear somewhere on this list and I chose the Lisp family, the traditional weapon of choice for the hacking elite. Its longevity certainly contributes to its stature, but Lisp isn't just a 50 year old Kool aid. Whereas most languages are packaged with bulging featuresets, Lisp presents the hacker with the building blocks of a language, and the flexibility to craft it into a language perfect to any task.

Perhaps the most convincing justifications for Lisp's place on this list are the hundreds, if not thousands, of variants created since its introduction. It isn't just possible to experiment with Lisp's syntax and perfect the language, it is worth it. Even now, fifty years after its introduction, Lisp is lighting up the notoriously fickle world of technology under the banner of Clojure.

Linux kernel

The "one-liner" is often deemed to be the pinnacle of the hacker's art. It is the triumph of resourcefulness and brevity over vast quantities of code, documentation and test suites, which are a little too reminiscent of a cube farm. Despite is size.

Linux was almost dropped from my list because, like the logic gate and the internet, it is one of the pillars of a hacker's world, but I kept it because it was created by hackers, for hackers, and it is here to represent all the great open source projects (mysql, apache, gcc et al.) without which the tech world wouldn't work as slickly as it does.

The hashtable

Choosing a hashtable could be seen as sending this list out with a whimper rather than a bang, after all when I think of a hashtable, I think of undergraduate CS courses and standard libraries, not the technological equivalent of the pyramids. However, for those of us in what is often called the information industry, organising and finding information is our bread and butter and no tool is more used for this task than the humble hash table.

I say humble, but a hashtable's high-level design is a remarkable piece of Computer Science, and when implemented well it can be an even more remarkable piece of Software Engineering. Potentially a Relational Database such as MySQL could have filled this slot on my list, or possibly the B+ tree that so many databases are built on, but neither of these play such a fundamental role in a hacker's life as the hash table, so despite their often awesome scale, none of the hashtable's competitors actually organise as much data as the hastable.

Attractive to theorists and engineers alike, the hashtable has been on the frontline of data processing, storage and mining since its invention in the fifties, and that is why it is the perfect hack. It is simple, it is robust, and it has almost unlimited usefulness.

The problem with Clojure's type hints and Arc's if statement

2012-05-29T00:00:00-07:00

Arc's (if ) statement has the same function as the classical Lisp (cond ) statement, but the parentheses around the conditional blocks have been dropped with the intention of condensing the code. For example, where in Scheme one writes

(cond ((= x 1) (do-this))
      ((= x 1) (do-something))
      (else (alternatively)))

The Arc equivalent is

(if (= x 1) (do-this) 
    (= x 1) (do-something) 
    (alternatively)

A form with fewer parentheses. These parentheses weren't decorative though, they exposed the structure of the code, and they are a minimal price to pay for extra clarity.

Recently I have been finishing off a clojure mode for Lisping, my iPad lisp editor, and I noticed a similar problem with Clojure's let bindings and argument lists whilst writing the auto-layout algorithms. The Scheme let statement

(let ((x 1)
      (y 2))
  (body-goes-here))

becomes

(let [x 1
      y 2]
  (body-goes-here))

Once again, without parentheses around the bindings, the code requires much more careful formatting for the sense to be clear. This problem is exacerbated by Clojure's optional type hints where there need not even be an equal number of elements in each line.

(let [^String x "x string"
      y 2]
  (body-goes-here))

JavaScript minifiers are a convincing argument that any code requires some formatting for the structure to be clear, but for me the line between Python and Brainfuck is drawn when there are no longer syntactic elements visually delimiting the structure of the code. By dropping these parentheses around bindings, Arc and Clojure are toeing that line.

The rationale in both cases is to reduce the quantity of parentheses, which seems a good idea given how they deter prospective Lispers, but Lisp's longevity and its unique slot in the world of programming languages stems from the straightforward correspondence between the parse tree and the source code, and to retain that these parentheses around bindings need to stay.

As a side note, in the case of Arc dropping these delimiters is a false economy because without parentheses around the (if ) clauses one must insert (do ) statements to include a sequence of S Expressions inside the (cond) clause.

This isn't a serious criticism of Clojure; it certainly won't stop me using it and I think it is a shame that Arc died young, but without sounding like someone who believes we should all be coding on Genera, I do think it is important for the new Lisps to retain the features that have made Lisp a language worth reinventing 55 years after its first appearance.

What does Github's top ten languages list tell us about its userbase?

2012-05-25T00:00:00-07:00

Earlier I saw a post on HN about the top ten languages on GitHub, and what struck me was how different the list is to Tiobe's top ten languages. My immediate question is who exactly uses GitHub?

This question was on a lot of lips earlier this week when rumours leaked out about Github raising a venture round. As a fan of both Github's philosophy and product, I hope it isn't true because for a company as proud of their bootstrapped status as they are, this trip to the Venture Capitalists looks like a sign of trouble.

TIOBE's most popular languages

C
Java
C++
Objective-C
C#
PHP
Visual Basic
Python
Perl
Javascript

Github's most popular languages

Javascript
Ruby
Python
Shell
Java
PHP
C
Perl
C++
CoffeeScript

Looking at Github's list it seems clear that Githubbers are best classified as web developers. I imagine C made the list because of its use in open source projects for whom Github is the new Sourceforge, but with a minor web language like CoffeeScript amongst its ten most popular, Github is undoubtedly the home of the "Web crowd".

Very notable is the absence of Visual Basic and C# from GitHub's list. Visual Basic is not the language of the hackers, entrepreneurs and open source coders for whom Github is the star around which we orbit, it is a language of enterprise and cubicles, a world in which decentralised source control is dangerously close to communism, where SVN and Perforce still reign supreme.

Unfortunately for Github it is the code drones that have the money, not the Rails hipsters. The TIOBE index is extracted from job advertisements, and it reflects what programmers are paid money to code in. Consequently it is a list of languages in which there is money available for hosting.

If I had to guess at the motives behind Github's VC round it would be that their 1.6 million users are cheapskates, they are hosting open source projects, personal projects, and early stage startups, all renowned for a reluctance to spend money. This VC money is reserved for their push into Perforce's mundane, but lucrative, stomping ground of enterprise source code management.

Would Steve Jobs have applied to Y Combinator?

2012-05-18T00:00:00-07:00

A quick glance at yclist shows how Y Combinator has become the war factory for the Web 2.0 revolution. If your aim is to take an existing application, strip away the cruft, replace it with a stellar user experience and see your product adopted by the world then Y Combinator is for you. Y Combinator's unique skill at turning around these web 2.0 companies is the reason why none of the copycat incubators have an alumni list or investment portfolio containing names such as Dropbox, Reddit, Clickpass, Posterous or Codecademy. I can't think of a better list. Except perhaps one containing Facebook, or Twitter, or a new Google or Apple. Is Y Combinator as limited in scope as it is successful? Is the next Steve Jobs or Bill Gates filling in Y Combinator's application form right now, or are they taking the independent route?

Although critical in nature, this post isn't just me taking a swing at Y Combinator and web 2.0 movement. Y Combinator deserves a great deal of credit for tempting college students away from banking jobs towards an industry where they can "change this world", unfortunately the focus of the resulting companies isn't so much on changing the world as it is fixing bad applications. This "change the world" mantra is universally quoted by Y Combinator founders (and deluded database CEOs), but the next great blogging platform is still a blogging platform, and a fancy database is still a database. I can't wait for a Y Combinator startup to build a viable and bloat free alternative to Excel/Numbers/OO Calc, it won't be changing the world in the way Visicalc did. Google on the other hand are building autonomous cars and Elon Musk is using his paypal fortune to build electric supercars and space rockets.

The lack of ambition and innovation is certainly not because of a lack of talent with Y Combinator founders. This point hardly seems worth expanding upon. Technologically speaking, the best of the best head straight to the orange room to begin their careers.

I think the problem is rooted in the business model that seems to be preached by Y Combinator: build a great product, choose investment over income, and exit as quickly as possible. Angels investors, quite reasonably, want to see a return on their investment, and what better way is there to realise this than a juicy acquisition cheque? The approach of remaining independent, and investing profits back into to the company followed by technology zealots such as Jeff Bezos and Steve Jobs is unattractive to an investor. Investores are well aware that many more people are capable of building a company worthy of acquisition than are capable of building a company like Apple, Amazon or Microsoft that is worth owning in its own right.

With this business model in mind, building better versions of existing products is the optimal approach. Not only do you have a model to start from and a guaranteed market, but if you succeed, you also have a very large, very scared, and very rich company who will buy you out for almost any amount of money rather than compete with you.

I doubt Y Combinator is going to change its (and by extension the Valley's) phenomenally profitable approach to investing, and I'm also sure that there are still plenty of crazy people out there working to change the world, but it is a shame that Y Combinator doesn't take a few of them under its wing so that the rest of us can feel the benefit of their creativity a little sooner (and with a much better user interface).

The genesis of a perfect programming language

2012-05-06T00:00:00-07:00

Ask a room of young programmers "what is the best programming language?" and each will immediately spring to the defence of his or her favourite development tool, loudly explaining how it triumphs over all others in all situations. This will continue until the elder statesman of the office, disturbed by the cacophony of youthful conviction, swings around from his terminal, clears his throat and to the hushed awe of his colleagues he announces that the best programming language is the best one for the job. He is right of course, but in the family tree of programming languages certain branches have been more fruitful than others, and it is these I would like to discuss.

If I were to design a programming language, my starting point would be to ask myself - what are my favourite programming language features? To avoid offence, I will present this as a list of features I admire, rather than features I don't. A list of dos, not do nots, for language designers. Before this though I will summarise the two most significant problems these language designers should tackle.

Parallelisability Of my laptop's 2 CPU cores and 72 GPU cores, it is tough to max out just the CPU, let alone unlock the power of the GPU. Mutexes, Semaphores, and CUDA make it possible, but unnecessarily difficult, and in the new age of multicores threading is too important to be left to the standard library. This isn't only relevant to backend languages, no application should ever again leave the user thumping their keyboard in frustration as what could easily be a background task locks up the interface. Garbage collection was once a fringe feature, and it is now unthinkable to leave it out of a language's core. Soon it will be the same with threading support.
Typed languages with the feel of a scripting language The well-trodden path to glory for an internet entrepreneur is to write version 1 of the next great product in Python, or Ruby, or Javascript, launch it to high acclaim, instantly run into scaling problems, and use VC money to hire a small army of C++ programmers with pony tails and faded star wars T-Shirts to beef up their backend.

I've no complaint with this, but it leads to the mistaken impression that great applications are written in dynamic languages. They aren't. Great prototypes are written in dynamic languages, but the lasting replacement will be written in a dull typed language like Java/C++/C#. Facebook went as far to write HipHop to translate their many millions of lines of PHP into many more millions of lines more of C++ to avoid writing one to throw away. A language with Ruby's agility and C++'s suitability to large scale software would be a far better solution to this problem.

I love Scheme for its simplicity. It is nothing more than the minimal set of language constructs and predefined forms necessary for a language to be usable, yet somehow it achieves a terseness to rival a syntax heavyweight such as Perl. Whereas Lisp's industrial descendants - Scala and F# - arm programmers with the resuable software objects they need to build useful applications, Scheme focusses on supplying the building blocks for a programming language, and the flexibility to adapt it perfectly to your task - it is a hacker's language. So far my fondness with Scheme has coexisted with disappointment at its unsuitability for real world tasks, but the emerging R7RS standard, with its minimal core, new package system, and vast standard library looks set to provide us with a language true to the Scheme spirit but also usable for real applications.

As I mentioned above, a new language must focus on multithreading, and right now the leader in this regard is Erlang. Its creator, Joe Armstrong, realised that clever multithreading libraries treat the symptom, not the cause - shared state between threads. He wove the Actor model of concurrency into the Erlang language itself, importing techniques from functional programming to eliminate shared state and the entire class of associated threading bugs, creating what is without a doubt the sturdiest foundation for large server applications today.

My attraction to C++ is straightforward; I like native code. JIT compilers vastly increased the efficiency of the CLR and the JVM, so why not go the next step and move the native code compiler to the compilation stage? After all the "write-once run-anywhere" upside of managed code has been made redundant by web applications and javascript. Once again, this isn't uniquely a concern for back office software engineers scraping the last dregs of performance out of their big iron, nothing kills an interface like a spinning beachball, and it is a lot easier to write fast interface code in a fast language.

Much of my day to day programming is in Objective-C, and although I have grown fond of the language as a good balance between the usability of Java and the efficiency of C++ the only Objective-C feature I miss when coding in other languages is named arguments(e.g. [self drawRectOfSize:size atPoint:point inColour:colour]). Doxygen, rdoc, javadoc and similar tools do a great job of generating API documentation from source, but I prefer the self-documenting nature of Objective-C method calls. In fact I am so attached to the concept of self documenting code that I would be happy with compiler warnings for every variable and function name not composed from a valid word or phrase.

Notably I have passed over object orientation. It is a common and desirable feature, but object orientation for me serves two, often conflated, purposes. One is to encapsulate and package code so that teams of programmers in a large organisation can work together. With C++ a small group of senior coders can lay out interfaces and object models, leaving their junior counterparts to fill in the blanks. This is not unique to object oriented languages; Monads play this role in Haskell, and Erlang, another non-OOP language, has an elegant package system for this purpose.

Object orientation is also valuable as a mechanism for polymorphism. Sharing common interfaces across disparate datatypes is the foundation of code reuse today, but object orientation is just one approach to polymorphism, Haskell achieves it via typeclasses or type inference. Even C++, OOP's flagwaver-in-chief, now supports templating, which is effectively a competing compile time strategy for writing polymorphic code.

Templating is popular, but it is a poor man's version of the type inference of ML, Haskell, Scala et al. Consider the function

int abs(int val) {
      if (val < 0) return -val;
      return val;
}

Is it really necessary to type the function's return type? why not let the compiler infer its type from the type of the input? Even better, why not leave out the type of the input and let the compiler instantiate all the necessary types from the following template

abs(val) {
    if (val < 0) return -val;
    return val;
}

It is certainly a lot prettier than the C++ template version.

I have omitted a language's ecosystem from consideration - its standard library, its development tools, and the continuing evolution of the language. I have ignored these because although indescribably important to someone who wishes to use, rather than study, the language, the quality of a standard library isn't attributable to a design choice so much as it is to the corporate sponsor's deep pockets. C# and Objective-C are two examples of developing languages with great standard libraries and development tools, unsurprising considering their champions, Microsoft and Apple respectively.

The Opa project deserves an honourable mention for its efforts to end the practice of writing a web application's frontend in javascript, and the backend in a web application framework. At heart AJAX is an protocol for remote procedure calls, so why not code it as such? With Opa you write one application, designating some code to be executed on the server, and some to be executed on the backend. If this isn't enough, Opa is type inferred. It may be immature, but surely Opa is the future of web programming.

Other miscellaneous features I admire are: Python's use of whitespace to ensure that the layout of the sourcecode matches the structure of the software; Alice ML's elegant inclusion of parallel futures, an simple, but exciting approach to multithreading; Haskell's laziness - laziness is a mandatory trait for any good hacker, why not the language they code in too?

By compiling a list of features for my perfect programming language I have fallen victim to the same trap that the novice programmers did in my introduction. Instead of choosing a problem, and searching for the best solution, I am cherry-picking features from existing languages and laying the foundation for an uneasy mix of paradigms. Erlang may be an excellent example of how to multithread, but it is a better example of how a language designer has picked a problem - in this case multithreading - and pieced together a language around that need. A language designer should mimic the focus of Erlang's creators, not necessarily the features they chose to solve their problem.

Another excellent guiding principle for a language designer is a conviction that a language should shackle the programmer to good habits. The syntax of a good language compels the programmer to code "well". For example, Python enforces the use of whitespace for formatting, Objective-C ensures that all arguments to a function are documented, and Erlang guarantees that you can't write race conditions. I mentioned above that laziness is a trait of good hackers, so where possible the programming language should act as their chaperone and coerce them into good habits. This is the root of my preference for static type systems. If it can be sorted at compile time, whether it is a potential type conflict or a case of poorly laid out code, it should be sorted out at compile time.

This post was intended not so much as an article, but as an organisation of my thoughts. It is inconceivable a hacker could work with any language for a lengthy period of time and not dream about improving it, but I am increasingly convinced of the need for a multithreading-centric, statically typed, type inferred language that compiles to native code but possesses the feel, readability and programmer efficiency of Python. That said, I have had a Go tutorial on my reading list for some time now. Its Wikipedia description fits my criteria, maybe it is the one?

EDIT: corrected go! to go

Beauty tips for programmers

2012-05-02T00:00:00-07:00

The most famous written words are quotations; the author communicates in a sentence an idea that would take most of us an entire book, consequently quotations are memorised, regularly repeated, and generally held up as a pinnacle of composition. An equivalent in the programming world might be a hash table - undoubtedly a great deal of work is done behind the scenes, but a hash table exposes profound functionality with a trivial interface. Does the analogy run deeper though - the adjective beautiful is often applied to code, but does it make sense? is there an artistic merit in code beyond the efficiency and functionality of the resulting executable. Are hash tables beautiful?

The converse is clearly true; we've all seen ugly code. Poorly structured, poorly laid out, or worse spaghetti code is an eyesore as well as a nightmare for its maintainer. Well structured and elegantly formatted code is pleasing on the eye, but that doesn't qualify it to be beautifully written in the same sense as Steinbeck's novels are beautifully written. What does?

Not the programming language used, in fact the choice of programming language has no bearing on whether or not code is beautiful. Python's ordered whitespace appeals to my bauhaus sensibilities, and PERL source looks like a high level programming language disguising itself as machine code, but code is not beautiful because it is written in Haskell, and not all software written in Java is ugly. When a programmer talks about beautiful code, they are discussing the algorithm, not the syntax used to implement it.

Consideration of algorithms brings us to mathematics, another non-artistic pursuit in which elegance and beauty are frequent considerations. In A Mathematician's Apology Hardy observes that "Beauty is the first test: there is no permanent place in the world for ugly mathematics". He goes on to state that whereas a painter makes patterns with shapes, a mathematician does so with ideas. A coder must add an engineer's sensibilities to those ideas to transform them into software, but the link between the technical world and the artistic world is valid.

In Hackers and Painters, Paul Graham points out that both painting and hacking are about refinement. No piece of artwork is born in a day. X-Rays explicitly show how great painters often changed their minds during composition, and Kubrick once retook a shot 148 times before he was happy. I don't know how an artist refers to this process of refinement, but in the coding world we call it optimization.

A piece of artwork and a section of sourcecode are both built with a purpose. The hacker's goal is so concrete it is encapsulated in a unit test before work begins, whereas the artist sets out with the more ephemeral objective of capturing emotion. Nevertheless both parties proceed similarly, attacking their work as creatively as possible followed by a period of revision and refinement. Noone argues that the resulting artwork can possess beauty, so why not consider the code beautiful too?

Most likely the answer is that artists set out to inspire emotions in us, and we like emotions. The programmer who perfected an algorithm to calculate taxes had a much less glamorous task. However for me, as much a hacker's obsession with precision and performance may contrast with the artist's less definable goals, the artist doesn't have a monopoly on beauty. Hash tables are pretty too.

The Parable of the Happy Cog

2012-04-23T00:00:00-07:00

Reading Paul Graham's comment on Bronnie Ware's post about the regrets of the dying I was struck by his advice "don't be a cog". For all his great articles and advice, I can't disagree with this more. We're all cogs of one sort or another; the advice should be to "become a happy cog".

I respect the Hacker news community; it is composed of smart people with the gumption to follow Bronnie Ware's advice to live life taking risks and chasing dreams, but I find the community a little too pleased with itself for having done so. If you don't work for a boss, you work for your users, or possibly your investors. You may have an enviable role in the machine, you may even be a big cog, but you are a cog, and if you aren't a happy cog then that is a shame.

Look at this, it is the last post of a blogger about to succumb to cancer. With death approaching he has focussed on the subject most important to him - his family. It is the most coglike and genetically predetermined role a human being can play, but despite the well-lived life described elsewhere on his website, his family is the only subject worthy of his final thousand words. At the end of the post, with his words about to run out, he seems incapable of thinking of anything else, and he is driven to discard even full sentences as he scrambles to express his love for his family.

The moral I take from Bronnie Ware's article is that we have a choice between being happy and being sad. It is never presented as such, it manifests as a long string of minor choices between doing what we want to do, and what we are "supposed" to do. We should choose aspirations over convention, and we should never postpone our dreams lest we be writing our last blogpost sooner than we think.

I've never worked for anybody. I hope never to do so. I like to call myself an entrepreneur, if only of the most junior grade, and I am as addicted to building cool technologies as anyone in the Hacker News faithful. I am not criticising the lifestyle or the goals, but the dissatisfaction that drives us to create great products is a part of our lives, and without care one can peg one's happiness to the same unattainable goalposts of perfection that one should use when developing a product. Do not confuse living the life your investors want you to live with living the life you want to live.

Anyhow, I will set aside these philosophical ponderings for now because this cog has an iPad Lisp editor to perfect. Drag and drop is coming in this week's update, and Clojure will follow soon after.

In praise of Objective-C

2012-04-14T00:00:00-07:00

It is tough to write anything about Apple without being accused of fanboism, however I really do admire Objective-C. Originally created at Stepstone it has become synonymous with Apple's platforms, and I believe it leads the pack of general purpose programming languages. This is not something I feel with the fervour of a Lisper defending his or her parentheses, but Objective-C occupies a market leading middleground between the performance centric C++ and the programmer focussed Java.

Java's early press focussed on its portability, "Write once, run anywhere" was the promise, and it seemed enough to make Java the programming language of the internet. It wasn't sufficient; javascript "won", but Java lived on as a general purpose language. Java's designers had realised that programmer time was more valuable than CPU time and they endowed their new language with a vast standard library and runtime features such as garbage collection. This approach stuck, and languages are now judged as much by their package manager, integrated documentation and ecosystem as they are by the syntactic choices of the designers.

Unfortunately, Java's designers didn't seem to value CPU time at all. The language has a nasty reputation for sluggish interfaces, and its execution speed drags well behind C++'s. Pointer aliasing or not we are many generations of optimisers away from languages such as Java overtaking C++ so if you need fast code C/C++ is the obvious choice. As a strict superset of C you can add Objective-C to that list.

Objective-C's triumph is that although it shares C++'s efficiency, Cocoa, its standard library, rivals and sometimes exceeds the libraries available for .Net or the JVM platforms. Objective-C saves you from making the choice between the high-performance C++ or the programmer-friendly Java by adding a substantial runtime to a language that compiles to native code.

iOS has had the benefit of superb hardware, but as Apple learned to their cost in their fight with Microsoft for the 80s desktop market, a platform lives and dies by the software that runs on it. The App Store made it easy for developers to sell software, but it was Objective-C/Cocoa that ensured these developers could write high quality software just as easily as they could sell it. Apple's choice to provide a great programming language, standard library and lets not forget one of the best IDEs around in Xcode, was a big part of iOS's success.

Sadly Objective-C is only available on Apple platforms. I am not a fanboi and I would love to work with Cocoa on Ubuntu or Android. GNUStep is good, but its frustratingly far from completion and Apple would be insane to release Cocoa on other platforms, and give them the leg up it gave iOS. If Mark Shuttleworth wishes his Ubuntu to become the next OS X, he would be wise to take a leaf out of the late Steve Jobs's book and ensure that his system is as beautiful on the inside as it is becoming on the outside.

EDIT. In the first paragraph of the original version of this post, I erroneously stated that Objective-C was designed at NextStep.

Lisping 1.0.1

2012-04-14T00:00:00-07:00

Lisping 1.0.1 is now available as an update in the App Store. It fixes a unicode issue and adds support for inline comments and the retina display.

I'm putting the final touches to Lisping 1.0.2 now. Aside from internal refactoring and bug fixes the new features are:

Configurable display I have added a settings option to allow you to change the colorscheme and font size. Further options will be added to this in future release
Transformers There are editing operations for which Lisping is currently a little cumbersome, for example enclosing a selection in parentheses, or stripping parentheses from around a selection. Following requests I have added a transformers menu with add and strip parentheses options. Requests for other useful transformers would be gratefully received, so as always, get in touch.
Viewer mode Although the ringed placeholder elements are fundamental to Lisping's editing experience, I have had comments that they get in the way when browsing code. With this in mind I have added a viewer mode without placeholders in which code is selectable and executable, but uneditable.

Why I prefer Scheme to Haskell

2012-04-09T00:00:00-07:00

Before you sharpen you keyboards and spit acid at me as punishment for the title, let me point out that I haven't written this post to bash Haskell (no pun intended), in fact I love it. Haskell introduced me to algebraic data types, domain specific languages, pattern matching, and many other programming constructs that feel like lost limbs in other languages, but Haskell willfully resists any attempt I make to write useful code with it. Scheme lacks almost all of Haskell's innovations in favour of a minimalistic flexibility, but it mixes a practicality with the functional beauty that makes it Haskell for human beings.

Last weekend I began writing a prototype for a data mining application. I started in Haskell; it is fast, elegant, and once you've used the Parsec module it is tough to imagine writing a parser without it. My first step was to extract data from a straightforward XML file. This is a simple task with ruby: download an xml parser, read a tutorial, and maybe ten minutes later you have the data in memory ready to work with. After a frustrutating hour with my Haskell version I was still struggling to understand the monadic API required to work with the XML parser and I gave up. This has always been my problem with Haskell; it is beautiful, but it is useless for hacking. Scheme doesn't have Haskell's vast standard library or an efficient compiler, but it does have Haskell's terseness, and more importantly it makes coding a joy.

As elegant and appealing as Haskell's purely functional foundation is, it prohibits simple, but crucial, impure tasks such as writing to files and communicating over networks. Monads - Haskell's answer to this deficiency - are little more than a wobbly crutch; they enable coders to write useful software in Haskell, but they make it neither intuitive nor easy.

If you wish to debug a function in Haskell you can't insert a printf to inspect its inner workings unless that function happens to be on the IO monad. This is a serious barrier to Haskell's usability that is rarely discussed, because you can only fix what you see, and if you can't inspect the inside of a function, you can't fix it. Scheme on the other hand sacrifices a little purity so that you can slap a (display ) in your code and find out what is going on. I'm sure a Haskell programmer right now has his or her head in his or her hands whilst questioning how I could miss the inappropriateness of this behaviour in a lazy, well optimized language with a debugger. They would be right, but the inability to debug like this is a price I am not willing to pay for Haskell's academic purity.

The second problem with Monads is related to their great strength - they are synonymous with Domain Specific Languages. The mission statement for domain specific languages is stupendous - don't write a complex program to solve your problem, write a simple program in a programming language you have designed solely for that task. I've already mentioned the best use of DSL/Monads - Haskell's parsec module. With Parsec the Haskell function to parse a file is identical to the Backus Naur Form description of the parse grammar - how much clearer could it be? They say that imitation is the highest form of flattery, and every parser function I have written outside of Haskell since meeting Parsec has been a shoddy facsimile of Parsec in the chosen language. The success of Parsec and its ilk has filled Hackage (the Haskell module repository) with hundreds of DSLs covering any task you care to mention.

Yes, literally hundreds of them. Hundreds of little programming languages, one for BNF parsing, one for parsing xml, one for creating PDFs, all perfectly suited to their task. Each is different, and each has its own learning curve. Consider a common task such as parsing an XML file, mutating it according to some JSON you pulled out of a web API and then writing to a PDF. In Ruby or a similar object oriented language you expect to find three APIs/gems, all with a similar object oriented syntax, but for three Haskell DSLs designed for three different tasks to share syntax implies that their authors failed to optimise them for those tasks, hence instead of five minutes with API documentation you have hours of DSL tutorials ahead of you before you can begin work. It is this subtle difference that plants Haskell on the academic side of the town and gown divide. With Haskell the focus is on the language, not the problems you are solving with the language.

Scheme is intentionally simple, in fact I don't imagine it would take a competent programmer more than an hour to become comfortable with the syntax, but it is simple in a manner that makes it flexible, not inhibiting. Take for example template metaprogramming. Although templating is less useful in dynamic languages, Lisp hackers have had access to templates for years through the macro system, but Haskell, like C++, required a language extension to make it available.

Scheme's historical role as the language of innovation is a big part of why I chose it for my iPad development environment Lisping. When reinventing coding on touchscreens by allowing the user to edit the syntax tree, it seemed fitting to choose a language associated with so many new technologies over its history.

It would be dishonest not to point out that despite it's utility to hackers, Scheme shares Haskell's unsuitability for production code. The difference is that Scheme is limited by its minimalistic standard library, not by a flaw in the language, and you can "upgrade" to a syntactically similar, but heavyweight, cousin such as Common Lisp, or Clojure, to get work done.

I'm not saying Scheme is better than Haskell. Programming languages are tools, and one should always choose the best tool for a job, but I do think that Haskell's obtuseness makes it a poor choice for hackers. Learn all you can from its functional beauty, and campaign for its best features to be added to your language of choice, but that is all Haskell will ever be to me, a platonic ideal of a programming language, lighting the way to the future, but unusable in itself.

EDIT: thanks to those who pointed out the existence of Debug.Trace, but I was trying to express my frustration at not being able to insert a little impure code temporarily to tweak behaviour and help with debugging.

Serbo-Croatian translation by Jovana Milutinovich from Webhostinggeeks.com.

Where is Lisping headed?

2012-04-07T00:00:00-07:00

Lisping is in the App Store, initial sales and feedback are extremely encouraging, and the first update to fix an issue with unicode, add support for the retina display, and improve inline comments is on its way through the approval process. This begs the question - what comes next? where is Lisping headed?

One of my first priorities is support for other Lisps, specifically Clojure. There is no hope of executing Clojure on the iPad as it is dependant on the iOS-prohibited Java runtime, but Lisping will provide an unbeatable way to edit Clojure on the go. Potentially the clojurescript project or even a cloud offering will eventually allow for execution as well.

Lisping's appearance will become configurable in the next few updates. I intend to add a preferences plugin, because as fond as I am of Solarized dark, it is not to everyone's taste. Also, font and size need to be configurable.

Soon after that I will experiment with iPhone/iPod Touch devices and make Lisping a universal application. Lisping's parse tree editor will be even more valuable on the smaller screens where keyboard interaction is even less practical.

Stepping back a bit, Slide To Code is heading towards a Python development environment. As I perfect Lisping's editing mechanics, the code is being refactored ready to start on a Python version. There are many problems with editing python in this way, but it is a great language, and it will be a perfect fit for Lisping's parse tree editor.

Edit Scheme on your iPad via the Parse Tree - Lisping launched

2012-04-05T00:00:00-07:00

Lisping has finally made it into the App Store. For those who don't know, Lisping is a Scheme development environment designed around the concept of editing the parse tree, an approach chosen because of its suitability to the iPad's touchscreen interface.

Anyone who has tried to edit code on the iPad through a traditional textview knows that it doesn't work well. Editing source code character by character is a concept wedded to the keyboard and it is inappropriate for the iPad, a device with no keyboard. Lisping abandons this model and allows you to edit your code via the parse tree. Rather than manipulating ranges of characters Lisping focusses on selecting, creating and moving syntax elements, a task ideally suited to the iPad's touchscreen interface, and also - more than a little bit fun.

With a syntax identical to its parse tree, Scheme is an ideal language to be edited in this way. Additionally its flexibility and terse syntax make it the perfect language for coding on the go. Lisping's built-in parser and parse-tree based layout engine take care of some of many of Scheme's more tedious aspects such as closing parentheses. With Lisping you simply cannot write code with an unbalanced number of parentheses. The REPL too just became easier, if you want to see what a chunk of code does, it is as simple as touching it and pressing execute. A popover will appear with the results. See for youself in this demo.

Lisping is a fully capable Scheme Development environment for the iPad, but as such a radical departure from the traditional code editing model it is also an experiment. If you have any comments, suggestions, or complaints please get in touch with me at slidetocode@gmail.com.

Lisping submitted to the App Store

2012-03-25T00:00:00-07:00

Lisping headed off to the iOS App Store reviewers today. I can't say exactly how long it will be until it emerges in the store, but I hope it will be ready for sale this time next week at the latest.

This is a good moment to say thank you to all my beta testers. A piece of software as experimental as Lisping is impossible without guidance from user feedback, and the advice from my beta testers has been invaluable.

Development doesn't stop with release of course, and I'm keen to hear feedback, positive or negative as I continue to work on Lisping and touchscreen coding.

TouchScheme is dead, long live Lisping

2012-03-19T00:00:00-07:00

I very recently announced TouchScheme, my iPad scheme editor and interpreter. The response has been fantastic, and something it has taught me is that there is a need out there for an iOS Lisp editor more general than just a Scheme editor. After release my first major upgrade will be to allow for editing of Clojure and Common Lisp in addition to Scheme. In anticipation of this day, I am renaming TouchScheme to Lisping before release, because I can't do it afterwards.

Meanwhile, thanks to all who have volunteered to Beta Test, I hope to distribute the first beta today.

Coding reinvented on the iPad, edit the parse tree

2012-03-17T00:00:00-07:00

I learnt two important lessons during an idyllic holiday in Thailand over new year: Firstly I should relax more often, and secondly I should be able to code on my iPad when I am done relaxing. My resolve to relax more evaporated as soon as my fingers touched my keyboard back home, but the realisation that my iPad will never be a computer in my eyes until I can comfortably code on it didn't leave me. The accepted wisdom is that the iPad is a device for content consumption, not content creation, but I want my iPad to be more than an apparatus for donating more money to Apple, and I returned from Thailand on a mission to be able to code on my iPad. I am writing here to reveal my solution.

I discovered in Thailand that the available interpreters were unusable. They use iOS's inbuilt text editing elements, which work great for emails where you type a document from beginning to end with only minimal alterations, but for coding tasks where frequent and disparate minor alterations are expected they are hopeless. If we are to code effectively on our tablet devices, we need editors designed from the start for touchscreen interaction, not editors that are attempting to recreate the familiar keyboard experience on the iPad.

My solution, as you can see in the video above, is to edit the parse tree. Editing source code character by character is a concept wedded to the keyboard and after discarding it, my iPad source code editor came together fast. Rather than manipulating ranges of characters I have built an editor solely focussed on selecting, creating and moving syntax elements.

The first choice for a language to be edited by its parse tree can only be a member of the Lisp family, languages whose syntaxes are identical to their parse tree. Although later I will extend the concept to more mainstream languages, in this project Scheme has played its traditional role of a language perfect for experimentation and development.

TouchScheme is nearing its release day. A product like TouchScheme is necessarily an experiment, and development will continue apace after release. I'm hoping for feedback to guide me while I do this. Feel free to get in contact with me at slidetocode@gmail.com.