Yes, that's exactly the trade-off I've been thinking of. When I program I like to start out writing tests at the start when I don't yet know what properties I should try to enforce. Over time I start finding places where tests aren't good enough, and start feeling the need for more rigorous checking. But by then I've painted myself into a corner with my choice of language, and it's too hard to switch to a more rigorous platform.
In Mu my plan to break this dichotomy between coverage and soundness is to allow arbitrary metadata to be attached to instruction operands. As a result you aren't stuck with the type annotations and checking that I build into the system. You can have arbitrary metadata and will be able to write programs that can deduce and check this metadata to enforce all sorts of properties. Arbitrary checkers feels to me like a generalization of Lisp macros.
Yes, I wanted to say something similar in response to your phrasing that "type systems (including dynamic and static type systems) are another way to carve out the territory". When you say "dynamic type systems", in particular, I think Python and Ruby, and the modularity guarantees there seem much inferior to tests. But Haskell and the more advanced type systems might well disprove my thesis.
Typeclasses are amazing. They're similar to multimethods, but more powerful, and they have the same performance as regular functions.
You can use them to solve the expression problem, giving you a huge amount of flexibility, power, and conciseness. Or you can use them to create dynamically scoped variables which actually work correctly (unlike dynamic variables in most languages).
The thing is, typeclasses are only possible with static types. They're the main reason I changed Nulan to be statically typed.
----
By the way, I would like to point out that the static type systems in languages like Haskell is very different from the static type systems in other languages like C, C++, or Java. So even if you like/dislike one of them, you might end up liking the other. I used to lump all of the static type systems together, but I think that's a mistake.
There are bugs which are not caught by C, C++, or Java, but which are caught by Haskell. And there are programs which do not type check in C, C++, or Java, but which do type check in Haskell.
That sounds about right. Static types can only guarantee a certain subset of behavior. Unit tests work for a huge amount of behavior, because they run at run-time. They are very versatile. But there are still some things that unit tests cannot test.
In particular, unit tests cannot guarantee the absence of behavior. As an example, Haskell can use its static type system to say, "this function (and any functions it uses) is guaranteed to be pure". Or you can say, "this function (and any functions it uses) is guaranteed to only use certain side effects, but other side effects do not happen".
Languages like Idris go even further and can guarantee that a function terminates (never goes into an infinite loop)!
As another example, Haskell has Software Transactional Memory, and it uses the static type system to guarantee that arbitrary I/O cannot occur, which is important because the transaction might be retried multiple times.
I don't remember which language it was in, but I remember reading about a language that used static types to guarantee that exceptions are always handled. In other words, you will never have an uncaught exception.
Nulan uses its static type system to make a distinction between blocking impure functions and non-blocking impure functions. In JavaScript, there's a big difference between blocking and non-blocking, both in terms of behavior and performance. By using the static type system, it's possible to say "this function (and any functions it uses) is always blocking, and never non-blocking".
As you are aware, Rust uses its static type system to guarantee that pointers are always used correctly, and that data races cannot occur, even with concurrency.
But the biggest reason I like static types is because they make refactoring easier. Let's say you have a type that represents a person. So you have a name property, age property, etc.
Now let's say you want to add in a new property. Some of the existing functions will need to change to accommodate this new property. Without static types, you need to carefully go through your entire code base and find the functions which use the type, and then change their behavior to work with the new property.
Unit tests won't help you with that, because the unit tests are all testing the old properties, not the new property. So in the end, you have no choice but to just bite the bullet and spend a significant amount of time searching through the code and making the changes.
But with static types, it's as simple as adding the new property, compiling, fixing any type errors, compiling, fixing any type errors, etc.
Each type error tells you the file and line where the error occurs, making it easy to fix. You don't need to search through your code to find the places which need fixing: the type system does that for you. And after all the type errors are fixed, it's basically guaranteed that your code is correct.
Another example: a function used to always return a result, but now you need to change it so that it can potentially fail. So anybody that uses that function now has to account for the possibility of failure. With a static type system, you simply make the change and fix the type errors.
With unit tests, you better hope that you have 100% code coverage, because you're going to need it to find all of the functions which need to be changed. And don't forget that the unit tests need to be updated, and you probably need to add in new tests as well. That takes time, in addition to the time spent fixing the code.
So the end result is that it's faster and easier to refactor with a static type system, and it's more likely to be correct as well.
So I think static types and unit tests complement each other: they can both do things that the other cannot. Static types can make guarantees that unit tests cannot, and static types also remove some of the burden from unit tests, which speeds up development without sacrificing correctness.
On the other hand, unit tests can make certain guarantees that static types cannot. So in the end, I think you need both.
P.S. Static typing also helps tremendously with things like smart IDEs. I think Unison is a really cool example of that:
Yes, you're right. I'm not very familiar with Haskell, but I was thinking about it when I called the title "An alternative.."
I think of Haskell as being to tests what up-front design is to Agile. It's great if you're trying to carve out right and wrong behaviors along the dimensions it's been designed to help with (and my vague sense is that linear types and other such advances are trying to expand that set of dimensions). But how would you ensure that your program meets its performance guarantees? Or is guaranteed not to block between these two memory barriers? Tests are less rigorous and more work, but if you're willing to put in the elbow grease they'll work for anything you throw at them.
I would like to point out that type systems (including dynamic and static type systems) are another way to carve out the territory.
Especially in languages like Haskell, static types are used all over the place to define intentions and boundaries. Languages like Idris and Agda go even further, with dependent types and proof systems.
Unit tests are great, but there are many useful properties that unit tests cannot test, but static type systems can test. And vice versa: there are things that unit tests can test that static types cannot.
Also, things like QuickCheck[1] allow us to encode our intentions in very concise ways (much more concise than traditional unit testing).
Rather than saying this:
(reverse (reverse [])) is the same as []
(reverse (reverse [1])) is the same as [1]
(reverse (reverse [1 2])) is the same as [1 2]
(reverse (reverse [1 2 3])) is the same as [1 2 3]
(reverse (reverse [3 2 1])) is the same as [3 2 1]
(reverse (reverse [1 2 3 4 5])) is the same as [1 2 3 4 5]
...
You instead say this:
for every list, (reverse (reverse list)) is the same as list
And it will then generate the unit tests for you.
So, in general I agree with you. I just disagree with the idea that unit testing is the only way we have to specify our intentions. There are many ways, and we should use every useful tool which is available to us.
I think that the more we can specify our intentions (in a testable way), the better our programs will be.
13 types + 5 special forms + 1 global variable + 92 built-in functions.
And that's not including things such as first-class functions, lexical scope, first-class continuations, tail call elimination, or macros. I don't even know how to count those.
The fact is that practical programming languages need a lot of axioms. If you look at the list of built-in functions, they are all useful, and most of them are for I/O, which cannot be written in an axiomatic way.
I think it would be interesting to write an "eval" function in Arc, and see how few functions it needs in order to accomplish it. But I don't see much practical benefit to it.
This starts out as a pretty great article. By the end, I think the author's enthusiasm belies their goal to avoid baseless Lisp flattery.
Why does the article say it's nice to have lists-as-syntax, but then say Racket's syntax transformers are even better?
Why does the article say it's nice that X-expressions reveal "the sequential nature of the data elements" of a document, but then say it's nice that Scribble lets you embed Racket expressions at arbitrary places in your document?
IMO, all these things are nice options to have at the same time, and maybe that's what the author thinks too. It would be nice to hear more concrete reasons for preferring these features though, especially since concrete reasons are what the article is trying to offer.
---
"The problem with Lisp flattery is that it makes sense only to experienced Lisp programmers. To others—especially those who are trying to decide whether to learn and use a Lisp—it just comes across as unsubstantiated hoodoo."
This is true, but certain kinds of experience bring the hoodoo feeling back. When I see the elegance of computational trinitarianism, it makes me want to use algebraic data types instead of s-expressions to represent program structure. Some programming languages are absolutely in the axiomatic style, even in the sense that they're specified fully as a half-page diagram with lots of Greek letters all over. Coming back to a Lisp, everything's messy again, and it makes it really hard to swallow the hype.
I'd say there are some specific benefits that Lisps have to offer, but people easily confuse these benefits with other accidental properties of the languages.
Most of the historic advantages of Lisp have already been plundered by other languages. What does "Lisp" even mean now that garbage collection, first-class anonymous functions, interactive debuggers, run time code loading, and package managers are available for basically every popular language?
Does "Lisp" just mean macros? Python, Groovy, Idris, Agda, Haskell, OCaml, etc. each have some variation of macros or AST manipulation, but does that make them Lisp dialects? What about cases like JavaScript and Ruby, which have extensive libraries for parsing and manipulating their own code but have no support built into the language?
Angular.js watches for changes to your first-class environments and uses them to re-macroexpand the DOM tree. That's pretty Lispy!
You might say that the difference isn't technical but cultural. Well, Java "managed to drag a lot of [C++ programmers] about halfway to Lisp." JavaScript was explicitly "Scheme in the browser" until it was dragged about halfway to Java. I think there's a case to be made that, culturally, these languages were Lisps doing the best they could.
Do we call them Lisp dialects? I don't think we do. I think we only call something a Lisp dialect if its syntax is structured as lists of lists and if it's not a Forth dialect.
So what are the concrete benefits of lists as syntax? I use them for specific reasons:
- If I were stranded on a new platform or otherwise compelled to reinvent a good syntax from a limited set of tools, it would be pretty easy to start off with parens. This is due to the sequentiality of text as a code representation format. Parens let us conceptually break up a piece of text into a bunch of text-with-holes-in-it pieces joined together. In other words, trees of interpolated strings. Lists are an extreme case of interpolated strings where there is no string left over; it's just some abstract sequential format consisting entirely of holes. I tried using interpolated strings as syntax, and the string-parsing operations made macroexpansion too inefficient, so I settled on lists as a consolation prize.
- When every syntax is a list, it doesn't matter if the syntax is standard or custom. Every syntax looks just as good or bad as every other. This might help people adopt new, innovative operators they might have scoffed at in another language. Racket is an example of a project that cultivates a variety of innovative syntaxes to coexist all at the same time, such as Typed Racket, web server's stateless servlets, and the legacy r5rs, r6rs, and mzscheme languages.
Those are the advantages of parens to me. I'm currently extrapolating on the same principles in my designs for comprehensive s-expression quasiquotation and string quasiquotation. Adding quasiquotation to s-expressions is a lot like adding parens to flat text. :)
I mostly agree with this about how lisp advocacy can be misleading. My one quibble is with his deemphasis of macros. I think they're only #8 on his list because he's been using Racket. A more fluent system for macros makes them far more broadly applicable.
I absolutely agree that "everything is an expression" deserves to be #1. Rust is the first non-functional language to steal this idea: http://lucumr.pocoo.org/2012/10/18/such-a-little-thing. That probably explains Rust's relative success with macros as well. My big insight doing Wart was to realize that "homoiconicity" was really nothing more than "everything is an expression". There's nothing special about lists in particular besides that.
I think the "enlightenment" happened when the only other language people had programmed in was at the level of Fortran or C. Here's another quote in response:
"There have been two really clean, consistent models of programming so far: the C model and the Lisp model. These two seem points of high ground, with swampy lowlands between them. As computers have grown more powerful, the new languages being developed have been moving steadily toward the Lisp model. A popular recipe for new programming languages in the past 20 years has been to take the C model of computing and add to it, piecemeal, parts taken from the Lisp model, like runtime typing and garbage collection." (http://www.paulgraham.com/rootsoflisp.html)
At this point it's only macros that separate Lisp from other languages (though languages like Rust have been trying to steal that as well). So don't go looking for enlightenment. Everybody's trajectory is different.
Starting Anarki at the command line hasn't changed if you're using the "arc" script. If you're invoking boot.scm yourself (e.g. if you're on Windows like me), the command has changed a bit.
Before:
$ racket -f boot.scm -e "(tl)"
After:
$ racket -t boot.scm -e "(tl)"
Unfortunately, the Racket package builder wants to compile every .scm file it finds in the file tree, and it expects all of those to have a #lang or a (module ...) form surrounding the whole file. I had to change boot.scm to make that work. I tried several ways to make "-f boot.scm" work, but "-t boot.scm" was the closest I could get.
"The new language construct is much more lightweight than any code we might have written purely at the application level, even with Lisp macros."
The new language construct...
-(def editor ((o u the.me))
+(def editor ((t u me))
As language changes go, this is only slightly better than the change of using a short name.
Like, every time you type "fn" you could have typed "lambda", and that is indeed a certain kind of inconvenience.
In this case, every time you type "(t" you could have typed "(o" and "the." instead. So it has the slight benefits of a shorter name, and it has the added benefit that it makes you move your cursor less. (On the other hand, if you're refactoring an existing use of "(o", it makes you move your cursor more.)
This is the kind of thing I consider macro-expressible, rather than a language innovation. Libraries should be able to provide this. And in fact they can; my Lathe libraries have customizable pattern matching support already.
Yeah, Arc macros can't update the built-in pattern-matching syntax this way. My library does it by implementing its own special version of (fn ...) and such. For Hacker News in particular, it might have been easier to add (t ...) to the language rather than to overhaul the design of pattern-matching throughout the codebase to be extensible.
---
"You still have to declare which variables you plan to use in each function signature, so it's possible to look at the function and know exactly what state it depends on."
That's not even true. A call to (user-name ...) still depends on the state of the.me, and that's no longer visible in the signature.
It worries me that static knowledge would be used as a justification for this technique when the knowledge isn't even sound. That's where we get arbitrary language restrictions that have no real benefit.
I hope this static knowledge is meaningful in some weaker way that just wasn't quite articulated here.
---
To the author: I'm sorry I'm railing on this. I appreciate the information about how Arc's doing.
I think Arc is perfect for what you want - it was designed from the ground-up for "exploratory programming". Arc not only preserves its Lisp roots "under the hood", it also captures the spirit of Lisp, IMO. It clears away what I consider to be syntactical pollution in, say, Common Lisp.
Lisp has a bad reputation for being difficult; the most difficult thing about Lisp is the concept of quotation (and quasi-quotation). At the REPL (http://tryarc.org/):
> (list 1 (+ 1 1) 3)
(1 2 3)
This list contains a sub-list - (+ 1 1) - which is itself evaluated to give the final list that Arc returns.
> (list 1 '(+ 1 1) 3)
(1 (+ 1 1) 3)
Putting a single-quote in front of the sub-list causes the interpreter to not evaluate that list but, instead, to return it in quoted form, exactly as-is.
> '(list 1 (+ 1 1) 3)
(list 1 (+ 1 1) 3)
We can quote the whole list if we like, in which case, the whole list is returned as-is. Quotation is "blocking", that is, once the interpreter hits a single-quote, it stops evaluating the quoted list and any sub-lists within it.
> `(list 1 (+ 1 1) 3)
(list 1 (+ 1 1) 3)
The back-tick is the quasi-quote operator and has a similar effect as quote when applied to a whole list. However, it acts differently when we apply the unquote operator (comma) to sub-lists:
> `(list 1 ,(+ 1 1) 3)
(list 1 2 3)
Here, the whole list was quoted but the interpreter did not stop traversing the list, searching for unquoted lists. When found, the unquoted list was evaluated.
Keeping track of exactly when/where you need to use quotation or quasi-quotation can sometimes be challenging. Everything else about Lisp is completely straightforward, that is, no more difficult than any other, more common, language.
