This probably isn't what you mean, but...

One of my oldest toy languages (Jisp) had syntactic abstractions I called "micros," and I discuss them here: http://arclanguage.org/item?id=10719

tl;dr: My micros are fexprs that not only leave their arguments unevaluated but also leave them unparsed. The input to a micro is a string.

Much like how I just said macroexpansion in the compilation phase was like fexpr evaluation, what I've pursued with Penknife and Chops is like a compilation phase based on micro evaluation.

---

"Why not have both? As in, have a way of saying "this should all be done at compile-time" that doesn't involve fexprs at all, or involves a special variant of fexprs."

One way to have both is to have two fexpr evaluation phases, one of which we call the compile phase. This is staged programming straightforwardly applied to an fexpr language... and it's as easy as wrapping every top-level expression in (eval ... (current-environment)).

However, that means explicitly building all the code. If you want to call foo at the repl, you can't just say (foo a b c), you have to say (list foo a b c).

With quasiquote it's much easier for the code that builds the code to look readable. So suppose the REPL automatically wraps all your code in (eval `... (current-environment)). Entering (foo a b c) will do the expected thing, and we can say (foo a ,(bar q) c) if we want (bar q) to evaluate at compile time.

Now let's fall into an abyss. Say the REPL automatically detects the number of unquote levels we use in a command, and for each level, it wraps our code in (eval `... (current-environment)) to balance it. Now (foo a b c) will do the expected thing because it's wrapped 0 times, (foo a ,(bar q) c) will do the expected thing because it's wrapped once, and so on. We have as many compile phases as we need.

Possible correction: If any kind of 'quasiquote is ever going to be in the language, it should probably have special treatment so that its own unquotes nest properly with the REPL's meaning of unquote. An alternative is to use a different syntax for non-REPL unquotes (e.g. ~ instead of ,).

Yes, I currently think that all syntax should be at the reader level, rather than trying to use macros to define syntax. As an example of what I'm talking about, in Nu, [a b c] expands into (square-brackets a b c) letting you easily change the meaning of [...] by redefining the square-brackets macro.

Or the fact that 'foo expands into (quote foo) letting you change the meaning of the quote operator... or the fact that `(foo ,bar) expands into (quasiquote (foo (unquote bar))), etc.

I used to think that was great: hey look I can easily change the meaning of the square bracket syntax! But now I think it's bad. I have both conceptual and practical reasons for thinking this.

---

I'll start with the conceptual problems. In Lisp, there's essentially three major "phases": read-time, compile-time, and run-time. At read-time Lisp will take a stream of characters and convert it into a data structure (often a cons cell or symbol), compile-time is where macros live, and run-time is where eval happens.

Okay, so, when people try to treat macros as the same as functions, it causes problems because they operate at different phase levels, and I think the same exact thing happens when you try to mix read-time and compile-time phases.

---

To discuss those problems, let's talk about practicality. quasiquote in particular is egregiously bad, so I'll be focusing primarily on it, though quasisyntax also suffers from the exact same problems. Consider this:

  `(,foo . ,bar)

  (quasiquote ((unquote foo) . (unquote bar)))

  (quasiquote ((unquote foo) unquote bar))

---

Another problem with this approach is that you're hardcoding symbols, which is inherently unhygienic and creates inconsistent situations that can trip up programmers. Consider this:

  `(,foo (unquote ,bar))

---

Yet another problem is that it requires you to memorize all the hard-coded names for all the syntax. You have to remember to never define a function/macro called quote. To never define a function/macro called unquote, to never define a function/macro called square-brackets, etc... which means this will break:

  ; oops, redefined the meaning of the quote syntax
  (let quote ...
    'foo)

---

Also, this whole "read syntax expands into macros" thing is also inconsistent with other syntax. For instance, (1 2 3) isn't expanded by the reader into (list 1 2 3). That is, if you redefine the list function, the meaning of the syntax (1 2 3) doesn't change. But if you redefine the quote macro, then suddenly the syntax 'foo is different.

The same goes for strings. Arc doesn't expand "foo" into (string #\f #\o #\o) either. So redefining the string function doesn't change the meaning of the string syntax. So why are we doing this for only some syntax but not others?

---

All of the above problems go away completely when you just realize that read-time is a separate phase from compile-time. So if you want to change the meaning of the syntax 'foo the solution isn't to redefine the quote macro. The solution is to use a facility designed for dealing with syntax (such as reader macros).

This is just like how we separate compile-time from run-time: you use functions to define run-time stuff, macros to define compile-time stuff, and reader macros to define read-time stuff.

Uh oh, my warning bells went off. If I were you, I'd put some unit tests that verify that closures work properly. In particular, this might very well break in bullet (though I won't know without testing it):

  (def foo (x)
    (fn () x))

  ((foo 4)) -> 4

"They are expanded at runtime, and the result of the expansion is then evaluated in the lexical environment of the call site. That means you can use macros in higher-order functions; they truly are first class."

Ewww, runtime macros. I do not like. They combine all the awfulness of macros[1] without any of the benefits of fexprs[1], while also giving up the only benefit macros have[1]. The worst of all worlds, in my opinion.

---

"My intention had been to double back and fix the discrepancy with "real" lisps by doing the standard initial macroexpand traversal of the AST before evaluating."

Good. I think Lisps should either embrace macros (warts and non-first-classness included), or embrace fexprs and dump macros since they're not needed and just get in the way. Naturally, I'm in favor of fexprs unless speed is critical, and even then I'd prefer to just make the interpreter faster rather than dump the elegance of fexprs.

---

"I've just got to try these fexpr style macros; the idea of just controlling evaluation of operands, but otherwise being just like a regular function is very appealing."

It sure is! An example of a very beautiful Lisp that uses fexprs at its very core is Kernel (though it calls them operatives and uses the $vau form to create them):

http://web.cs.wpi.edu/~jshutt/kernel.html

http://www.wpi.edu/Pubs/ETD/Available/etd-090110-124904/unre...

ftp://ftp.cs.wpi.edu/pub/techreports/pdf/05-07.pdf

There are other Lisps that use fexprs (or at least things similar to fexprs) as well, such as Picolisp and newLISP (which erroneously calls them macros), but I'm especially fond of Kernel (for many reasons), but in part due to its static (lexical) scope.

---

* [1]: I'm only slightly exaggerating... but in all seriousness, first-classness is only one of the (multiple) benefits of fexprs, and even with first-class macros, you still need to worry about hygiene, which is basically a non-issue in Kernel (that is to say, in Kernel, hygiene is so incredibly easy to achieve that it naturally happens, because the language is so incredibly well designed, so I consider this a mostly "solved problem" in Kernel).

Plus, I suspect if you're basically macro-expanding macros at runtime, you'd actually get slightly faster speed with fexprs (not that speed is a huge issue, but it can be an issue, depending on what you want to do, so I mention it for completeness and because I have a personal interest in making powerful things go fast).

As far as I can tell, the only real benefit of macros is that they're always preprocessed, so they only need to macro-expand once. That is also why they're non-first-class.

I suppose a minor benefit is it allows you to treat macros as basically a template facility, but I find that benefit to be dubious at best, especially since it's so easy to use templating facilities in fexpr (or define your own).

Another minor benefit is that you can macro-expand a macro to do things like code walkers, but... I feel that should be part of a debugger/inspection suite or something.

---

It works:

    func foo (x): fn () x
    print ((foo 4)) // prints 4

---

That's right. The point of protect is to do something even if an exception is raised (or continuations are used). It's used for things like closing files. It's analogous to try ... finally in other languages[1], which don't let you grab the exception either.

---

"but I do believe that there should be some way to retrieve the original exception"

If you want try ... catch, then you should use on-err, which you do in the example above. Is there some use case where on-err doesn't work but protect might?

---

* [1]: To clarify:

  JavaScript
    try {
        ...
    } catch (e) {
        ...
    }

  Arc 3.1
    (on-err (fn (e) ...)
            (fn ()  ...))

  JavaScript
    try {
        ...
    } finally {
        ...
    }

  Arc 3.1
    (protect (fn () ...)
             (fn () ...))

Looking at the above, assuming you didn't care about Arc 3.1 compatibility, it would be possible to merge the two together, into a single operator:

  (protect (fn ()  ...)
           (fn (e) ...))

The catch is that this reflective kind of tool may not be very efficient... but fortunately, it doesn't have to be the lowest-level tool. In fact, with 'dynamic-wind around, it might not even need to be an axiom:

  ; TODO: Test this code. It's big and confusing and therefore almost
  ; certainly buggy.
  
  ; We're using '$ (as seen in Anarki) to get things from Racket.
  (= raise $.raise)
  (= dynamic-wind $.dynamic-wind)
  (mac b-and-a (bef . rest)
    (let (aft . body) rev.rest
      `(dynamic-wind (fn () ,bef) (fn () ,@body) (fn () ,aft))))
  (mac before (bef . body)
    `(b-and-a ,bef ,@body nil))
  
  ; The "refl" is for "reflective."
  (def refl-dynamic-wind (bef body aft)
    (with (resume-unknown-ccc-exit nil normalentry t)
      (before
        (let normal normalentry
          (wipe resume-unknown-ccc-exit normalentry)
          (let entry (bef:if normal '(enter) '(unknown-ccc))
            (case entry.0
              enter        (unless normal
                             (err "Can't enter normally"))
              raise        (raise entry.1)
              unknown-ccc  (when normal
                             (err "Can't enter using unknown-ccc"))
              known-ccc    (entry.1)
                (err:+ "Unknown entry " (tostring:write entry)))
        (let exit (aft:catch:let normalexit nil
                    (b-and-a
                      wipe.normalexit
                      (do1 (on-err [do `(raise ,_)]
                                   (fn () `(return ,(body))))
                           (= normalexit t))
                      (unless normalexit
                        (when (ccc [= resume-unknown-ccc-exit _])
                          (throw `(unknown-ccc))))))
           (case exit.0
             return       exit.1
             raise        (raise exit.1)
             unknown-ccc  (if resume-unknown-ccc-exit
                            resume-unknown-ccc-exit.nil
                            (err "Can't exit using unknown-ccc"))
             known-ccc    (exit.1)
               (err:+ "Unknown exit " (tostring:write exit))))))
  
  (mac refl-b-and-a (bvar bef . rest)
    (let (aft avar . body) rev.rest
      `(refl-dynamic-wind
         (fn (,bvar) ,bef) (fn () ,@body) (fn (,avar) ,aft))))))
  (mac refl-before (bvar bef . body)
    `(refl-b-and-a ,bvar ,bef ,@body nil nil))
  
  ; NOTE: Arc's 'after uses all sub-s-expressions but the first as the
  : catcher, while this uses only the last two.
  (mac refl-after rest
    (let (aft avar . body) rev.rest
      `(refl-b-and-a nil nil ,@body ,avar ,aft)))
  
  
  ; Example usage:
  
  (def reimplemented-on-err (catcher body)
    (refl-after (body)
      exit (case exit.0 raise
             `(return ,(catcher exit.1))
             exit))
  
  (def reimplemented-protect (body aft)
    (refl-after (body)
      exit (do (aft) exit)))
  
  (def reimplemented-dynamic-wind (bef body aft)
    (refl-b-and-a
      entry (do (bef) entry)
      (body)
      exit (do (aft) exit)))
  
  (def reimplemented-raise (error)
    (refl-after nil
      exit `(raise ,error)))

JavaScript's catch and finally aren't so different either:

  function myFinally1( body, aft ) {
      try { body(); }
      finally { aft(); }
  }
  
  function myFinally2( body, aft ) {
      try { body(); }
      catch ( e ) {
          aft();
          throw e;
      }
      aft();
  }

I realized afterwards that you can define protect in terms of on-err, but that isn't quite the same as using the same operator for both use cases. And, uh, that code looks awfully intimidating, so please forgive me for not looking deeply into it and trying to understand it.

Nah. While that's a straightforward analogy from catch and finally, 'protect deals with continuation early exits (not seen in JavaScript), whereas 'on-err isn't supposed to. Jarc's 'on-err did for a while, but I seem to remember that being fixed.

---

"And, uh, that code looks awfully intimidating, so please forgive me for not looking deeply into it and trying to understand it."

Fair enough. ^_^

Just in case anyone wants to use that monstrous code, feel free.

---

"In any case, my question still hasn't been answered: is there any use case for this at all?"

Inasmuch as a language designer is trying to expose as much as possible to the programmer, even things they don't expect to be useful, that's not a question that needs an answer. The real question is "Why not?"

But dido spelled out at least one goal: "I do believe that there should be some way to retrieve the original exception, rather than the last one raised by protect handlers"

I'm guessing dido's point is that a buggy protect handler shouldn't keep the programmer from discovering bugs under its protection. (It's being overprotective? :-p ) For instance, dido might like to see both errors at the command line.

---

For you and anyone else who didn't read the 'refl-dynamic-wind code, here's an outline of the guards' inputs and outputs, so you can get a sense of the flexibility it gives to the programmer:

  ; Possible inputs to the entry guard:
  (enter)
  (unknown-ccc)
  
  ; Possible results of the entry guard:
  (enter)          ; only if (enter) was the input
  (raise <error>)
  (unknown-ccc)    ; only if (unknown-ccc) was the input
  (known-ccc <destination continuation>)
  
  ; Possible inputs to the exit guard:
  (return <value>)
  (raise <error>)
  (unknown-ccc)
  
  ; Possible results of the exit guard:
  (return <value>)
  (raise <error>)
  (unknown-ccc)
  (known-ccc <destination continuation>)

Come to think of it, it's probably possible to keep hacking on 'refl-dynamic-wind until the interface is like this:

  ; Possible inputs to the entry guard:
  (enter)
  (continue <destination>)
  
  ; Possible inputs to the exit guard:
  (return <value>)
  (raise <error>)
  (continue <destination>)
  
  ; Possible results of either guard:
  (enter)
  (return <value>)
  (raise <error>)
  (continue <destination>)

For instance, in Kernel, continuation guards have access to both the destination continuation and the value being passed to it. If we implemented 'refl-dynamic-wind there, every one of the (continue <destination>) cases above could become (continue <destination> <value>).

* redef/remac/extend

* defcall

* `var` for accessing global variables (I will probably change the name)

* implicit parameters

* utilities for inspecting/manipulating paths, such as cwd, dirname, basename, etc.

* an `import` macro for easily loading files

There are also some libraries in the lib/ subdirectory, and applications I've written in the app/ subdirectory.

---

Why yet another branch on GitHub? I have three major projects right now:

1. The Nu compiler, which is used for Lite Nu and Arc/Nu. It's intended to be very compatible with Arc 3.1 while fixing bugs and making it possible to add additional features.

2. Arubic, which is my language based loosely on Arc.

3. arc2js which intends to allow you to execute Arc/Arubic code in the browser.

To put it another way, the three projects are essentially: Arc 3.1, Arubic, and JS. Arc/Nu is a part of project #1, but I got tired of implementing Arubic and arc2js, so I decided to add some convenience libraries on top of Lite Nu, which is a lot easier to do.

Extended Arc series: x-nu, x/nu ax-nu, ax/nu xa-nu, xa/nu xarc-nu, xarc/nu arcx-nu, arcx/nu

     x-nu     x/nu
  
    xa-nu    xa/nu
    ax-nu    ax/nu
    a+-nu    a+/nu
  
  xarc-nu  xarc/nu
  arcx-nu  arcx/nu
  arc+-nu  arc+/nu

  (apply wrap.and list.x)

  (wrap.and @list.x)

See, that's what I'm talking about. It seems to me that apply shouldn't eval its arguments. Instead, a function would be a fexpr that would eval all its arguments before doing anything else. Something like this:

  (= fn (fexpr (parms . body) env
          (eval (w/uniq (args env)
                  `(fexpr ,args ,env
                     (let ,parms (map (fexpr (x) nil (eval x ,env)) ,args)
                       ,@body)))
                env)))

Also, it might be a good idea to make a more general "wrap" function that would turn any fexpr into a function:

  (= wrap (fexpr (x) env
            (let x (eval x env)
              (fexpr args env
                (let args (map (fexpr (x) nil (eval x env)) args)
                  (apply x args env))))))

  (= fn (fexpr (parms . body) env
          (wrap (eval `(fexpr ,parms nil ,@body) env))))

I think in Kernel, a call to 'apply invokes an applicative with a list of argument values, not expressions it should evaluate. So an applicative isn't exactly an fexpr that calls 'eval on its args, because it doesn't actually eval its args in every instance it's called.

Instead, the arguments to an applicative are evaluated as part of the Kernel core semantics. When applying a value to a list, if the value is an applicative, it's unwrapped, and the value inside is applied to the result of mapping eval over the list.

It could use the original definition of those, if that were wanted.

This is actually consistent with Kernel's semantics, since it's not possible to overwrite a built-in variable, you can only shadow it. And because Kernel uses lexical scoping, it should use the binding in the original environment. So you'd get this behavior for free.

Of course, we're talking about wart here, so wart can do whatever the hell it wants to do. I merely mention this to point out that (as far as I know) that's not a problem in Kernel, which also has fexprs. So it shouldn't be a problem in wart either, assuming its implementation of lexical/dynamic variables is solid.

---

"calls 'apply with an fexpr that does something with its args other than immediately evaluating them (e.g. most "macros")."

That's why I brought up the case. If a language is designed for a sufficiently smart programmer (as is the case with Arc and Wart), the language designer don't have to worry about ensuring things like "eval always does X"; the programmer will leave these invariants alone if they're really that useful, and will break them if breaking them is even more useful. In fact, I think Arc sacrifices hygiene in favor of implementation simplicity in order to be more intuitive when broken.

Intuitive even after you break it. That could be an interesting take on the Arc philosophy. ^_^

But yeah, Wart could still do this in a way where all the abstraction leaks behave exactly as akkartik expects them to (whether that's hygienically or not).

---

"That was kinda akkartik's point: that apply doesn't work properly on macros."

Right, I meant to allude to a bit more, which I was about to elaborate on. I'll just elaborate here, then. :)

If 'apply wraps up its args as literal expressions, then (apply and foo) does work as everyone keeps trying to say it should work. :-p That is, it works just like (all idfn foo), so I don't see why anyone would even care to use it.

Meanwhile, (apply def foo), which to me doesn't make sense in the first place, may also "work," successfully giving an error since a literal expression isn't an assignable identifier.

I consider these to be inelegant consequences of the implementation approach, rather than useful behavior. But if akkartik sees it differently, I'd like to make sure the other inelegant parts (the potential lack of hygiene) don't come as a surprise. ^_^

But why would apply need to map quote on the argument list? If it simply doesn't evaluate the individual sub-expressions, then it should be the same thing. My current view for an ideal Lisp is that all evaluation should be done explicitly, in fexprs. No implicit eval by apply or the runtime or whatever.

That means that apply would pass the arguments unevaluated to the fexpr, which could then choose to eval them or not. And since a function is defined as a fexpr that evals all its operands first, the semantics are preserved for functions too.

Basically, I think fexprs should be the default (using eval when needed), rather than functions being the default (using quote to suppress evaluation). In other words, not evaluating should be the default.

---

"Meanwhile, (apply def foo), which to me doesn't make sense in the first place"

Why not? It seems to me that...

  (apply def '(foo (a b c) (+ a b c)))

  (def foo (a b c) (+ a b c))

  (def all-true(xs) (and @xs))

  (def all-true(xs) (apply and xs))

  (apply and '((1 2)))

  (and (1 2))

If 'apply literal-wrapped the args, it would work more like this:

  (and '(1 2))

"Meanwhile, with your proposal, the use of 'apply with functions would break: (apply list '((1 2))) would behave like (list (1 2))."

---

"But why would apply need to map quote on the argument list? If it simply doesn't evaluate the individual sub-expressions, then it should be the same thing."

Since you (Pauan) would have procedures be fexprs that explicitly eval'd their arguments, using apply on a procedure would cause the procedure to eval the args. Whereas (apply list '((1 2))) results in the list ((1 2)) in Arc, in this hypothetical language, 'list itself would try to eval (1 2) and get an error.

In other words, the Scheme/Arc/Kernel 'apply takes a list of argument values, not expressions, so (from a certain point of view) they're not actually suitable arguments for an fexpr call.

If we want Scheme/Arc/Kernel-like behavior, 'apply can make up for the procedure's own evals by quoting the args, or it can make up for them the way it does in Kernel, by unwrapping an applicative into something that skips the eval-my-args step altogether.

Personally, I think unwrapping is cleaner, but quoting makes (apply and foo) behave like (all idfn foo), which is what people seem to expect. Unfortunately, neither of these approaches does what you want with 'def.

---

"It seems to me that (apply def '(foo (a b c) (+ a b c))) should be exactly equivalent to (def foo (a b c) (+ a b c))."

Pauan, the step-by-step way you comprehended my response to this is great, but I wasn't reasoning in that much detail. :-p I was mostly going by analogy:

  (apply def '(foo (a b c) (+ a b c)))
  (def foo (a b c) (+ a b c))
  *** success ***
  
  (apply list '(foo (a b c) (+ a b c)))
  (list foo (a b c) (+ a b c))
  *** error: unbound variable 'foo ***
  
  (apply list '((1 2)))
  (list (1 2))
  *** error: can't call 1 ***
  
  (all-true (list '(1 2)))
  (apply and '((1 2)))
  (and (1 2))
  *** error: can't call 1 ***

I agree. I rather like that part of Kernel.

---

"Unfortunately, neither of these approaches does what you want with 'def."

For example your analogy for def sounds exactly like what Pauan is saying. What am I missing?

"Meanwhile, with your proposal, the use of 'apply with functions would break: (apply list '((1 2))) would behave like (list (1 2))."

The only area I currently disagree with rocketnia is this: "Unfortunately, neither of these approaches does what you want with 'def."

---

"For example your analogy for def sounds exactly like what Pauan is saying. What am I missing?"

Yes, it is exactly like what I was saying. The point was that if you changed apply so it worked with def (and other fexprs) it would then break functions like list. Finding a way so that apply can work on both fexprs and functions is one of the points of unwrapping functions.

---

"But that's how it should behave, right? Is that not what Pauan is suggesting as well?"

Absolutely not. It should behave like (list '(1 2)). That is how it would behave if apply unwrapped functions, or if apply mapped quote over the arguments.

Of course, one could make the argument that you should have to say (apply list '('(1 2))) instead... but in that case I'd rather have two separate operators: one that works on fexprs, and one that works on functions.

> Absolutely not. It should behave like (list '(1 2)).

sbcl:

  * (apply 'list '((1 2)))
  ((1 2))
  * (list '(1 2))
  ((1 2))

  > (apply list '((1 2)))
  '((1 2))
  > (list '(1 2))
  '((1 2))

Yes, that is the expected result.

---

"Are you saying this should work even if list was a macro/fexpr?"

Yes. But as soon as you try to make it work with fexprs you'll run into the following problem. Here's what I wrote earlier:

"To put it another way, (apply list '((1 2))) would be equivalent to (eval (cons list '((1 2)))) which is equivalent to (eval (list list '(1 2))) which is equivalent to (eval '(list (1 2))) which is equivalent to (list (1 2))."

What I'm saying is that if you changed apply so it worked with fexprs, then (apply list '((1 2))) would be equivalent to (eval '(list (1 2))). If you try evaluating that, you'll get an error saying something like "1 isn't a valid function call" or whatever.

That's because functions evaluate their arguments. So when the "list" function evaluates the argument "(1 2)" it tries to call the "1" function with the argument "2" which obviously fails.

So instead you have to call (eval '(list '(1 2))) to suppress evaluation. Note the extra quote! So what rocketnia said (and I now agree with) is that the "apply" function would have to map quote over each of its arguments. In other words, (apply list '((1 2))) would basically be the same as (eval '(list '(1 2)))

Alternatively, if functions were "wrapped" fexprs, then apply could "unwrap" the function, which would lead to more-or-less the same effect, sans a few details.

In other words, you'd either define "apply" like this...

  (def apply (f args env)
    (eval (cons f (map (fn (x) (list quote x)) args)) env))

  (def apply (f args env)
    (eval (cons (unwrap f) args) env))

This should work with both fexprs and functions in a (hopefully) intuitive way.

---

Oh, yeah, you might also decide to keep apply, but have it only work on functions. Then you'd use eval to "apply" fexprs. After looking at the issue, I'm not really opposed to that. It seems reasonable, provided that supplying arguments as a list is most common with functions. The lack of consistency does bug me a little, though.

---

* [1]: If Kernel already uses that definition, why does apply supposedly only work with functions? I believe (without actually checking) it's because the unwrap function throws an error if the argument isn't a function. So you can't call unwrap on fexprs.

"you might also decide to keep apply, but have it only work on functions. Then you'd use eval to "apply" fexprs. After looking at the issue, I'm not really opposed to that."

You know, I was tempted to define all-true as a macro:

  mac all-true(xs)
    `(and ,@xs)

  (apply foo '(a b c)) ->
  (foo a b c)

  (def apply (f args)
    (eval (cons f args)))

  (apply def '(foo (a b c) (+ a b c))) ->
  (eval (cons def '(foo (a b c) (+ a b c)))) ->
  (eval (list def 'foo '(a b c) '(+ a b c))) ->
  (eval `(,def foo (a b c) (+ a b c))) ->
  (def foo (a b c) (+ a b c))

  (apply list '(1 (2 3))) ->
  (eval (cons list '(1 (2 3)))) ->
  (eval (list list '1 '(2 3))) ->
  (eval `(,list 1 (2 3))) ->
  (list 1 (2 3))

  (apply foo '(a b c)) ->
  (foo 'a 'b 'c)

Let's redefine apply so it maps quote over the arguments:

  (def apply (f args)
    (eval (cons f (map (fn (x) (list quote x)) args))))

  (apply list '(1 (2 3))) ->
  (eval (cons list '('1 '(2 3)))) ->
  (eval (list list ''1 ''(2 3))) ->
  (eval `(,list '1 '(2 3))) ->
  (list '1 '(2 3))

  (apply def '(foo (a b c) (+ a b c))) ->
  (eval (cons def '('foo '(a b c) '(+ a b c)))) ->
  (eval (list def ''foo ''(a b c) ''(+ a b c))) ->
  (eval `(,def 'foo '(a b c) '(+ a b c))) ->
  (def 'foo '(a b c) '(+ a b c))

Before we can define apply in Kernel, we need to understand something. This is one of the central tenets of Kernel, and it's contrary to essentially every other Lisp, including wart.

Almost every Lisp uses something called "implicit evaluation", that is... there is this assumption that things are automatically evaluated, and you need to use quote to suppress this automatic evaluation.

Kernel is different, however. In Kernel, argument evaluation doesn't happen automatically. You have to make it happen by calling eval. This is called "explicit evaluation". In fact, Kernel embraces this so far that it doesn't even provide quote or quasiquote[1].

So in Kernel, all combiners (functions/fexprs) are actually fexprs underneath. All of them. And those fexprs then have to explicitly eval their arguments if they want them evaluated. So, conceptually, a function is just a fexpr that happens to evaluate all of its arguments before doing anything else. apply and eval don't evaluate the arguments, the functions do[2].

Well, except for one thing. In Kernel, functions are actually a separate type from fexprs. So Kernel provides three base primitives for dealing with combiners: $vau, wrap, and unwrap:

  $vau constructs fexprs

  wrap takes a combiner (function/fexpr) and returns a function that evaluates
  all of its arguments before passing them to the combiner

  unwrap takes a function and returns the underlying combiner (function/fexpr)

  (def apply (f args)
    (eval (cons (unwrap f) args)))

  ((unwrap foo) a b c) ->
  (foo 'a 'b 'c)

In most cases, the unwrapped value will be a fexpr. That is, (unwrap list) returns a fexpr that is exactly like the list function, except it doesn't evaluate its arguments[3].

That's why the above equivalence is true: (unwrap foo) returns a combiner that doesn't evaluate its arguments[3], whereas foo is a combiner that does. So unwrapping foo equates to not evaluating the arguments, which is equivalent to mapping quote over the arguments.

Going back to the list function:

  (apply list '(1 (2 3))) ->
  (eval (cons (unwrap list) '(1 (2 3)))) ->
  (eval (list (unwrap list) '1 '(2 3))) ->
  (eval `(,(unwrap list) 1 (2 3))) ->
  ((unwrap list) 1 (2 3)) ->
  (list '1 '(2 3))

  (apply (wrap def) '(foo (a b c) (+ a b c)))

  (unwrap (wrap f)) is eq to f

But in wart, let's assume that you want apply to work directly on fexprs, without needing to wrap them first. In that case, wart can say that:

  (unwrap f) is eq to f, when f is a fexpr

  (apply def '(foo (a b c) (+ a b c))) ->
  (eval (cons (unwrap def) '(foo (a b c) (+ a b c)))) ->
  (eval (list (unwrap def) 'foo '(a b c) '(+ a b c))) ->
  (eval `(,(unwrap def) foo (a b c) (+ a b c))) ->
  (def foo (a b c) (+ a b c))

---

* [1]: It is trivial to define quote as a fexpr, but Kernel doesn't do that by default, and it intentionally encourages you not to do that.

* [2]: Of course, this doesn't mean that it needs to be implemented that way... you could have wrap return a specially tagged value, and then hardcode eval so when it sees that value in the car, it unwraps it and evaluates the arguments.

* [3]: Actually, because it's legal to call wrap on a function (which returns a function that evaluates its arguments twice), unwrapping it may return a function rather than a fexpr. I'm assuming for the sake of this post that that situation is quite rare, and that unwrap always returns fexprs.

* [4]: The "Revised-1 Report on the Kernel Programming Language" actually doesn't say this. But it's mentioned in John Shutt's dissertation, "Fexprs as the basis of Lisp function application".

If we went with a system that had partial wrappers, we'd run across the foreboding corner case of a wrapper that had no effect at all. We might wrap anything in this wrapper an arbitrary number of times and not notice until we start peeling off layers again. But that actually brings us a kind of consistency: We can decree as part of the language design that everything eventually unwraps into an infinite regress of boring wrappers. This gives us a great excuse for 'apply to work on fexprs.

  (fn ('a b c . 'd) ...)
  ==>
  (wrap '(t nil nil . t)
    (fexpr (a b c . d) ...))

I don't think I'm grasping this nearly as well as you and Pauan. Watch me flail around here (and a few parent commits as well): http://github.com/akkartik/wart/commit/71d9962fbf

---

Will wrap/unwrap play well with macros? Macros are fexprs whose output is implicitly eval'd in the env of the caller. Won't that extra eval throw a spanner in the works?

---

Here are two test examples we've been using a lot in this thread (phrased in wart terms):

  and @(list '(1 2 3))

  def @'(foo(a) a)

I'm assuming macros in wart are just extra sugar for fexprs, and in that case it shouldn't cause any problems if eval doesn't evaluate the arguments of fexprs (which it obviously shouldn't).

To clarify, let's go through the process one step at a time. I'm going to be using this macro:

  (mac let (var expr . body)
    `((fn (,var) ,@body) ,expr))

  (fexpr (var expr . body) env
    (eval `((fn (,var) ,@body) ,expr) env))

  (apply let '(a 5 a)) ->
  (eval (cons (unwrap let) '(a 5 a))) ->
  (eval (list (unwrap let) 'a '5 'a)) ->
  (eval `(,(unwrap let) a 5 a)) ->
  ((unwrap let) a 5 a) ->
  (let a 5 a)

  (eval '((fn (a) a) 5) ...)

I don't understand where that environment is coming from in the fexpr.

  (let x 3 (mac foo (a) ...))

  (let x 4 (foo x))

Why would you do that, and how does wart handle macros, then?

---

"What would env be when the call to mac is translated to an fexpr in your scheme?"

It would be the environment where the macro is called. That is, in the dynamic binding of the macro call, x would be 4.

Using the environment of the place where the macro is defined would be static scope... which I assume wart already has, so that's redundant and pointless because you can just do this to evaluate the expression in the static scope of the fexpr:

  (let env (current-environment)
    (fexpr () nil
      (eval ... env)))

  def unwrap(f)
    ret newf copy.f
      zap quotify rep.newf!sig

  def quotify(sig)
    (if
      !sig
        nil
      ~cons?.sig
        (cons quote sig)
      quote?:car.sig
        sig
      :else
        (cons quotify:car.sig
              quotify:cdr.sig))

Of course this doesn't work on:

  unwrap.and @(list '(1 2 3))

When I was talking about unwrap, I was assuming that all functions are created with wrap, so all of them can be easily unwrapped, rather than unwrap generating a new fexpr. In other words, I was expecting this:

  (unwrap (wrap f)) is eq to f

"Since unwrap doesn't change and at all.."

---

Wart already has eq if you want it:

  iso addr.a addr.b

I wouldn't consider "I don't have eq, so do without it." to be a good reason anyway. So the reason my unwrap copies its argument is that it's not at all obvious to me that it matters in a lisp with eq.

---

When you repeatedly say 'hacky' you shed more heat than light[1]. I'm not sure what the flaw is that you see, and whether it's the same flaws that make me embarrassed about it.

For example, the fact that it generates a new copy each time, I'm not sure that's a bad thing.

The implementation relies on the representation of functions in wart, but that's because it's the implementation[2], and implementations can be changed. The conception doesn't seem compromised to my imperfect understanding: unwrap.f is just like f except that evaluation of args is disabled. My concern is really: is this implementation behaving as unwrap should? Are there corner cases where behavior diverges from the desired?

[1] I personally find eq to be quite 'hacky' in the way it requires knowing about how atoms are interned. But again, that's neither here nor there.

From a practical standpoint[1], unwrap is used in apply, which means you end up copying a function and running your quotify function over it every single time you use apply. My method should be a lot faster[2], with no additional cost in flexibility or power. It should also be much shorter and easier to implement as well. In fact, I'll define wrap and unwrap right now, using Arc syntax[3]:

  (def must-combiner (x)
    (if (in type.x 'fn 'fexpr)
        x
        (err "expected combiner but got" x)))

  (def wrap (x)
    (annotate 'fn must-combiner.x))

  (def unwrap (x)
    (rep must-combiner.x))

---

* [1]: I still partially disagree with you even on theoretical grounds. I think that defining (unwrap (wrap f)) to be eq to f is a good equivalence that could prove useful in practice, or at least make it easier to reason about programs because it's so simple and intuitive (based on the names "wrap" and "unwrap").

However, if you've defined iso so it works on functions (that is, functions with the same arguments and body are equal), then I'm fine with that and don't care so much about eq, since iso is the default in wart.

* [2]: Especially since wart is written in C++, so you should be able to make things like the fn wrapper very fast (to type check and extract the fexpr) and memory efficient.

* [3]: I'm assuming that annotating something with the same type will wrap it a second time, which Arc doesn't currently do.

If I cared about a superior or more efficient implementation I'd go use racket or kernel. I build things to better understand them.

My purpose in showing my unwrap was to show that I still don't understand your suggested implementation of apply. Not to get bogged down in considerations of performance. This is wart, where everything is in slow-mo. It can take a little more consing :)

Perhaps I'm getting stuck in a local optimum here. Now that I've slummed it out with an implicit-env eval for a while, perhaps I'll appreciate the benefits of an explicit env far faster if I just switch to it.

Anyways, thanks for the hand-holding. I should figure out the rest on my own.

---

And oh, I didn't show wrap because it's impossible in a language that allows some params to be eval'd (http://arclanguage.org/item?id=15813). There's no nice way to guarantee that:

  iso f wrap:unwrap.x

Only a little. Although I think ignoring performance is an overall good thing (premature optimization and all that), I think it's still important to choose the best option (in terms of benefit-cost) that's currently available. If somebody gives you an implementation that's better in every way with no cost, why would you refuse it...?

---

"I still don't understand your suggested implementation of apply."

A non-phase-separated macro implemented as a Kernel-style fexpr is simply of the form ($vau <args> env (eval <body> env)), possibly with red tape so that env isn't in scope during the body.

I don't have answers to the other questions....

Exactly. Macros are SUPER trivial to add on top of fexprs, but the reverse is not true. In fact, I'll go ahead and define $mac right now:

  ($define! $mac
    ($vau (name parms . body) env
      (let ((u (gensym)))
        (eval (list $define! name
                (list $vau parms u
                  (list eval (list $sequence body) u)))
              env))))

And now you can use it to write macros in Kernel (I don't think that's a good idea, though[1]...):

  ($mac $let (var expr . body)
    (list (list* $lambda (list var) body) expr))

  ($mac $let (var expr . body)
    `((,$lambda (,var) ,@body) ,expr))

* [1]: In Kernel, you sometimes find fexprs of the form ($vau ... env (eval ... env)) which is basically what the above $mac form does. To avoid the extra redundancy (boilerplate) of the call to eval, you might want to define a $mac form, but it should be anonymous, as in ($mac (a b c) ...):

  ($define! $mac
    ($vau (parms . body) env
      (let ((u (gensym)))
        (eval (list $vau parms u
                (list eval (list $sequence body) u))
              env))))

  ($define! $foo ($mac ...))
  ($set! env $foo ($mac ...))
  ($let (($foo ($mac ...))) ...)

http://lambda-the-ultimate.org/node/3640#comment-51665

All the differences they discuss revolve around phase separation and the need to perform macroexpansion at call sites. So there's implicitly no other reason it can't be done.

Thanks a lot, guys! I have some rewriting to do in wart :)

---

tl;dr: quote encourages you to think in terms of implicit eval, quasiquote makes it way too easy to break hygiene accidentally, and macros are indirect and differ significantly from functions, thus making certain programs harder to write/understand/debug/etc.

---

Like I said earlier, the trick to understanding it is to think in terms of fexprs evaling things, rather than using quote to suppress evaluation. The simplest way to do that is to just remove quote entirely from the language, thus deeply encouraging you to think in terms of fexprs.

Another problem is that if you encourage quasiquote, then people are more likely to use it to create symbols in fexprs, but that then introduces all the hygiene problems of Arc's macro system. Let me give an example. Let's suppose that you defined $mac in Kernel, and also quasiquote. Let's define an Arc-style $let fexpr:

  ($mac $let (var expr . body)
    `(($lambda (,var) ,@body) ,expr))

  ($let $lambda 5
    ($let ...))

  ($define! $let
    ($vau (var expr . body) env
      (eval (list (list* $lambda (list var) body) expr)
            env)))

"Dangerous computation behaviors (e.g., hygiene violations), while permitted on general principle, should be difficult to program by accident"

As for macros... their problem is that trying to specify an expression that when evaluated will give the desired result is usually more complicated, more difficult to understand, and thus more error-prone because it's indirect. For instance, compare these two definitions of $or?:

  ;; implicit eval
  ($define! $or?
    ($mac (x y)
      ($let ((temp (gensym)))
        `(,$let ((,temp ,x))
           (,$if ,temp ,temp ,y)))))

  ;; explicit eval
  ($define! $or?
    ($vau (x y) env
      ($let ((temp (eval x env)))
        ($if temp temp (eval y env)))))

---

* [1]: But if you ever do want $quote, it's super easy to define:

  ($define! $quote
    ($vau (x) #ignore x))

* [2]: Arubic has a "hygienic" quasiquote called quasisyntax (borrowing from Racket's terminology):

  (mac let (var expr . body)
    #`((fn (var) . body) expr))

  (mac let (var expr . body)
    `((,fn (,var) ,@body) ,expr))

  (mac do1 args
    (w/uniq g
      #`(let g ,(car args)
          ,@(cdr args)
          g)))

A misguideline. :-p It's my least favorite part of Kernel's philosophy. I'd rather say desirable things should be easy to do by accident.

Pretty much everywhere Kernel uses G3, I'd rather see another justification. For instance, if Scheme has a "dangerous" feature people commonly use as though it did something slightly different, and Kernel replaces it with something else that's harder to make that mistake with, I might be swayed by an orthogonality argument.

By the way, Kernel replaces gensyms with "keyed static variables." As it happens, the R-1RK does justify this with an appeal to orthogonality!

What I'm saying is that $quasiquote in specific is bad. But I'm perfectly fine with something like my $quasisyntax operator, which is hygienic.

The problem with $quasiquote is that $quote is the default[1], so in order to be hygienic (which is what you want most of the time), you have to $unquote everything[2].

My $quasisyntax[3] operator, on the other hand, $unquote's things by default while still letting you easily break hygiene when you want to.

In other words...

  #`(foo (bar qux) corge)

  (list foo (list bar qux) corge)

Now, you might be sitting there saying, "but I want to break hygiene!" and I agree, it does need to be easy to break hygiene when you want to, just not to break it by accident.

So what do I need to change to allow for hygiene violations? Actually, I don't need to change anything. The above operator already supports hygiene breaking, without needing to bake it in, unlike $quasiquote which requires $quote to be built-in.

Let me give an example. Consider this definition of $afn, which is almost hygienic except for the deliberately inserted symbol 'self:

  ($define! $afn
    ($mac (parms . body)
      `(,$rfn self ,parms ,@body)))

  ($define! $afn
    ($mac (parms . body)
      #`($rfn ,'self parms . body)))

So you can define a $quasisyntax operator in Kernel that allows you to use $quote'd (unhygienic) values, without $quasisyntax actually needing $quote at all! In that case it just serves as syntactic sugar for cons/list.

But, if you decided to add in a $quote operator, you can immediately use it with $quasisyntax to break hygiene, without needing to change/redefine $quasisyntax.

See how easy that is? So, with $quasisyntax, hygiene is the default, but you can still super-easily break hygiene, just by using ,' to insert the quoted symbol.

This is what I was talking about earlier, about making it hard to do dangerous things by accident while still making it possible (and hopefully easy) to do it on purpose. Whether John agrees with me on this particular point or not, I don't know, but that's how I see it.

---

* [1]: This is logical in a Lisp that uses $quote to suppress evaluation, but Kernel uses explicit-eval and doesn't even define $quote by default; so $quasiquote is inconsistent with the very core of Kernel itself. On the other hand, as far as I can tell, the definition of $quasisyntax is perfectly compatible and consistent with Kernel in every way, though I concede that Kernel isn't particularly interested in things like syntactic shortcuts.

* [2]: You might look at the above comparison of $afn and think, "pfft that's not a big deal" and you'd be right. The differences in the definition of $afn are small.

But in even slightly more complicated macros, the need to $unquote everything becomes a bigger issue, especially since it's very easy to forget to $unquote something (I've done this myself with $quasiquote, at least a few times, and I'm usually a very careful person; I usually double and even triple check my code).

It also looks ugly seeing the , character spewed out everywhere, therefore encouraging you to not use it, therefore encouraging you to break hygiene (by default) rather than breaking hygiene consciously, when you want to.

* [3]: I, personally, would probably call it something other than $quasisyntax, and I would probably use ` or @ rather than #` for brevity and because it looks nicer.

* [4]: Incidentally, this solves some issues I had had with $quasisyntax in Arubic... consider this old Arubic definition of afn:

  (mac afn (parms . body)
    #`(rfn 'self parms . body))

  #`(annotate ''mac ...)

  `(annotate 'mac ...)

I don't agree that the original quasiquote should be difficult.

It's pretty much a matter of political spin. There's no need to work to make quasiquote difficult, but there's little need to work to make it easy either, since that effort would be better spent on quasisyntax. We can just neglect to give quasiquote any sugar, at the same time as we spoil quasisyntax with sugar of its own. We can also neglect to define quasiquote in the main language distribution altogether.

---

"I, personally, would probably call it something other than $quasisyntax"

tl;dr: I agree with you overall, so perhaps we're just using different terminology or wording, resulting in confusion.

Right, I'm not saying that users shouldn't be allowed to define $quasiquote if they want to, but given its problems and the fact that $quasisyntax is better overall, there's no need to provide $quasiquote by default in the language. And if there were a better alternative ($quasisyntax) in the language, then users would be encouraged to use it instead of defining $quasiquote themselves.

Even though Kernel goes out of its way to not provide $quote, it's trivial to add it in (literally just 1-2 lines of super-simple code). And it should only take ~20 lines or so to define $quasiquote, the same as it does in Arc. The only difference is Kernel doesn't have syntactic sugar for $quote, $quasiquote, or $quasisyntax.

In particular, I disagree with this statement of yours: "I already understood the benefit of your quasisyntax operator. I agree that it makes desirable things easy to do by accident."

As far as I can tell, quasisyntax does not make it easy to do "desirable things" (like breaking hygiene) by accident. It does make it easy to do them on purpose.

I think our primary disagreement is that we're using different meanings for the word "accident". It seems to me that when I say "it should be hard to do dangerous things by accident" you take that to mean "I, the language designer, shall deem what shall be dangerous, and you, the user, shall not be allowed to do those things without jumping through crazy hoops!!"

---

But that's not what I'm saying. When I'm talking about "on accident", I'm referring to the user, not the language designer. Let me give an example. Let's suppose you had an operator in your language for deleting files.

To look at a real-world example, let's use the Unix command "rm". If you want to remove the file /foo/bar you would say this:

  rm /foo/bar

  rm / foo/bar

Thankfully that's not what actually happens (at least on my computer): rm doesn't remove directories without specifying the -r option. So for removing files you say `rm foo` and for removing folders you say `rm -r foo`. Of course, that doesn't help if you accidentally type `rm -r / foo/bar`...

That's an example of making dangerous things difficult to do by accident, while still making it easy to do them on purpose when you want to.

---

So when I say that dangerous things should be difficult to do by accident, this is really just a different way of saying that "computers should do exactly what we want them to do, no more and no less".

Alternatively, it could also just be a rephrasing of the principle of least surprise, that computers should do what people expect them to do, rather than going off and doing crazy things that the user didn't want.

You don't want the computer to go berserk and do things you didn't intend for it to do. But in the (rare) case that you do want to, say, remove your entire harddrive, you should still be able to do so, and ideally it should be easy as well. Just not easy by accident.

---

My quasisyntax operator makes it super-easy to break hygiene when you want to, but it doesn't make it easy to do it by accident. You have to manually and explicitly add ,' to a symbol to make it unhygienic. Contrast that with quasiquote where it's easy to forget to unquote something, therefore quasiquote has the same power as quasisyntax, but makes it easy to break hygiene by accident.

Essentially the difference between quasiquote and quasisyntax is that quasiquote makes it easy to make mistakes, but quasisyntax does exactly what you would expect it to do, without making mistakes, while still making it really easy to break hygiene when you want to.

---

For a more detached example, consider someone who implements the same algorithm in Arc and Haskell. Their Haskell version uses call-by-need when their Arc version uses call-by-value, and at first they may even understand and disregard this discrepancy. Later they discover that one of these is detrimental--an undesirable accident. I consider the other one to be a desirable accident.

To me, this isn't just about whether the computer does what the user wants/expects it to do. It's about whether the computer does what the user doesn't know they want it to do. IMO, programming is all about specifying what one wants in really specific terms, even down to details one isn't truly sure about yet.

---

"It seems to me that when I say "it should be hard to do dangerous things by accident" you take that to mean "I, the language designer, shall deem what shall be dangerous, and you, the user, shall not be allowed to do those things without jumping through crazy hoops!!""

That's pretty much right. :) You have a good point that "by accident" is a key phrase, and your example of an accident is a nice thing to make difficult.

However, I still take any use of G3 in Kernel with a grain of salt, in case it turns out to be accidentally used in an "it should be hard to do things" way instead. There are many places where it's used for legitimate clarity of program behavior, but I'm not so convinced by these uses:

"Distinguishing long identifiers by case alone is error-prone, hence contrary to G3."

"One compromise is to G3 of §0.1.2. Naive traversal algorithms are dangerous; and, inevitably, the more readily available naive traversal algorithms become, the more readily the programmer can use them by accident."

"Most algorithms, however, are intended by the programmer to be im- mutable, and therefore, when an object is primarily meant to represent an algorithm, mutating it is a dangerous activity that ought to be difficult to do by accident (G3 of §0.1.2)."

I'm not convinced that these things are dangerous. I expect these kinds of accidents to cause run time errors nearly 100% of the time. They could do dangerous things in particularly contrived cases, but IMO, contrived cases are indistinguishable from non-accidents. (Yeah, I'm not being very empirical about this. >.> )

Sure, but I'd argue that A) you want hygiene most of the time, so forgetting to $quote something happens less often, and B) in the case where you forget to $quote something with $quasisyntax, it usually throws an error. The reverse is not true: if you forget to $unquote something with $quasiquote, it's usually not an error, it fails silently until the bug is later discovered (possibly after a long time).

In other words, $quasisyntax not only chooses a default that you (the programmer) want more often (hygiene), but it also tends to throw errors earlier in case you (the programmer) mess up. That's why I consider $quasiquote to be bad. But in some hypothetical language where you want to break hygiene a majority of the time... then $quasiquote would probably be superior to $quasisyntax.

---

"To me, this isn't just about whether the computer does what the user wants/expects it to do. It's about whether the computer does what the user doesn't know they want it to do."

If so, then in order for the user to be guided into areas where they don't know they want it to behave in that way, then the language needs to encourage/discourage various behaviors in some way. Every language does this, in different ways, and to different degrees.

---

"but I'm not so convinced by these uses"

1) I agree, that doesn't seem to have much to do with G3 simply because if you accidentally use the wrong case for a long identifier, it's very likely to throw a runtime error that the variable can't be found.

2) I actually think this does justify G3 (though I agree it's pretty weak), for the same reasons I mentioned that using $quasiquote encourages you to use bad hygiene. If you don't provide a solid traversal algorithm, programmers are going to write their own awful algorithms which are no doubt broken.

I don't think Kernel goes out of its way to prevent you from writing stupid naive algorithms, and writing stupid naive algorithms is really easy. Instead, that's a rationale for not including naive algorithms in the core. You can still write your own naive algorithm, it just isn't going to be baked into the language, especially since the only gain of a naive algorithm is execution speed, not power or flexibility.

3) That particular part of the Report is talking specifically about operand capturing, e.g. fexprs. What it's saying is that when you apply a value (usually an acyclic list) to a fexpr, that fexpr could potentially mutate it.

This could cause some unexpected "action at a distance" so Kernel instead makes an immutable copy of the value and applies that to the fexpr instead. I'm not actually sure how I feel about that...

Incidentally, I believe that's how most (all?) Lisps do it: if you use (apply foo bar) in Arc, the foo function gets a copy of the list bar, not bar itself.

If you want the fexpr to be able to mutate its arguments... I would have suggested to pass the (mutable) list as a single argument to the fexpr, but... I'm not sure if that's actually possible... but it should be, right...? I would assume so.

---

"I'm not convinced that these things are dangerous."

I don't think they're any more dangerous than breaking hygiene. But Kernel already specifically mentions breaking hygiene as being dangerous, therefore calling those things dangerous is consistent with Kernel's viewpoint. So I suppose it depends on how dangerous something is before you personally (the programmer) call it "dangerous". But John Shutt apparently did consider them dangerous, hence their design in Kernel.

...which is what I said :) ...

"...but it also tends to throw errors earlier in case you (the programmer) mess up."

I agree with this too. Throwing an error early is a nicer accident than throwing an error late.

---

"If so, then in order for the user to be guided into areas where they don't know they want it to behave in that way, then the language needs to encourage/discourage various behaviors in some way. Every language does this, in different ways, and to different degrees."

I don't see how this has a point any different from what I'm saying. :-p

---

"I don't think Kernel goes out of its way to prevent you from writing stupid naive algorithms, and writing stupid naive algorithms is really easy. Instead, that's a rationale for not including naive algorithms in the core. You can still write your own naive algorithm, it just isn't going to be baked into the language, especially since the only gain of a naive algorithm is execution speed, not power or flexibility."

In a customizable language (which Kernel isn't), to do something the most obvious way makes that part of the language more intuitive to customize, even if the most obvious thing is naive.

In a language with a culture of sharing black-box libraries (which Kernel would probably be, but for instance wart might not be), to do something the same way a library would do it makes for a more consistent experience.

Instead, for consistency's sake, Kernel advocates for library writers to write non-naive algorithms, ultimately making library-writing more of a chore.

This is totally a "worse is better" line of argument, so I don't expect to convert you with it so much as to convince you the ideals aren't so clear-cut.

---

"If you want the fexpr to be able to mutate its arguments... I would have suggested to pass the (mutable) list as a single argument to the fexpr, but... I'm not sure if that's actually possible... but it should be, right...? I would assume so."

I think Kernel calls the copy algorithm "copy-es-immutable," and IIRC, what it does is [treewise cons idfn _], but with an immutable cons. So if your mutable conses are hidden behind a variable name (to be evaluated) or a thunk, they'll survive. If you're just trying to eval (foo (my mutable list)), that won't work.

---

"I personally wouldn't call them "dangerous". I think that word should be reserved for disastrous things like, say, deleting your entire hard-drive by accident."

I agree. That's why I brought up those uses of G3; they just seem like examples of accidental errors being prevented, rather than accidental disasters being prevented.

---

"I would call things like breaking hygiene and whatnot "undesirable violations" (with the caveat that they can be desirable in certain circumstances, so they shouldn't be banned outright). If you have a better word to describe it, I'd like to hear it."

I don't know what you mean by "things like breaking hygiene and whatnot," unless you mean all the perhaps-not-so-dangerous things John Shutt considered dangerous when writing the R-1RK, and that isn't a very well-defined set of things. :-p I would probably call them "bugs."

Maybe G3 becomes something like this: While permitted on general principle, bugs should be hard to introduce by accident.

Hmm, I think this could be another, more upfront way to phrase the way G3 is actually used: The language should allow more expressiveness than is recommended for everyday use, but programmers on the fringe cases should have to jump through extra hoops.

Right, I'm agreeing with you a lot, except that you seemed to be a bit against some of the things Kernel does, and I was pointing out that Kernel does that because it's trying to guide you toward a certain way of thinking that (hopefully) results in better programs. Whether you agree with Kernel's guidelines or not is another story.

---

"Instead, for consistency's sake, Kernel advocates for library writers to write non-naive algorithms, ultimately making library-writing more of a chore."

Library-writing already is a chore. If the only libraries available are hacky semi-broken things, then why would I use them? I can just write my own hacky semi-broken program faster than it would take to understand the library (usually).

So one of the primary benefits of libraries is that you can put a lot of extra effort into it, and then people use the top-notch shiny awesome library so they don't have to put all that effort into getting things perfect. Kernel just seems to embrace that and try to make even the core language itself rock-solid.

---

"This is totally a "worse is better" line of argument, so I don't expect to convert you with it so much as to convince you the ideals aren't so clear-cut."

I am generally an advocate of "worse is better", or else why would I be using Arc? When discussing Arc, I do tend to have a "worse is better" attitude about it. But at the same time, I also care about "doing the Right Thing", so my thinking shifts when talking about Kernel.

Essentially, Kernel seems to me to be a marvelous language that embraces the "do the Right Thing" attitude, whereas Arc seems to be a marvelous language that embraces the "worse is better" attitude. I like both of them, they have different atmospheres and feelings about them.

So, I think it really does depend on what you want to do, what you expect out of a language, etc.

I must say, though, Kernel has captivated me recently. I'm a total sucker for minimalism and elegance, while at the same time caring about speed and practicality. Inner conflict!

---

"unless you mean all the perhaps-not-so-dangerous things John Shutt considered dangerous when writing the R-1RK"

I was being hand-wavy and meaning things similar to hygiene breaking, like the 2nd and 3rd things on your list of things you mentioned in your post. So, not necessarily all the things John Shutt considered dangerous, but certainly some of them.

---

"Maybe G3 becomes something like this [...] I might actually support a G3 worded this way"

  ;; implicit eval
  ($define! $or?
    ($mac (x y)
      ($let ((temp (gensym)))
        `(,$let ((,temp ,x))
           (,$if ,temp ,temp ,y)))))

  ;; explicit eval
  ($define! $or?
    ($vau (x y) env
      ($let ((temp (eval x env)))
        ($if temp temp (eval y env)))))

  (mac do1 args
    #`(let g ,(car args)
        ,@(cdr args)
        g))

That would make it essentially a fully hygienic system that's vaguely similar to Scheme's macros, but it's much easier to implement, and you can choose to easily break hygiene at any time (by using quote) rather than fussing with syntax->datum.

It's also fully compatible with Arc 3.1 macros[1], and you can freely choose to use quasiquote or quasisyntax, or hell, use both in the same macro!

Of course, this requires you to actually write the macros in that style. Normal quasiquote macros won't gain any benefit from hygiene in this model: you'd need to convert them to use quasisyntax instead (which is quite easy and even mechanical to do).

---

The bits that are non-traditional are: eval can only operate using the current lexical scope and dynamic bindings, and the eval inside macros happens implicitly because I can't provide the caller scope to it.

  (fn ('a b c . 'd)
    ...)

  (fexpr (a b c . d) env
    (with (b (eval b env)
           c (eval c env))
      ...))

That is, a fn would either be wrapped or not depending on whether a quote appears in its argument list. I'm actually not fond of that, and would prefer to explicitly use fexprs when I want to not evaluate things, but... it is consistent with wart's already existing support for things.

I didn't at first either. But I walked through the steps one-by-one until I figured it out. Here they are:

apply is a function, so it evaluates all its arguments. When calling (apply list '((1 2))) its arguments evaluate to the function list and the list ((1 2)). It then calls the function list with the argument (1 2). This is equivalent to (list (1 2)), as rocketnia said.

---

To put it another way, (apply list '((1 2))) would be equivalent to (eval (cons list '((1 2)))) which is equivalent to (eval (list list '(1 2))) which is equivalent to (eval '(list (1 2))) which is equivalent to (list (1 2)).

  (def apply (f args env)
    (eval (cons f (map quote args)) env))

---

  >
  > (map quote args)
  >

  (def map (f seq)
    ...
    ... (f elem) ...
    ...)

  (def mappend (f . args)
    (apply + nil (apply map f args)))

  (def string args
    (apply + "" (map [coerce _ 'string] args)))

With Arc, it's simple: the return type of + is always the same as the type of the first argument to +. I can understand if you (dido) don't like this kind of polymorphism or implicit coercion, but it does fit in well with Arc's philosophy.

I would also like to note that if it were up to me, (+ 'foo "bar") would also be valid: it would return the symbol 'foobar. And (+ (obj a 1) (obj b 2)) would return (obj a 1 b 2). Arc 3.1 doesn't do this by default, though, unfortunately.

I would also like to point out that I'm not against using + for only addition. In Arubic I would probably prefer a different operator for concatenation, like `join`, or maybe `cat`.

---

Here's an old thread where we discussed this issue a bit: http://arclanguage.org/item?id=14057

Once you start down the Arc path, forever will it dominate your destiny. :-p Really though, I'm glad to hear you're seeing your goal through before you veer off in some other direction.

I too struggled with this. I wanted to change ar significantly in ways that I considered better. Then when I started the Nu project, it too made many incompatible changes to Arc 3.1. I still believe these ideas are good and improve Arc 3.1, but they're incompatible nonetheless.

But now my opinion is that any new language based on Arc should be cleanly separated. So I've started my work on Arubic (my own language based on Arc) as a separate project from Nu. To be more specific, it's one language implemented with Nu, the other one being Arc 3.1.

Concatenation has an identity element, and it's associative, making it a monoid operation. I just think of '+ as a polymorphic monoid operation. When the monoid is a group, unary '- comes in to be an inverse operator. When the group is a field, '* comes in to be a multiplication operator, with unary '/ as its inverse.

So I don't have any mathematical objection to using '+ for concatenation, and I actually think it fits quite well. I've even thought about using '+ for function composition, since that's another monoid. (However, recently I've been more attracted to the idea of managing each monoid theory as a separate entity, rather than sniffing the arguments to figure out which operation to use. This is made easy by Haskell type classes, which essentially do the sniffing at compile time based on the static type system.)

While it make sense for a news aggregator to prevent thread necromancy in order to have only fresh content on the new page, on a forum like this, it only provokes burying of interesting content, (especially painful since we no longer have a custom search) and needless thread duplications.

Right now it seems to me that the best alternative to the Arc forum would be convore[1], which ar uses[2]. It has some problems, though. In particular I remember it being quite difficult to paste in code snippets. That would be problematic for a forum related to Arc. Any other suggestions?

---

* [1]: https://convore.com/

Although I appreciate the interest, arc2js is currently unusable. I haven't taken the time to port it to either Arc 3.1 or Arubic, so as far as I know you can't actually use it. It's on my TODO list to fix it, but that's not very comforting!

https://github.com/Pauan/ar/tree/lite-nu

  $ arc
  > (def foo((o a)) a)
  #<fn:foo>
  > (foo)
  #<fn:...nu/01 nu.rkt:106:50>

I don't know why it doesn't have an issues tab. In any case, any issues can either be posted here or e-mailed to me.

---

One other change I had to make was to add keywords to the call to open-output-file in nu.rkt:

  (open-output-file f #:mode 'text #:exists mode)

It does, however, allow Arc to pretend that "everything is Unix" since 'text mode automatically converts "\n" to "\r\n" on Windows. But infile uses 'binary mode, so in that case it should probably use 'text mode too, to maintain the illusion.

b) Feel free to reorganize the page to bring say ar up as well.