(def dispfile (val file)
    (w/outfile f file
      (disp val f))
    val)

  (def dispfile (val file)
    (let tmpfile (+ file ".tmp")
      (w/outfile o tmpfile (disp val o))
      (mvfile tmpfile file))
    val)

  (tofile "tmpfile" (disp "this is some text"))

http://github.com/nex3/arc/commit/c70a51a4d5

My first impression is: too verbose. But it's quite easy to define consif and conswhen in terms of maybe, so I think that is best:

  (def consif (a b)
    (maybe cons a b))

  (def conswhen (a b)
    (maybe cons (check a f) b))

---

How about letmaybe?

  (letmaybe x f ...
    ...)

  ->

  (let x ...
    (when (f x)
      ...))

  (let x init
    (when (f x)
      ...body...))
  ==>
  (checklet x f init
    ...body...)

  (def fn-checklet (test x body)
    (test&body x))
  (mac checklet (var test x . body)
    `(fn-checklet ,test ,x (fn (,var) ,@body)))

I would consider any Arc program to be "part of Arc" for compatibility purposes because any Arc program is supposed to use only the core and things defined in arc.arc (along with its own libraries, of course).

Obviously all bets are off if it uses the $ macro to drop into Racket, but then you're not really writing Arc code anymore, so I'm ignoring that case.

---

"(assign || (sym ""))"

That's a pretty good way of solving that one particular use-case, but it won't help in general because there might be some Arc program that actually does use the |...| notation more extensively.

However, I suspect such programs are quite rare, and with good reason. So I don't think it's a major deal if an Arc implementation chooses to not implement the |...| notation. However, such incompatibilities should be documented, so people using your implementation know what works and what doesn't (and why it doesn't work).

I don't understand. If I write a hello world arc program, are you saying that's part of arc?

"..because any Arc program is supposed to use only the core and things defined in arc.arc."

For the sake of compatibility with Arc, yes. An implementation claiming compatibility with Arc should be capable of running programs designed for Arc 3.1.

---

"what do you consider to be the core of arc?"

We were discussing what constitutes "part of Arc" for compatibility purposes. "srv.arc" is not a part of the core of Arc, nor are any libraries written in Arc. But it is still "part of Arc" for the sake of compatibility, in the sense that a compatible implementation of Arc must be capable of running them, warts and all.

I admit, this felt unnatural at the start because I had to wean myself away from thinking in terms of transformation. I used to have a function called stringify to convert symbols to strings[1], but eventually I noticed that plain string does the job just fine.

In wart at least I'm consistent about ending real predicates with a '?', so it won't be mistaken for a test itself. Wart has also collapsed obj into table so the pattern of constructor names is stronger.

Before I was lukewarm on the name, but this reminded me of the pronunciation difference. Pronounce it like the noun (predikit) and it sounds confusing, but pronounce it like the verb (predikayte) and it sounds great.

Personally, I think this is all moot, but I'd go with "checkify" or "to-check".

  testify
    Pro: Only 7 characters long.
    Pro: Bunches with related words containing "ify."
    Con: Bunches with unrelated words containing "test."
    Con: Is a neologism if used in English.
    Con: Has non-sequitur homonyms in English (one meaning "claim").
    Con: The vocal majority here at Arc Forum seems to dislike it. :-p
  
  predicate
    Pro: Bunches with related words containing "icate."
    Con: Debatable pronunciation.
      "predikayte"
        Con: Is a neologism if used in English, I think. It
          reinterprets the verb as a verbalization of the "predikit"
          version, when "predikit" is actually something of a
          nominalized form of "predikayte" in the first
          place[Wiktionary]. Is it already used this way?
        Con: Has non-sequitur homonyms in English (one meaning
          "assume").
      "predikit"
        Pro: Same as a related term used in English discussion.
      "predikahtay"
        Con: Has a non-sequitur homonym in Italian (meaning
          "preach")[Wiktionary].
    Con: Since it has the same spelling as a noun, it may conflict with
      other noun-based names (e.g. accessors, local variables).
    Con: In English discussion, needs special formatting to look like a
      variable name.
  
  checkify
    Pro: Bunches with related words containing "check."
      Pro: "Check" is a related term that can be dropped casually into
        English discussion.
      Con: "Check" has many non-sequitur homonyms in English (one
        meaning "restrict").
    Pro: Bunches with related words contianing "ify."
    Con: Is a neologism if used in English.
  
  to-check
    Pro: Bunches with related words containing "check."
      Pro: "Check" is a related term that can be dropped casually into
        English discussion.
      Con: "Check" has many non-sequitur homonyms in English (one
        meaning "restrict").
    Pro: Bunches with related utilities containing "to-".
    Con: Is a downright technical term if used in English.
    Con: "To check" could be seen as an infinitive form.

Older versions of Nu had lots of unit tests, yeah, but I haven't ported them over to the latest version of Nu yet and probably won't do so anytime soon, as I've unfortunately lost the motivation to work on anything Arc related.

---

"Nu's benchmarks are meant to be runnable on multiple Arc implementations for comparison purposes, so they could be a good start."

Yeah, but for now all the Arc implementations need to be built on top of Racket, so it works for ar, Arc 3.1, Nu, etc. but not, say, Rainbow. Getting it to work with non-Racket processes is probably doable - albeit difficult - and it would come at the cost of accuracy in the tests.

Or perhaps you're talking specifically about functions mutating their environment... well you can do that in Kernel too:

  (wrap ($vau ... env ...))

  ($vau ... env1
    ($vau ... env2
      ...))

  (fn '...
    (let env1 caller-scope
      (fn '...
        (let env2 caller-scope
          ...)))

If I just saw 'caller-scope in pseudocode somewhere, I'd implement it as an anaphoric variable of 'fn (and I'd implement a non-anaphoric variant of 'fn to base it on, so we'd basically be at vau again :-p ). So at the global scope, my implementation would just treat 'current-scope as a global variable.

  (env= caller-scope foo 5)

  (let global caller-scope
    ...
    (env= global foo 5)
    ...
    (eval ... global))

  ($let ((global (get-current-environment)))
    ...
    ($set! global foo 5)
    ...
    (eval ... global))

On the other hand, it's quite easy to define wart's magical fn in terms of vau... with no leaky abstractions! The only catch is that caller-scope will always refer to the global scope inside vau forms... that could be changed too, by extending/overwriting vau, but I didn't do that:

  (= get-current-environment (wrap (vau () e e)))

  ;; don't define this if you don't want caller-scope at the global level
  (= caller-scope (get-current-environment))
  
  ;; simple fn, without magical quoted args or caller-scope
  (= fn (vau (parms . body) env
          (wrap (eval `(,vau ,parms nil ,@body)))))

  ;; omitting other stuff like afn, no, caris, list, etc.
  ;; but they can all be defined in terms of simple fn
  ... 
          
  ;; makes these variables hygienic
  (with (with  with
         eval  eval
         let   let)
         
    ;; this recreates the part of wart that is currently handled in C++
    (= parse-fn (fn (parms env)
                  (if (caris x 'quote)
                      (list (cdr parms) nil)
                      ((afn (x vars parms)
                         (if (no x)
                               (list (rev parms)
                                     (rev vars))
                             (caris (car x) 'quote)
                               (self (cdr x)
                                     vars
                                     (cons (cdar x) parms))
                             (let c (car x)
                               (self (cdr x)
                                     `((,eval ,c ,env) ,c ,@vars)
                                     (cons c parms)))))
                       parms nil nil))))
    
    ;; overwrite with the complex fn which has quoted args and caller-scope
    (= fn (vau (parms . body) env
            (w/uniq new-env
              (let (parms vars) (parse-fn parms new-env)
                (eval `(,vau ,parms ,new-env
                         (,with ,vars
                           (,let caller-scope ,new-env
                             ,@body)))
                      env))))))

  (fn (a 'b . c)
    ...)

  (vau (a b . c) g1
    (with (a (eval a g1)
           c (eval c g1))
      (let caller-scope g1
        ...)))

One more caveat: I'm not sure how you implemented caller-scope. If it's hardcoded into every function such that functions can't have an argument called "caller-scope" then the above will work the same way. But if it's a global implicit parameter like in Nu and ar, then you'll need to search the arg list for an argument called "caller-scope", which is doable but would require some more code, especially for handling argument destructuring.

---

Anyways, I'm not saying you should necessarily do it this way in wart, but... I think it's cleaner, conceptually, for functions to be defined in terms of vau, rather than vau being defined in terms of functions.

Hmm... yeah, it's a pretty leaky abstraction. It's one of these features with ambient authority:

  (if designed like a global variable assigned at the beginning of each
  call)
  
  Right now, get the most recently started call's caller's environment.
  
  
  (if designed like a continuation mark)
  
  Right now, get the deepest-on-the-stack call's caller's environment.
  
  
  (if designed like a local variable bound by each (fn ...))
  
  Given a non-global lexical environment, its local variables must have
  been initially bound by some call, so get the lexical environment
  that was used to evaluate that call.

...Actually, the first two of these designs are broken! Say I define a simple fexpr like this one:

  (def idfn '(result)
    (eval result caller-scope))

  ((fn (a) ((fn () (idfn caller-scope)))) 1)  ; eval'd in global scope
  ((fn () (idfn caller-scope)))  ; eval'd in local scope of (fn (a) ...)
  (idfn caller-scope)            ; eval'd in local scope of (fn () ...)
  (eval result caller-scope)     ; eval'd in local scope of idfn
  caller-scope                   ; eval'd in local scope of (fn () ...)

Does Wart have this awkward behavior, akkartik?

---/change of topic---

To get back to the point, the local-variable-style version of the feature (the only version that works?) doesn't strike me as completely undesirable. It lets the programmer evaluate expressions in arbitrary ancestor stack frames, and that isn't a shocking feature to put in a debugger.

Wart returns the scope with a set to 1. Is that the one you mean? It's different from the value of:

  ((fn (a) caller-scope) 1)

http://github.com/akkartik/wart/blob/98685057cd/008eval.cc#L...

But I don't think it will become Kernel. Kernel seems really elegant as long as you give up quote and quasiquote. I suspect if you bolt them onto Kernel it'll have slid from its sweet-spot peak of elegance. A language that encourages quoting and macros will diverge significantly from Kernel.

http://github.com/akkartik/wart/blob/98685057cd/008eval.cc#L...

It tries to bind caller-scope after the params, so this fails:

  wart> def foo(caller-scope) 34
  wart> foo 3
  007scope.cc:77 Can't rebind within a lexical scope
  dying

In wart scopes are just tables. It's not a good idea to modify them, but nobody will stop ya :)

"Kernel's "get-current-environment" is not the same as the environment argument passed to fexprs."

Can you elaborate? In

  ($vau foo env
    ..)

  ($let ((env1 (get-current-environment)))
    ...
    ($vau foo env2
      ...
      ($let ((env3 (get-current-environment)))
        ...))

env2 is the dynamic scope where the $vau is called, it does not include bindings for foo, env1, env2, or env3.

env3 is the environment of the $vau itself. This environment inherits from env1, and thus is lexical just as env1 is. And so, it contains all the bindings of env1, in addition to bindings for foo, env1, and env2, but not env3.

So, by using this combination of lexical and dynamic scope, you can express a huge variety of different things in Kernel.

---

Let me explain via a more elaborate example:

  ($let ((global (get-current-environment)))
    (display (eval 'x global))
    
    ($set! global $foo
      ($vau (x) env
        (display (eval 'x env))
        (display (eval 'env env))
        
        ($let ((inner (get-current-environment)))
          (display (eval 'x inner))
          (display (eval 'env inner))
          (display (eval 'global inner))))))
          
  ($let ((x 10))
    ($foo 20))

Then, we create a $vau form and assign it to the variable $foo in the global environment. This is the same as saying ($define! foo ...) at the top level, but is necessary when inside a $let.

Now, at the bottom, we use $let to bind the variable 'x to 10 and then call $foo with the single argument 20. First, $foo tries to evaluate the variable 'x in the dynamic environment "env". This should display 10, because it's using the binding in the $let where $foo is called.

We then evaluate the variable 'env in the $vau's dynamic environment. But the variable 'env does not exist in the dynamic environment, so it throws an error.

Then, we use a $let inside the $vau to grab a hold of the $vau's inner environment and bind it to the variable 'inner. We then evaluate the variable 'x in the $vau's inner environment. This should display 20, because it's using the binding inside the $vau, which is bound to the $vau's argument 'x which was 20 when calling the $vau.

We then evaluate the variable 'env in the $vau's inner environment, which evaluates to the dynamic environment, because inside the $vau, the variable 'env is bound to the dynamic environment.

Lastly, we evaluate the variable 'global in the $vau's inner environment. This evaluates to the global environment because the $vau's inner environment inherits from the top-level $let where the variable 'global is bound.

--

Here is a picture showing the environments in the above code:

http://img42.imageshack.us/img42/1870/vauenvironments.png

The global environment is pink, the first blue environment is the $let's environment that inherits from the global environment, plus a binding for the variable 'global. The green is the $vau's environment which inherits from the $let's environment, plus a binding for the variables 'x and 'env. And the yellow is the $let's environment that inherits from the $vau's environment.

In Kernel, the dynamic environment is exactly analogous to dynamic scope. The difference is that lexical scope is the default, and you have to explicitly ask to evaluate things with dynamic scope, rather than earlier Lisps which used dynamic scope by default and didn't even have lexical scope.

Special variables in Common Lisp are similar to dynamic scope, except you're flagging only certain variables to be dynamic, whereas the dynamic environment in Kernel lets you evaluate any variable with dynamic scope.

---

"If we only create bindings using let and combiner parameters, the dynamic environment is just the caller's static environment, is that right?"

Yes. In fact, in Kernel, $let is defined just like it is in Arc, using $lambda. And $lambda uses $vau, and $vau is the only way to create new environments in Kernel, as far as I know. So, environments are all about $vau.

---

  ($define! $foo
    ($vau (x) env
      (list x env)))
      
  ($define! $bar
    ($vau (x) env
      ($foo x)))
      
  ($bar 10)

  ($define! $bar
    ($vau (x) env
      ($foo x env)))

  ($let ((x 10))
    ($bar x))

  ((wrap ($vau (x) env
           ($bar x env)))
   10)

---

"I thought about whether I could build lazy eval directly using $vau"

Ah! I considered this approach when I was building dynamic variables in wart, but it seemed less confusing to attach the scope to the binding. Wart is unlike common lisp in this respect. SBCL:

  * (defvar foo 34)
  * (defun bar() (foo))
  * (bar)
  34
  * (let ((foo 33)) (bar))
  33 ; foo is still dynamic. WEIRD.

  wart> = foo 34
  wart> def bar() foo.
  wart> bar.
  34
  wart> let foo 33 bar.
  34 
  wart> making foo 33 bar.
  33 ; dynamic scope

  ($vau () env
    (eval 'x env))

---

If you're talking about something similar to Racket's parameterize, you should be able to build that using dynamic-wind. Then something like this...

  ($parameterize ((foo 10))
    ...)

  ($let ((env   (get-current-environment))
         (orig  foo))
    (dynamic-wind ($lambda ()
                    ($set! env foo 10))
                  ($lambda ()
                    ...)
                  ($lambda ()
                    ($set! env foo orig))))

In fact, I'm gonna write a $parameterize form right now:

  ($define! $parameterize-redirect
    ($vau (target tree . body) env
      ($let ((inner (get-current-environment))
             (env   (eval target env)))
        (dynamic-wind
          ($lambda ()
            ($set! inner orig
              (map ($lambda ((var expr))
                     (list var (eval var env)))
                   tree))
            (for-each ($lambda ((var expr))
                        (eval (list $define! var expr) env))
                      tree))
          ($lambda ()
            (eval (cons $sequence body) env))
          ($lambda ()
            (for-each ($lambda ((var expr))
                        (eval (list $define! var expr) env))
                      orig))))))
                      
  ($define! $parameterize
    ($vau (tree . body) env
      (eval (list* $parameterize-redirect env tree body) env)))

  ($let ((x 5))
    ... x is 5 ...

    ($parameterize ((x 10))
      ... x is 10 in here ...
      )

    ... x is 5 again ...
    )

What about this?

  (def get-current-environment ()
    caller-scope)

  get-current-environment.
  thunk.caller-scope.

  (get-current-environment)
  ((fn () caller-scope))     ; Wart
  (($vau () e e))            ; Kernel

I think the fundamental reason for this is that macros don't evaluate their arguments, but functions do. Your example macro is written like as if it were a function, but it's not a function. If you instead wrote it as a macro, like so:

  (mac foo (a b) `(cons ',a ',b))

---

"This apply works for some macros (like Pauan's def example), just not all of them."

It still works if you call it like this: (apply foo '(1 '(2 3)))

As I mentioned in my e-mail, I agree that is an inconsistency when trying to make apply work with fexprs, which is part of why I prefer Kernel's approach of requiring you to wrap the fexpr before calling apply on it... but that doesn't mean that a version of apply that works on fexprs is necessarily broken, just a bit hacky and inconsistent.

I wouldn't call apply broken for the reason you demonstrate above (with the foo macro), because foo really should be written as a function, or its definition should be changed to make use of the non-evaluating nature of its operands.

---

It seems to me that when you talk about apply "working correctly" you're expecting it to somehow guess that the foo macro is behaving like a function, so applying it should treat it like a function, even though it's a macro. But precisely because macros/fexprs might or might not evaluate their arguments, apply cannot guess that.

apply can know whether its argument is a function or fexpr, but not whether the fexpr will actually evaluate its operands or not... so it seems your concept of apply "working correctly" would require apply to essentially read your mind, or do some sort of crazy static analysis and make guesses.

Let me put it like this. When calling apply on something, let's pretend that apply maps quote over the argument list, so that these two are equivalent:

  (foo 1 '(2 3))
  (apply foo '(1 (2 3)))

  (foo 1 (2 3))
  (apply foo '(1 (2 3)))

Really, if you want to solve this, I would suggest having two operators, one for the function-like behavior, and one for the fexpr-like behavior. I'll call these two "apply" and "operate", respectively. So, assuming foo is a function, then the following are equivalent:

  (foo 1 '(2 3))
  (apply foo '(1 (2 3)))
  (operate foo '(1 '(2 3)))

  (apply (wrap foo) '(1 '(2 3)))

  (eval `(,foo 1 '(2 3)))

---

"I'd love comments on this way of decomposing things."

"Do you have a better example of a macro that breaks with apply, a macro that's actually written as a macro, rather than as an almost-function?"

How about and in the previous thread? I tried to use it like this:

  (def all-trues(xs)
    (apply and xs))

  (all-true (list '(1 2))) ; => error

But that's completely incorrect because you always always have to know whether it's a function or a macro, even in a language like Arc or Scheme, because they are fundamentally different things.

Even fexprs aren't the same as functions because they don't evaluate their arguments, and argument evaluation is important, so it's very important that you understand whether something is a function or fexpr, and if it's a fexpr, which arguments are evaluated, and how.

That's a large part of the reason Kernel prefixes fexprs with $ by convention: it really helps you to notice right away the difference between the two, and it's an important difference.

---

"How about and in the previous thread?"

Yeah, I think that qualifies as a good example. It breaks for the exact same reason that your foo macro breaks: it evaluates its arguments, so you have to double-quote them:

  (all-true '('(1 2)))

"It's pretty clear that 'apply for macros' is an ill-posed, arguably meaningless problem, and I thank you for helping me understand that."

But the whole point is that the list is just a list of elements! Imagine it's coming from a whole complex sequence of functions. Don't think of it like this:

  (all-true '((1 2))

  (all-true l) ; with l bound to ((1 2))

"that's completely incorrect because you always always have to know whether it's a function or a macro, even in a language like Arc or Scheme"

What I called the 'point of lisp' was my sense of the ideal. The promise of a high-level language is that you can create new names and use them like primitives that the language came with.

  (let x 3
    bar.x) ; Does the language come with let? Who cares?

Sure, I already considered that. That is, after all, the most common use for apply. And I agree that's ugly. You'd have to say something like (all-true (wrap-quote l)) or somesuch. Ugly.

---

"Isn't that simply hideous?"

Sure is!

---

"The promise of a high-level language is that you can create new names and use them like primitives that the language came with."

And fexprs/macros let you do that. That wasn't my point. What you said is that the point of macros is being able to call things without caring whether the thing is a macro/function/whatever. That view is incorrect because the meaning of the expression differs depending on whether it's a function or macro.

For instance, in the "let" expression you just gave, it doesn't evaluate the variable "x", but you only know that because "let" is a macro. If it were a function it would throw a "x is undefined" error. So you have to know whether something is a function or macro in order to use it correctly.

The same is true of apply: it has two different (and inconsistent) behaviors depending on whether its argument is a fexpr or a function. And that's a problem because it's confusing and makes it really easy to get it wrong. Hence why Kernel makes you explicitly use wrap to specify your intent.

---

"You've explained the technical reasons why 'apply for macros' is hard, and that makes sense. But the solution is to not use apply with macros, not tie ourselves up in knots with quote."

Sure, and even with that approach, if your language has wrap/unwrap, then you can still use apply on macros, you just need to wrap them first:

  (apply (wrap foo) ...)

  (eval `(,foo ...))

No, I thought about that when I built let. And then I forgot about it. Now when I see that expression I think, "oh, I'm creating a binding." The point of having power is to not have to derive things from first principles every time we see them.

"And fexprs/macros let you do that. That wasn't my point. What you said is that you want to be able to call things without caring whether the thing is a macro/function/whatever."

Yes, but you still need to carry around that knowledge in your subconscious, and retrieve that knowledge every time you use "let". That was my point: that the distinction between them is important.

---

"The point of having power is to not have to derive things from first principles every time we see them."

Yes... and I'm agreeing with you.

You said that the point of macros is not having to care whether something is a function/macro or not. But on a very fundamental level you do have to care, just like how you have to care about whether a variable is a string or a number.

I was only disagreeing with that statement, that you don't need to care about the differences between fexprs/functions. All the problems with apply come about because of misunderstandings of the differences between the two:

  ;; functions
  (foo '1 '(2 3))
  (apply foo '(1 (2 3)))

  ;; fexprs
  (foo 1 (2 3))
  (apply foo '(1 (2 3)))

  wart> = l '((1 2))
  wart> (and @l) ; error
  wart> (wrap.and @l) ; error
  wart> (eval `(,wrap.and ,@l)) ; error
  wart> (eval `(,wrap.and ,@(map (fn(_) (cons quote _)) l))) ; error

  (def operate (f args)
    (eval `(,f ,@(map (fn (_) (list quote _)) args))))

I'm not saying that's necessarily a good idea, but if it comes up often enough, might as well write a helper function for it.

---

And when I said "bring out eval" I meant like this:

  (eval `(,and ,@l))

Ah, this works :)

  (eval `(,and ,@(map (fn(_) (cons quote _)) l)))

  def all-true(xs)
    (and @mapquote.xs)

  > > Isn't that simply hideous?
  >
  > Sure is!