Arc Forumnew | comments | leaders | submitlogin
Rethinking macros
2 points by rkts 6037 days ago | 38 comments
There seem to be a number of macros that could be replaced by plain functions except for some small cosmetic difference. For instance, the while macro could be rewritten as a higher-order function and used as follows:

  (while [blah]
    [blah blah blah])
The problem is that the bracket functions expect an argument, which doesn't make sense in this case. We could make the argument optional, but then _ would get bound to nil, shadowing any existing binding.

The alternative is to write

  (while (fn () (blah))
    (fn () (blah blah blah)))
which is just ugly. And so we make while a macro instead of a function.

And this depresses me, because macros, as we all know, are more troublesome than HOFs, for a million reasons: they're hard to read, they're error-prone, you have to fiddle with gensyms, you can't use nice things like afn and aif because of variable capture, etc. etc.

Here's my idea: why not define things like while as HOFs, and have the language generate a macro that handles the cosmetic issues for us? Example:

  (defho while ([test ()] . [f ()])
    ((afn ()
       (when (test)
         (f)
         (self)))))
The parameter list says to wrap the first form in an (fn () ...), and all the remaining forms in another (fn () ...). So supposing you wrote

  (while (blah)
    (blah blah blah))
this would expand to

  (while-aux (fn () (blah))
    (fn () (blah blah blah)))
where while-aux is a function generated by defho.

Here's another example, a drop-in replacement for awhen:

  (defho awhen (x . [f (it)])
    (if x (f x)))
This system breaks down for macros that ask the caller for the name of the variable to capture (e.g. for). To resolve this, we might have syntax to denote parameters that shouldn't be evaluated, and more syntax to denote where the contents of a variable should be included in the macroexpansion. Here's an example (overloading the asterisk for both purposes):

  (defho for (*v i max . [f (*v)])
    (while (<= i max)
      (f i)
      (++ i)))
Obviously, this idea won't eliminate macros entirely, but I suspect it would greatly reduce the need for them--perhaps even the majority of macros could be replaced by defhos.

Here are a few more examples:

  (defho loop (start [test ()] [update ()] . [f ()])
    (while (test)
      (f)
      (update)))

  (defho whilet (*v [test ()] . [f (*v)])
    ((afn ()
       (awhen (test)
         (f it)
         (self)))))

  (defho accum (*v . [f (*v)])
    (let xs nil
      (f [push _ xs])
      xs))
I suggest you compare these definitions with those currently in arc.arc. The difference is striking.

Of course this is all speculative and I haven't bothered to implement it, so there may be some rough edges that I haven't thought about yet.

Do you guys think this is a good idea? Has anyone done something similar before?



1 point by almkglor 6036 days ago | link

You know, what I'm really thinking is that placing the annotations in the argslist looks horrible. The body itself is not bad, but the argslist is really, really horrible.

How about this instead:

  (mac while (cond . body)
    (w/hof ( () ,cond
             () ,@body)
      ((afn ()
         (when (cond)
           (body)
           (self))))))
Where w/hof expands to:

  (do
    ; inefficient since we set it everytime the macro is called, but...
    (set gs42 ; set so it won't say "redefined"
      (fn (cond body)
        ((afn ()
           (when (cond)
             (body)
             (self))))))
    `(gs42 (fn () ,cond) (fn () ,@body)))
The above strikes me as being easier to implement.

Of course unfortunately it won't work properly for variable capture.

Edit: Hmm, maybe it'll actually be able to support some variable capture:

  (mac awhile (v cond . body)
    (w/hof ( ()   ,cond
             (,v) ,@body)
      ((afn ()
         (let tmp (cond)
           (when tmp
             (body tmp)
             (self)))))))
HOF-style anaphora:

  (mac afn (params . body)
    (w/hof ( (self ,@params) ,@body )
      (let self nil
        (= self
           (fn rest
             (apply body self rest)))
        self)))

-----

1 point by rkts 6036 days ago | link

Well, we can cut down on the parentheses in the parameter list. This looks ok to me:

  (defho while ([test] . [f])
    ...)

  (defho awhen (x . [f it])
    ...)
As to w/hof, it may be easier to implement, but it seems kind of hard to use.

-----

1 point by almkglor 6036 days ago | link

> As to w/hof, it may be easier to implement, but it seems kind of hard to use.

Then perhaps you can define 'defho in terms of 'mac and 'w/hof, maybe? The Arc solution: throw more macros at it.

Regarding (x . [f it]) :

In canonical arc2, [f it] is (fn (_) (f it)), so (x . [f it]) becomes (x fn (_) (f it)) . In Anarki, [f it] is (make-br-fn (f it)), so (x . [f it]) becomes (x make-br-fn (f it))

This makes the use of [] problematical.

'w/hof is trivially implementable, 'defho much less so due to the syntax.

w/hof is also more general, at the expense of being more talkative.

In fact it might be better to do things this way:

  (mac while (cond . body)
    (w/hof (cond (fn () ,cond)
            body (fn () ,@body))
      ((afn ()
         (when (cond)
           (body)
           (self))))))
This allows a few shortcuts in anaphora:

  (mac aif (cond then (o else))
    (w/hof (cond ,cond
            then (fn (it) ,then)
            else (fn () ,else))
      (if cond
          (then cond)
          (else))))

-----

1 point by rkts 6036 days ago | link

> 'w/hof is trivially implementable, 'defho much less so due to the syntax.

Oh. I haven't bothered to learn how to hack the syntax in Arc, so I'll take your word on that.

-----

1 point by shader 6037 days ago | link

Aren't the "only" differences between macros and functions the time of evaluation, the fact that they don't evaluate arguments, and the fact that they return code to be evaluated instead of a value?

What if we made those traits optional modifiers, similar to the * shown here, so that the programmer would have complete control over how the function operated? Then we could make functions that return code, but evaluate arguments, or avoids argument evaluation but returns a value, etc. And we could flag our functions to say whether they should be evaluated at compile time, or only at run time.

Any common patterns of modifiers could be abstracted by a "macro." So a function that ran at compile time, didn't eval args, and returned code, would be "(mac," whereas a function that evaluated arguments, returned a value, and ran at run time would be "(def" etc. etc. We could make more for other varieties of functions.

This doesn't make code shorter necessarily, but gives greater flexibility when defining functions.

Now, what did I forget that makes this obviously a bad idea?

If I understand it right, the [] syntax requires a function of a certain signature. Interesting idea.

-----

1 point by almkglor 6037 days ago | link

> Now, what did I forget that makes this obviously a bad idea?

Interesting, so we can get something that runs at compile time, evals the args (which we presumably evaluate in the runtime environment) and returns a value.

Oh, you can't get the runtime environment at compile time yet? Hmm.

> If I understand it right, the [] syntax requires a function of a certain signature. Interesting idea.

No, it transforms it to a function with that signature:

  (defho foo ([hmm (it)])
    hmm)
  =>
  (mac foo (hmm)
    `(foo-internal (fn (it) ,hmm)))
  (def foo-internal (hmm)
    hmm)

  (macex '(foo it))
  =>
  (foo-internal (fn (it) it))

-----

1 point by shader 6037 days ago | link

lol, I realize that you could try and apply those three at the same time, but the "compile time" modifier was supposed to be used by the programmer to tell arc that it was safe to do just such a thing. Other wise, it would be evaluated at run time. So while you could use those three at once, it would like you say have an undefined consequence.

Basically, I thought that it might be useful to have functions that didn't eval all their args. At that point, they're only a few steps away from macros, and I thought it would be cool if we could unify them.

>transforms to a function Just shows I didn't understand it right. :) That makes more sense, but I'm still not sure I understand it. Oh well.

Does your p-m:def support matching based on function sigs?

-----

1 point by almkglor 6036 days ago | link

> Basically, I thought that it might be useful to have functions that didn't eval all their args.

  (def foo-internal (x y z)
    (do-something x y z))

  (mac foo (x y z)
    ; eval x, don't eval y, eval z
    `(foo ,x ',y ,z))
> Does your p-m:def support matching based on function sigs?

Yes, although I'm not sure I quite understand what you're asking.

For example factorial works out like this:

  (p-m:def factorial
    "a factorial function"
    (1)  1
    (N)  (* N (factorial:- N 1)))

-----

1 point by shader 6036 days ago | link

Actually, what I meant by match on function sigs was: while writing a higher order function that wants a function as an input, can you require that it be a function that's signature implies one var, two vars, a list etc?

As to the non-evaling functions, I suppose we could have a macro that defined a macro / function combination; the resulting macro would expand into a call to the function with ' before each item in the arg list.

Oh, and I think that `(foo ,x ... is supposed to be `(foo-internal ,x ... Otherwise you have an infinitely recursive macro. :)

-----

1 point by almkglor 6036 days ago | link

> can you require that it be a function that's signature implies one var, two vars, a list etc?

Nope ^^

> Oh, and I think that `(foo ,x ... is supposed to be `(foo-internal ,x ... Otherwise you have an infinitely recursive macro. :)

Yep ^^

-----

1 point by almkglor 6037 days ago | link

Well, if the final syntax doesn't actually change (i.e. higher-order function while still looks like macro while) then what difference does it make just how, exactly, the base system works?

Other than to, say, make it difficult for hackers who've gotten used to `(, ,@) syntax?

Personally, I don't quite get what the [] and *v things are supposed to mean. Can you explain them?

As an aside, it might be better to implement it after all - I honestly don't see much difference for the end-user (you're still using macros anyway), but I do see some difficulty for the macro writer.

-----

1 point by rkts 6037 days ago | link

The point is that it makes a large class of macros easier to define. Really, compare my examples with the definitions in arc.arc; the former are much shorter and easier to understand than the latter.

My examples are from the core language, obviously, but the suggestion would benefit anyone wanting to extend the language with their own macros.

Placing a parameter inside brackets just says that the argument should be expanded into an fn. The thing after the parameter name is the parameter list that the fn should have. As to the asterisk, I don't know how to explain that better than I already have.

-----

1 point by almkglor 6037 days ago | link

Well, it would be a good addition to Anarki if you can actually implement it, and can prove your point that it's easier to use than using macros - obviously to do that, you can't just rewrite existing code, you have to write new code that uses your version of macros and see if it does indeed work better.

Further, using arc.arc as a basis is pointless; the reason those macros are there is because it's a waste of time to rewrite them. Most of the macros I've written in Arc depend on new syntax, not just rearranging expressions: look at p-m: and w/html , which I doubt are possible in ordinary HOF style.

So yes: while it certainly looks interesting, it doesn't seem to leverage the "common enough" theme quite enough.

Also, I somehow feel that what you really want are hygienic macros, which would look much more similarly to what you're doing.

-----

1 point by rkts 6037 days ago | link

Yes, hygienic macros could work too, provided we devise a good way to hack anaphora onto them. Another possible solution would be to just have a very concise syntax for function literals. Either way, I think that the rampant use of (unhygienic) macros where they are not necessary is a problem and needs to be addressed somehow. Especially since Arc is a Lisp-1.

-----

2 points by almkglor 6037 days ago | link

> hack anaphora onto them

Why not just leave unhygienic macros for the anaphora?

> Another possible solution would be to just have a very concise syntax for function literals.

Bingo. cref the discussion on currying some months back.

-----

1 point by rkts 6037 days ago | link

Because unhygienic macros are a pain in the ass. Surely I'm not the only one who thinks this?

-----

1 point by applepie 6036 days ago | link

Maybe you'd like them more if you called them "macros" instead of "unhygienic macros" ;)

No, really, macros, being essentially compilers, give you enough power to build everything you'd ever want into Lisp, including "hygienic" macros, and even to build facilities to write hof-like things in less painful ways.

Maybe they're a pain in the ass if you don't "go meta", in the same way computers are a pain in the ass if you don't build OSes and compilers first.

-----

1 point by stefano 6036 days ago | link

The real point in favor of unhygienic macros is that they are less constraining and, personally, I find them easier to write and read than hygienic macros. I don't find a bad idea to have both hygienic macros and unhygienic macros.

-----

1 point by rkts 6036 days ago | link

Sigh...

Obviously I like unhygienic macros when they are necessary. The problem is, I keep writing higher-order functions and getting tired of typing the "fn" over and over, and then I have to convert the function to a macro, making it twice as long and hard to read. Hygienic macros help, but they still are not as easy to write as plain functions.

I know I'm not the only person who has this problem, but maybe I'm the only one who realizes I have this problem. I see people on the Internet raving about the AMAZING POWER of macros, and most of their examples are just higher-order functions with some small cosmetic changes. Most of the macros in Arc are of the same kind.

I'm not denying that macros are powerful. I just think there is a gross inefficiency in using them where you shouldn't have to, just because of minor syntactic concerns.

I wanted to solve this with a short, clean syntax for function literals, but I haven't been able to come up with one and neither, apparently, has anyone else. So instead I decided to try something that would generate macros out of HOFs.

I thought this would be evident from my post, but apparently it wasn't.

-----

1 point by shader 6036 days ago | link

Maybe we could use { args | body } ? I don't think the braces are taken.

Now, maybe that's a bad idea; instead, we could redefine the brackets, so that a pipe divides args and body, and if it doesn't have a pipe, it defaults to binding _ ? I don't know how hard that would be, or some variation on the concept, but it would be a bit shorter than (fn (args) (body)), if you don't want to type that all the time.

And how exactly does 'w/hof work, as defined so far? And if it's "standard" now, why not just implement it?

-----

3 points by almkglor 6035 days ago | link

Since Anarki defines [ ... ] as (make-br-fn ...), it's actually possible to have the syntax:

  [params || body]
Which would be:

  (make-br-fn (params || body))
(The double bar is needed because a single | is reserved for weird symbols)

By simply redefining make-br-fn, you can redefine the syntax of [ ... ] to an extent

-----

1 point by shader 6035 days ago | link

I wondered if redefining [...] might be possible. It seems to me that the new double bar syntax is practically a super-set of the old one: if it doesn't have the double bar, just treat it like the old bracket function and define _ to be the argument; otherwise use the argument list provided. Including an empty list, I would hope.

Any word on how hard that would be?

-----

1 point by almkglor 6035 days ago | link

  (let old (rep make-br-fn)
    (= make-br-fn
       (annotate 'mac
         (fn (l)
           (if (some '|| l)
               (do
                 your-stuff)
               (old l))))))
Be careful of supersets: someone's code might unexpectedly break ^^

-----

1 point by shader 6035 days ago | link

Isn't that to be expected in an evolving open source language :)

Do you think || is the best choice, or something else?

-----

2 points by rkts 6035 days ago | link

I think it's a bad choice, personally. I'm not crazy about the single pipe either, but || is awful.

Tangent: this may be a dumb question, but do we really need the pipe character for symbols? I know I've never used it. Why not disallow spaces (and the like) in symbols, and free the pipe for new syntax?

-----

2 points by shader 6035 days ago | link

If you don't like the pipe, then recommend something :)

Other possibilities, in no particular order:

  [ # ]
  [ - ]
  [ = ]
  [ -> ]
  [ : ]
  [ => ]
  [ > ]
  [ ~ ]
  [ % ]
  [ ! ]
  [ $ ]
  [ ^ ]
  [ & ]
  [ * ]
  [ @ ]
  [ + ]
  [ | ]
  [ || ]
  [ ? ]
Most of those are either bad looking or already taken. Anything stand out as a good / ok / not bad choice?

-----

2 points by almkglor 6035 days ago | link

:, =>, and -> don't look bad.

# can't be redefined

Let's try some mockups:

  [ a b c :
    (/ (+ (- b) (sqrt (+ (* b b) (* 4 a c))))
       (* 2 a))]

  [: (thunk this)]

  [ a b c ->
    (/ (+ (- b) (sqrt (+ (* b b) (* 4 a c))))
       (* 2 a))]

  [-> (thunk this)]

  [ a b c =>
    (/ (+ (- b) (sqrt (+ (* b b) (* 4 a c))))
       (* 2 a))]

  [=> (thunk this)]

-----

1 point by shader 6021 days ago | link

So, did we ever make a decision about this? Does someone who knows more than I do about this want to implement it?

Also, is there a way to compose or nest these lambda shortcuts? Or would that make this almost impossible to implement?

-----

1 point by almkglor 6021 days ago | link

Nesting doesn't seem impossible: the reader, I think, will handle nesting as:

  [foo [bar]]

  (make-br-fn (foo (make-br-fn (bar))))
As for implementation, it's easy:

  (given ; this gives us access to the old
         ; implementation of [] syntax; it
         ; is used when we don't find the
         ; separator
         old (rep make-br-fn)
         ; use a variable to easily change
         ; the separator
         separator ': ;for experimentation
    (= make-br-fn
       ; a macro is just a function that has
       ; been tagged (or annotated) with the
       ; symbol 'mac
       (annotate 'mac
         ; the reader reads [...] as
         ; (make-br-fn (...))
         (fn (rest)
               ; find the separator
           (if (some separator rest)
               ; note the use of the s-variant givens
               ; the "s" at the end of the name of givens
               ; means that the variables are specifically
               ; bound in order, and that succeeding variables
               ; may refer to earlier ones
               (givens ; scans through the list, returning
                       ; an index for use with split
                       ; (no built-in function does this)
                       scan
                       (fn (l)
                         ((afn (l i)
                            (if (caris l separator)
                                i
                                (self (cdr l) (+ i 1))))
                          l 0))
                       ; now do the scan
                       i (scan rest)
                       ; this part destructures a two-element
                       ; list
                       (params body)
                         ; used to get around a bug in split
                         (if (isnt i 0)
                             (split rest i)
                             (list nil rest))
                 ; it just becomes an ordinary function
                              ; body includes the separator,
                              ; so we also cut it out
                 `(fn ,params ,@(cut body 1)))
               ; pass it to the old version of make-br-fn
               ; if a separator was not found
               (old rest))))))
Edit: tested. Also reveals a bug in split: (split valid_list 0) == (split valid_list 1)

  (= foo [ i :
           [ : i]])

  ((foo 42))
edit2: p.s. probably not really easy much after all^^. As a suggestion, (help "stuff") is good at finding stuff.

edit3: added comments

-----

1 point by shader 6021 days ago | link

Hmm. It doesn't seem to work with the older version. If I try ([+ _ 10] 3) it complains: "reference to undefined identifier: ___"

It used to complain "#<procedure>: expects 1 argument, given 3: + _ 10", but something seems to have changed between updates :)

-----

1 point by almkglor 6020 days ago | link

Have you tried restarting Arc and then repasting the code?

Probably some dirt left from older versions ^^

-----

1 point by shader 6035 days ago | link

I agree, those aren't bad.

I think that out of those, : makes the most sense. They all make logical sense with arg lists, but : looks the best without any.

-----

1 point by almkglor 6035 days ago | link

> Tangent: this may be a dumb question, but do we really need the pipe character for symbols? I know I've never used it. Why not disallow spaces (and the like) in symbols, and free the pipe for new syntax?

Wanna start implementing a reader for Arc?

-----

1 point by rkts 6035 days ago | link

Looks like this is configurable in MzScheme. Do

  (read-accept-bar-quote #f)

-----

1 point by rkts 6036 days ago | link

How would you write a function with no arguments?

-----

1 point by shader 6036 days ago | link

leave the part before the pipe empty, I suppose

  {| body}
it might need a space between the brace and the pipe, but I don't know

-----

1 point by shader 6037 days ago | link

Pardon my ignorance, but what is anaphora as it relates to macros?

-----

2 points by almkglor 6036 days ago | link

It's a macro which automagically binds a name. For example, 'afn:

  (afn ()
    (your-code))
  =>
  (let self nil
    (= self
      (fn ()
        (your-code))))
Or aif:

  (aif x
    (your-code))
  =>
  (let it x
    (if it
      (your-code)))

-----