"Other frameworks for lexically-scoped macros, notably syntactic closures (Bawden and Rees 1988) and explicit renaming (Clinger 1991), use a notion of lexical context that more directly maps to the programmer's view of binding scopes. Unfortunately, the more direct representation moves binding information into the expansion environment and, in the case of syntactic closures, tangling the representation of syntax and expansion environments."

I had exactly this entanglement happening in Penknife at one point. At the time a Penknife macro is defined, the local environment is captured, and later on, syntax generated by the macro is wrapped up in a way that refers back to that environment. At one point I was just putting the environment itself in the wrapper.

For your purposes, that might be just fine, especially if you're just going to eval the macro's result right when you get it. Then again...

PyArc's modules sound exactly like I want Penknife's (not yet existing) modules to be, and you seem to value the ability of the library users to configure the libraries to their own purposes, much like I do. I suspect, in our languages, programs will need to load the same library multiple times for different purposes, especially when it comes to diamond dependencies, where two individual libraries might depend on a common third library but configure it in different ways.

With Penknife's approach to extensible syntax, it turns out parsing is really slow, so I've organized the language so that it can parse a library in one pass and then run the resulting code over and over. ("Parse" basically means "compile" in Penknife, but Penknife does one or two additional things that could be called compilation, so I shy away from that word altogether.) That exposes a weakness of tangled syntactic closures: The compiled representation holds the original captured environment, rather than the one that should be used this time around.

Okay, so, I fixed the issue with it thinking a variable was undefined when it wasn't. I then found out something very interesting:

  ; foo.arc
  
  (assign message* "hello")
  (mac ret () 'message*)
  
  ; bar.arc
  
  (import foo "foo.arc")
  (foo!ret) -> "hello"

  (= ret foo!ret)
  (ret) -> error: message* is undefined

I'm pretty okay with this. It's kind of a wart that macros don't work quite right in that situation, but the fact that they do work when using the table is pretty darn nice. This also theoretically gives the caller the choice of whether to eval the macro in the defined environment, or the caller environment. The only issue, I think, is that this behavior may be confusing, at least until you've been bitten by it once or twice; by then you should have it memorized. :P

  (def foo ()
    (look-up-something-in syntax-rules*))

This is one reason it's challenging to do modules in Arc. You have to construct 'foo so that its references to 'look-up-something-in and 'syntax-rules refer to the meanings that exist at the time 'foo is defined (already done in Arc, but something to consider as you do things from scratch), and then you have to somehow make the same name have different meanings in different contexts (hard to do on top of official Arc, thanks to Racket's opaque namespaces, but very possible to do from scratch).

My eval hack is only about updating the internal table rules when somebody changes ssyntax-rules* This avoids checking it too frequently, potentially making it run faster. So, the following should work fine:

  (= foo ssyntax-rules*)
  (= ssyntax-rules* nil)
  
  foo            -> previous ssyntax rules
  ssyntax-rules* -> nil

  (= ssyntax-rules* foo) -> works fine again

The core is a module, right? At least in concept? And shouldn't you be able to modify something in another module when you need to, to fix bugs etc.? (Not that you'd be able to fix bugs in the core, necessarily. :-p )

---

"There's a bug with it right now, but I should be able to fix it. The environments get created properly, but when doing the actual lookup, it thinks the value is undefined when it isn't. That's pretty normal, since the interpreter is still missing lots of functionality."

It sounds almost like you shallowly copy the outer scope when you create a local scope, which means you miss out on later definitions made in the outer scope. Is this right? It's a pretty wild guess, and even if it is right, then I'm interested to know how you're shallowly copying arbitrary environment types. :-p

  (import foo "foo.arc")
  (= foo!bar 'qux)

  built-ins*!ssyntax-rules*  ; modify ssyntax globally

"then I'm interested to know how you're shallowly copying arbitrary environment types. :-p"

Ah, I don't know why I didn't see that. ^_^

---

"Environments are just wrappers around Python's dict type, so I can use the copy() method to make a shallow copy. I might need to make my own custom copy function later, though."

global_env is treated specially because it's the base for all other modules: it contains the built-in functions that modules cannot create (at least not in Arc), and also the core functions in arc.arc for convenience.

With a Racket namespace, you can get and set the values of variables, you can reset variables to their undefined state, and you can get the list of variables, so it's very much like a hash table. But suppose I want to have two namespaces which share some variable bindings. I may have been able to get and set the values of variables, but it turns out I can't manipulate the bindings they're (in my mind) stored in. I can't create two namespaces which share a binding.

  (= shared (new-namespace))
  (new-namespace shared)
  (new-namespace shared)

I rely on this behavior in Lathe, where I reduce the surface area of my libraries by defining only a few global variables to be macros that act as namespaces, and then replacing those global variables with their old values when I'm done. In an Arc that expands macros at run time, like old versions of Jarc, these namespace macros just don't work at all, since the variables' meanings change before they're used.

I also rely on it in my Penknife reimplementation. I mentioned (http://arclanguage.org/item?id=14004) I was using a DSL to help me in my pursuit to define a function to crank out new Penknife core environments from scratch. The DSL is based on a macro 'pkcb, short for "Penknife core binding," such that pkcb.binding-get expands to something like gs1234-core-binding-get and also pushes the name "binding-get" onto a global list of dependencies.

There's also a global list of Arc expressions to execute as part of the core constructor function. Adding to the list of expressions invalidates the core constructor function.

Whenever you call the core constructor, if it's invalid, it redefines itself (actually, it redefines a helper function) by evaluating code that has a big (with ...) inside to bind all those 'pkcb variables. But actually, I actually don't have a clue what 'pkcb variables to put in the (with ...) at that point. Fortunately, thanks to Arc's macro semantics, the act of defining triggers the 'pkcb macro forms, and it builds a list telling me exactly what I need to know. Then I define the core constructor again using that list in the (with ...).

I do all this just to save a little hash-table lookup, which I could have done with (or= pkcb!binding-get (pk-make-binding)). :-p If that's not important to you, I'm not surprised. It's hardly important to me, especially since it's a hash table of constant size; I mainly just like the thought of constructing the bindings eagerly and ditching the hash table altogether when I'm done.

If that's what aw meant by "Arc compiler", then I misunderstood. I took it to mean "compiling Arc code into a different language, such as assembly/C/whatever"

Given what aw said, though, eval doesn't invoke the Arc compiler... once read-time is over, the data structure is in memory so eval can just use that; it doesn't need to do any more parsing.

Ah, okay. The semantic differences may not exist then. ^_^

What you referred to as compile phase I refer to as read-time, since I think that's more specific. In any case, everything passes through read/eval, which is actually exposed to Arc, unlike MzArc where eval is just a wrapper.

I don't know about calling it read-time. Arc being homoiconic and all, there's a phase where a value is read into memory from a character stream, and there's a phase where a value is converted into an executable format, expanding all its macro forms (and in traditional Arcs, expanding ssyntax).

Read time is kinda taken, unless (read) really does expand macros. If compile time isn't specific enough for the other phase--and I agree it isn't--how about calling it expansion time?

---

"In any case, everything passes through read/eval, which is actually exposed to Arc, unlike MzArc where eval is just a wrapper."

I don't know what consequences to expect from that, but I like it. It seems very appropriate for an interpreter. ^^

---

"Thus, implementing an additional check in eval could add up, if I'm not careful."

Calling it expansion time isn't really correct because it doesn't always expand; sometimes it just takes data like 10 or (foo bar) and converts it into the equivalent data structure.

---

"I don't know what consequences to expect from that, but I like it. It seems very appropriate for an interpreter. ^^"

Is there something specific you don't like about Google Sites?

Some of the things I was looking for:

- no markdown. I despise markdown syntax (when writing about code) with the myriad special syntax that I can never figure out how to escape.

- a way to plug in code examples from my upcoming "pastebin for examples" site, which I hope I'll be able to create a Google gadget to do.

- a clean layout without a lot of junk gunking up the page

You're going to have modules? Do tell. Something like CommonJS modules, maybe? ^_^

---

1. Needs to be able to define prefix, infix, and suffix versions for the same character.

What do you intend to do with "a!!!b"?

Also, what do you plan to do about precedence? Currently a.b has higher precedence than a:b, but many of us prefer for a:b.c to mean (a (b c)) rather than (compose a (b c)) (http://arclanguage.org/item?id=13172, http://arclanguage.org/item?id=3174), so in Penknife (http://arclanguage.org/item?id=13071) I've made an everything-is-left-associative-with-the-same-precedence approach. But by doing that I also sacrifice prefix notation, and I have postfix notation only as a degenerate case of infix (with an empty right side).

---

2. Needs to have separate ssyntax per module, so ssyntax defined in one module doesn't interfere with another module.

That should be easy enough if you already have a plan for modules. Modules just need to keep both an everyday variable namespace and an ssyntax namespace. In fact, I think ssyntax could be implemented with regular globals, something like Penknife's approach.

---

This could be as simple as having a ssyntax-argument-rules global, in addition to ssyntax-rules* but I'm not sure if I want to go that route. Ideas?*

Oh, right, I think that's the typical Arc way to set up an extensible framework. Consider the 'setter, 'templates, 'hooks, and 'savers* tables, as well as Anarki's 'defined-variables, 'vtables, and 'pickles* tables, all defined in arc.arc. (On the other hand, consider that macros aren't kept in a 'macros* table.)

Yes, but not like CommonJS. I was going to save the explanation for when it's released, but I suppose I might as well tell my plan now.

Here you go: http://arclanguage.org/item?id=13977

> "What do you intend to do with "a!!!b"?"

It would be treated as 3 infix calls. So if ! expands to (l 'r) then it would be ((a '!) 'b) That may not be a good idea. By the way, a!!!!b would be ((a '!) '!) b

As for precedence, that's why I made it an alist rather than a table: it's sorted, so I can compare it in a top-to-bottom fashion. In other words, precedence is determined by the item's index within the alist.

> "That should be easy enough if you already have a plan for modules."

  server {
    listen   80 default;
    server_name  readwarp.com alias www.readwarp.com;
    ..
  }

That's because ((fn (nil) nil) 10) is supposed to be the same as ((fn (()) nil) 10), which destructures the first 0 elements from 10, treating it as a degenerate cons list.

I actually use this behavior in practice, 'cause it's useful to have at least some parameter syntax that doesn't bind any variables, rather than just using a conventional name like "ignored" or "_". Most of the time I use it in a much-too-big 'withs block, where I abuse one of the bindings for sanity-checking or other side effects. ^^ I wouldn't worry about it too much, though.

On the other hand, nil's behavior makes it more verbose to determine whether something's usable as a local variable; you have to check (and x (isa x 'sym) (~ssyntax x)). In Lathe, I define 'anormalsym to do just that. (It might be more accurate to change it to (and (isa x 'sym) (~in x nil t 'o) (~ssyntax x)).)

Speaking of (~ssyntax x), ssyntax is usable for local variable names, but naturally, you can only get the values from the Racket side:

  arc> (let a.b 4 a.b)
  Error: "reference to undefined identifier: _a"
  arc> (let a.b 4 ($ a.b))
  4

nil is such a weird value... it's false, a symbol, and an empty list all at once. I actually had to do some hacky stuff to get () and '() and nil to behave properly.

"Speaking of (~ssyntax x), ssyntax is usable for local variable names, but naturally, you can only get the values from the Racket side:"

  (is 'nil nil) -> nil

On the other hand, (listp nil) should be true. In fact, its precise definition should probably be (and is, or is equivalent to) this:

  (def listp (x)
    (or (is x nil)
        (and (acons x)
             (listp (cdr x)))))

Actually, I could represent nil as an actual 'cons cell, so that assigning to the car or cdr would work. Crazy? Probably. Especially since nil is a singleton and you can't create more nil's, so it would be a global change.

That's a bit crazy. :)

PicoLisp does something reminiscent. Every one of its data structures (numbers, symbols, nil and conses) is implemented using the low-level cons cell structure (i.e. a pair of machine words). [1] They talk about nil's representation fulfilling its dual nature as both a symbol whose value is nil and a list whose car and cdr are nil; both the symbol predicate and the list predicate return true when applied to nil:

  : (sym? NIL)
  -> T
  : (lst? NIL)   
  -> T

--

[1] http://software-lab.de/doc/ref.html#vm

[2] http://software-lab.de/doc/ref.html#nilSym

--

Not sure about empty tables, though... maybe it would be fine to treat those as nil. We already treat an empty list as nil, after all.

Actually, what you're suggesting is very similar to how languages like JavaScript and Python do it, with 0 null undefined NaN "" etc. being falsy. So in JS, you can do this:

  var foo = 0;
  if (foo) {} // will not be evaluated

  foo = "";
  if (foo) {} // will not be evaluated

  foo = NaN;
  if (foo) {} // will not be evaluated

Also, if = doesn't work, assign probably would:

  (assign nil '())

  (assign nil 'foo) -> foo
  nil               -> foo
  'nil              -> nil
  ()                -> ()
  '()               -> nil
  (assign nil '())  -> nil
  (is nil ())       -> t
  

I too have sometimes wished for that in JavaScript, but let me tell you a little story. I was writing a syntax highlighter, and got it working fine in Chrome and Firefox 3.5, but there was a bug in Firefox 3.0.

You see, I was using this bit of code here:

  output.push(text.slice(curr.index[1], next && next.index[0]));

But in Firefox 3.0, the slice method behaves differently depending on whether you pass it `undefined`, or don't pass it any arguments. So, I had to use this instead:

  if (!next) {
      output.push(text.slice(curr.index[1]));
  } else {
      output.push(text.slice(curr.index[1], next.index[0]));
  }

Consider this hypothetical example in Arc:

  (your-func 5 (and x y z))

By behaving differently when nil is passed in vs. not passing in an argument, you might cause the above example to break. Or perhaps it would work, but the behavior would be subtly different... introducing bugs.

By having different behavior depending on whether an argument is passed or not, you force callers to do this, instead:

  (iflet val (and x y z)
    (your-func 5 val)
    (your-func 5))

I share this sentiment. One thing we could do is have a space of hidden-from-the-programmer variables which tell you whether other variables have been bound. They can be accessed using a macro:

  (= given-prefix* (string (uniq) "-given-"))
  (mac given (var)
    ; NOTE: I don't think this will work properly for nil, but nil is
    ; never a local variable name anyway.
    (sym:+ given-prefix* var))

---

"Maybe doing destructuring on rest args would help solve that problem in most cases?"

  (def test args
    (if (assoc 'c args)
          (pr "c was passed")
        (pr "c was not passed")))

  ; Current options:
  
  (def test args
    (let ((o c)) args
      (pr:if (len> args 0)
        "c was passed"
        "c was not passed")))
  
  (let missing list.nil  ; This is just something unique.
    (def test ((o c missing))
      (pr:if (is c missing)
        "c was not passed"
        "c was passed")))
  
  
  ; Some hypothetical options and non-options:
  
  (def test (& (c))
    (pr "no way to tell if c was passed"))
  
  (let missing list.nil
    (def test (& (c))
      (pr "still no way to tell if c was passed")))
  
  (def test (& args)
    (let ((o c)) args
      (pr:if (len> args 0)
        "c was passed"
        "c was not passed")))
  
  (def test (& (&both args (c)))  ; Destructure twice.
    (pr:if (len> args 0)
      "c was passed"
      "c was not passed"))
  
  (def test ((o c nil c-passed))
    (pr:if c-passed
      "c was passed"
      "c was not passed"))
  
  (def test ((o c))
    (pr:if given.c
      "c was passed"
      "c was not passed"))
  
  (def test (c)  ; Parameter lists are just destructuring.
    (pr:if given.c
      "c was passed"
      "c was not passed"))
  
  (def test (&both args (c))
    (pr:if (len> args 0)
      "c was passed"
      "c was not passed"))

  (mac defreq (name args . body)
    `(w/uniq gen
       (def ,name ,(map (fn (x) `(o ,x gen)) args)
         ,@(map (fn (x) `(if (is ,x gen) (err:string "parameter " ',x " is required"))) args)
         ,@body)))

  (defreq foo (x y) (+ x y))
  (foo)     -> x is required
  (foo 1)   -> y is required
  (foo 1 2) -> 3

  (mac defreq (name vars . body)
    (if (isa vars 'cons)
          (let exp (len vars)
            `(def ,name args
               (let giv (len args)
                 (if (< giv ,exp)
                       (err:string "expected " ,exp " arguments (" giv " given)")
                     (apply (fn ,vars ,@body) args)))))
        `(def ,name ,vars ,@body)))


  (defreq foo (x y) (+ x y))
  (foo)     -> error: expected 2 arguments (0 given)
  (foo 1)   -> error: expected 2 arguments (1 given)
  (foo 1 2) -> 3
  
  (defreq foo args args)
  (foo)     -> ()
  (foo 1)   -> (1)
  (foo 1 2) -> (1 2)
  

  (defreq foo (x y . args) (list x y args)) -> error

  Welcome to Racket v5.1.
  > (define abc (read-syntax #f (open-input-string "(a b c)")))
  > (define a.bc (read-syntax #f (open-input-string "(a . (b c))")))
  > (define abc.null (read-syntax #f (open-input-string
                                       "(a b c . ())")))
  > abc
  #<syntax::1 (a b c)>
  > a.bc
  #<syntax::1 (a b c)>
  > abc.null
  #<syntax::1 (a b c)>
  > (syntax-e abc)
  '(#<syntax::2 a> #<syntax::4 b> #<syntax::6 c>)
  > (syntax-e a.bc)
  '(#<syntax::2 a> . #<syntax::6 (b c)>)
  > (syntax-e abc.null)
  '(#<syntax::2 a> #<syntax::4 b> #<syntax::6 c> . #<syntax::10 ()>)

  ; This exploits an Arc 3.1 bug to drop to Racket (as seen at
  ; http://arclanguage.org/item?id=11838). It's unnecessary on Anarki.
  (mac $ (racket-expr)
    `(cdr `(nil . ,,racket-expr)))
  
  (def discard-input ((o str (stdin)))
    " Reads all the characters from the stream, giving up partway if it
      can determine the stream would block. The return value is `nil'. "
    (while:and $.char-ready?.str readc.str))
  
  
  arc>
    (do (pr "enter something> ")
        (discard-input)
        (prn "The first character was " (tostring:write:readc) ".")
        (discard-input))
  enter something> something
  The first character was #\s.
  nil
  arc>

These utilities aren't extensible (kind of a must for me, nowadays). Furthermore, they're not built to have especially nice error messages. ^_^;

What they are is slightly more convenient, of course. A lot of them are inspired by Arc or my Penknife draft, but they're not comprehensive takes on Arc or Penknife by any means. Mostly, they save parentheses.

  ; before
  (begin 1 2 3)
  
  ; after
  (du 1 2 3)  ; "do" was taken (yeah, silly example)
  
  
  ; before
  (cond [(plan-a)  4]
        [(plan-b)  5]
        [else      6])
  
  ; after
  (ifs (plan-a)  4
       (plan-b)  5
                 6)
  
  
  ; before
  (let* ([a 1] [b 2])
    (display "hi")
    (let ([c 3])
      (+ a b c)))
  
  ; after
  (w- a 1 b 2
    (display "hi")
      c 3
    (+ a b c))
  
  
  ; before
  (foo (bar 3 (baz (qux 1 2 3))))
  
  ; after
  (: foo : bar 3 : baz : qux 1 2 3)
  
  
  ; before
  (lambda (a b) (string<? (car a) (car b)))
  
  ; after
  (: abfn : string<? (car a) (car b))
  
  
  ; before
  (let next ([a 1] [b 2])
    (next b a))  ; loop forever
  
  ; after
  (nextlet a 1 b 2
    (next b a))