Arc Forumnew | comments | leaders | submitlogin
Execute code in the scope of a hash table
5 points by mpr 3175 days ago | 8 comments
So, I am in the process of writing a macro, w/tab, that binds the keys of a hash table to their values. If we have a hash table, z, that looks like

    hash#((a . 1) (b . 2))
then w/tab will bind the variable a to 1, and b to 2. w/tab also takes a body, and executes the body in the scope of these variable bindings. So far I have this

    (mac w/tab (ht . body)
      (w/uniq bindings
        `(let ,bindings (flat (tablist ,ht))
           (eval `(with ,,bindings
                    ,@',body)))))
This works, although it would be nice to not have to do the (eval).

The next step is to update the hash table with any values that were changed in the body. Example,

    (w/tab z
      (= a 4))
After execution of this form, z should look like #hash((a . 4) (b . 2)). Here is my attempt at doing the updating,

    (mac w/tab (ht . body)
      (w/uniq (bindings k)
        `(let ,bindings (flat (tablist ,ht))
           (eval `(with ,,bindings
                    (= res (do ,@',body))
                    (each ,',k (keys ,',ht)
                      (= (,',ht ,',k) (eval ,',k)))
                   res)))))
I set the result of executing body to res, then loop through the keys of the hash table, trying to reassign their new values. The problem is that the (eval) call in the (each) loop throws an error. Apparently the expansion of (with) doesn't allow eval'ing the symbols of its bindings.

I'm stuck at this point, and would appreciate any advice on what I might do to write something that supports this behavior.

- mpr



5 points by rocketnia 3175 days ago | link

The essential trouble you'll have with this macro is that Arc macros don't know what variables are in their caller's lexical environment.

  (let a 1
    (w/tab (obj b 2)
      (+ a b)))
Since the place `b` will be looked up isn't determined unil run time, it has to use `eval`. But a call to `eval` always uses the global scope, so we lose `a`.

We could work around this by changing the way Arc's macroexpander worked. For now, let's not worry about that. I'll return to this later.

---

I see some particular bugs that can be fixed in your code.

First, you're inserting the bound values as flattened expressions in your generated code, so they'll actually be flattened executed. That is, if you write (w/tab (obj x '(+ (+ 2 y) 4)) ...), then I think `x` will be bound to the function `+`, `+` will be bound to 2, and `y` will be bound to 4.

If that's not what you want, we can make this change:

  -    `(let ,bindings (flat (tablist ,ht))
  +    `(let ,bindings (mappend [let (k v) _ `(,k ',v)] (tablist ,ht))
Second, you're assigning a nonlocal variable there, `res`. I don't know if you were doing that on purpose, but here's a version that avoids doing that:

   (mac w/tab (ht . body)
  -  (w/uniq (bindings k)
  +  (w/uniq (bindings res k)
       `(let ,bindings (mappend [let (k v) _ `(,k ',v)] (tablist ,ht))
          (eval `(with ,,bindings
  -                (= res (do ,@',body))
  -                (each ,',k (keys ,',ht)
  -                  (= (,',ht ,',k) (eval ,',k)))
  -               res)))))
  +                (let ,',res (do ,@',body)
  +                  (each ,',k (keys ,',ht)
  +                    (= (,',ht ,',k) (eval ,',k)))
  +                  ,',res))))))
Third, as you observed, the (eval ...) inside the (each ...) throws an error. What's going on is that `eval` evaluates things in the global scope, and you're trying to get a local variable. I recommend taking out the (each ...) loop and replacing it with a sequence of assignments. This means the keys you're assigning to will be determined based on the initial state of the table, rather than the final state, but that corresponds to the local variables that have been made anyway.

   (mac w/tab (ht . body)
  -  (w/uniq (bindings res k)
  +  (w/uniq (ht-list res)
  -    `(let ,bindings (mappend [let (k v) _ `(,k ',v)] (tablist ,ht))
  +    `(let ,ht-list (tablist ,ht)
  -       (eval `(with ,,bindings
  +       (eval `(with ,(mappend [let (k v) _ `(,k ',v)] ,ht-list)
                   (let ,',res (do ,@',body)
  -                  (each ,',k (keys ,',ht)
  -                    (= (,',ht ,',k) (eval ,',k)))
  +                  ,@(map [let (k v) _ `(= (,',ht ',k) ,k)] ,ht-list)
                     ,',res))))))
Fourth, this code evaluates the `ht` expression each and every time it does an assignment. This might be okay, except it's evaluating the expression in the local scope the first time and the global scope all the other times, which will probably lead to annoying errors. Let's evaluate it only once, in the local scope:

   (mac w/tab (ht . body)
  -  (w/uniq (ht-list res)
  +  (w/uniq (ght ht-list res)
  -    `(let ,ht-list (tablist ,ht)
  +    `(withs (,ght ,ht ,ht-list (tablist ,ght))
          (eval `(with ,(mappend [let (k v) _ `(,k ',v)] ,ht-list)
                   (let ,',res (do ,@',body)
  -                  ,@(map [let (k v) _ `(= (,',ht ',k) ,k)] ,ht-list)
  +                  ,@(map [let (k v) _ `(= (',,ght ',k) ,k)] ,ht-list)
                     ,',res))))))
I'm doing something tricky there: I'm inserting the table itself as a quoted value. It's possible to avoid this with a pattern like ((eval `(fn (x) (... x ...))) x), but Arc lets us just do (eval `(... ',x ...)).

Er, actually, something's making that code not work on Anarki, even though it does work on Arc 3.1. Embedding a function (instead of a table) works on Anarki, so let's do that:

  -                  ,@(map [let (k v) _ `(= (',,ght ',k) ,k)] ,ht-list)
  +                  ,@(map [let (k v) _ `(',[= (,ght k) _] ,k)] ,ht-list)
Here's the resulting code:

  (mac w/tab (ht . body)
    (w/uniq (ght ht-list res)
      `(withs (,ght ,ht ,ht-list (tablist ,ght))
         (eval `(with ,(mappend [let (k v) _ `(,k ',v)] ,ht-list)
                  (let ,',res (do ,@',body)
                    ,@(map [let (k v) _ `(',[= (,ght k) _] ,k)] ,ht-list)
                    ,',res))))))
As akkartik was saying, nested quasiquotation can be hard to get right. We can avoid nested quasiquotation if we factor the code a bit differently:

  (def w/tab-fn (ht body)
    (let ht-list tablist.ht
      (eval:w/uniq res
        `(with ,(mappend [let (k v) _ `(,k ',v)] ht-list)
           (let ,res (do ,@body)
             ,@(map [let (k v) _ `(',[= ht.k _] ,k)] ht-list)
             ,res)))))
  
  (mac w/tab (ht . body)
    `(w/tab-fn ,ht ',body))
Here it is in action:

  arc> (let a 1 (w/tab (obj b 2) (+ b b)))
  4
  arc> (let a (obj b 2) (w/tab a (= b 4)) a)
  #hash((b . 4))
  arc> (let a 1 (w/tab (obj b '(+ (+ 2 y) 3)) (join b b)))
  (+ (+ 2 y) 3 + (+ 2 y) 3)
---

As I mentioned above, there's one case that's sadly impossible to support without hacking on the language a bit:

  arc> (let a 1 (w/tab (obj b 2) (+ a b)))
  Error: "reference to undefined identifier: _a"
Okay, here's a really hackish way to add this capability to the language.

We can make this six-line modification to ac.scm:

  +(define latest-macex-env* 'nil)
  +
  +(xdef get-latest-macex-env (lambda () latest-macex-env*))
  +
   (define (ac-call fn args env)
  +  (set! latest-macex-env* env)
     (let ((macfn (ac-macro? fn)))
       ...))
  
   ...
  
   (define (ac-macex e . once)
  +  (set! latest-macex-env* 'nil)
     (if (pair? e)
       ...))
Now we can update the macro:

  (def w/tab-fn (caller-env ht body)
    (let ht-list tablist.ht
      (eval:w/uniq res
        `(with ,(mappend [let (k v) _ `(,k ',v)] tablist.caller-env)
           (with ,(mappend [let (k v) _ `(,k ',v)] ht-list)
             (let ,res (do ,@body)
               ,@(map [let (k v) _ `(',[= ht.k _] ,k)] ht-list)
               ,res))))))
  
  (mac w/tab (ht . body)
    `(w/tab-fn (obj ,@(mappend [do `(,_ ,_)] (get-latest-macex-env)))
       ,ht ',body))
Ta-da!

  arc> (let a 1 (w/tab (obj b 2) (+ a b)))
  3
We have effectively called `eval` in a local scope. We constructed the local scope table explicitly and then explicitly set it up again in the generated code.

---

If anyone would like to incorporate that six-line hack I demonstrated into Anarki, I think it will be a bit nicer if (get-latest-macex-env) were dynamically scoped rather than permanently mutated on each call. This will only ever matter if a macro invokes `eval`, `macex`, or `macex1` during its own macroexpansion, but hey, it could happen!

(This is effectively very similar to Kernel fexprs. The similarity would be even stronger if the environment were a table that mapped each variable to its local macroexpander. Arc doesn't have locally scoped macros, but I think this is a good way to go if we want them.)

-----

4 points by mpr 3174 days ago | link

Wow great post, thanks for the reply. I didn't know eval was always executed at global scope, and I was not thinking about the ht variable being executed multiple times at different scope, so thanks for pointing that out. The rest of your explanation about how to get it working is very clear and I will be a better arc hacker for it!

The occasion for this macro is an object system built around closures and hash tables. I will post more about it when I've made more progress.

-----

3 points by mpr 3173 days ago | link

Hey, could you expand on what we might do to hack local macros into arc? I'm really keen on writing macrolet

-----

2 points by rocketnia 3173 days ago | link

It would take a bit of refactoring: Changing the `env` list to a table, adding it as a parameter of `ac-macro?` (which should now look things up from that table first), and finally adding an `env` parameter to Arc's `eval`.

-----

1 point by akkartik 3175 days ago | link

I'm on the move at the moment, but since you brought up defvar yesterday I wonder if that would be a simpler way to go..

Also, it's not clear why you're using the top-level eval. I think it might be unnecessary. Double unquoting is hard to get right so worth avoiding if at all possible. Edit 2 hours later: oh, I see the reason for the eval.

-----

2 points by mpr 3174 days ago | link

I read the link you suggested in my previous post about defvar, and I see that it can be used to set dynamic behavior when a variable is referenced (??). Is this the correct interpretation, and what might be some uses of that?

-----

3 points by akkartik 3174 days ago | link

Yeah it lets you decide what to do when getting or setting a variable. The original link has an example at the bottom, but here's another one kinda related to what you seem to be trying to do:

  arc> (= h (obj a 1 b 2))
  #hash((a . 1) (b . 2))
  arc> (defvar a
               (fn args
                 (if args
                   ; write
                   (= h!a car.args)
                   ; read
                   h!a)))
  arc> a
  1
  arc> (= a 3)
  3
  arc> a
  3
  arc> h
  #hash((a . 3) (b . 2))

-----

2 points by mpr 3174 days ago | link

Oh I see now. That is pretty cool

-----