Arc Forum | A proposal for a module system...

Arc Forum

A proposal for a module system...

8 points by pau 6102 days ago | 23 comments

I've been thinking about namespace stuff. And I've looked through the posts, but apart from initial criticism about not being implemented, I didn't find much (did I miss something important?). I also think that this should be addressed soon.

Although I still wouldn't know how to implement it in Scheme (yet), I have a proposal, based on Python's module system. I think it is one of the simplest:

1. You don't have to say the module you are in (like in CL's "(use-package :blah)"). So no repetitive names, 1 name 1 place.

2. You state dependencies simply by using "import" (and not having to use a separate 'system' such as ASDF, which btw I never liked at all).

3. You can cherry-pick which symbols you want to have referenced in the current module without prefix (i.e. "from blah import foo").

Based on this, here is the proposal. Three (short) commands: "use", "useas", and "use* ", and new 'ssyntax' (am I abusing this?). You "import" a module with:

  (use 'foo)

This would load file "foo.arc" (found in 'ARC_PATH') in a separate symbol table (maybe use a plain Arc table?), with proper intra-references, etc. From the current module (let's assume that we want to use 'bar' from module 'foo'), you would do:

  (foo@bar ...)

Before PG used the dot, I was going to propose it, so '@' is not my favorite (and I actually like the dot as it is in arc1, but I prefer the dot for modules, it might seem more natural to outsiders...)

To 'import' a module with a different name:

  (useas 'foo 'fu)

This helps maintaining terseness, (as in defpackage :nicknames, which I usually make very short). The nice thing is that the user gets to decide the nickname.

Finally, to select which symbols you want to 'import':

  (use* '(bar baq baz) 'foo)

I also like hierarchical module systems, so things like:

  foo@baq@x

should be possible (albeit I have to say that I certainly wouldn't like things such as "arc@widgets@radio-button-menu-item@accessible-radio-button-menu-item"... ;) ). I've also been thinking about making symbols private within a module by simply naming them like:

  (def @internal-fn (a b) ...)

There is, to some extent, a convention in CL to name internal stuff starting with '%', which I find nice. With the dot, it would be nicer IMHO, since in UNIX hidden files are in fact prefixed with a dot. I'm not so sure about really making the access to internal symbols impossible (since I like CL double-colon syntax, it allows for separate test suites, for instance). For a start, something like 'a@b@c' could translate maybe to "(lookup 'a 'b 'c)", and then make it an error that two '@' appear in a row...

8 points by mascarenhas 6102 days ago | link

What is the problem with making packages be hashtables? Then you could use foo!bar to get function bar of package foo. Like this:

(use foo) (foo!bar 1 2 3)

With renaming:

(use foo as f) (f!bar 1 2 3)

I don't think Python-like importing into the global namespace is a good idea for a system where you are supposed to be able to change things in the REPL. Once you copy a module's function to your environment you aren't able to reload the module to change the function. This also means you need a module registry, so you don't load two copies of the same module.

-----

5 points by pau 6102 days ago | link

There is no problem with that, in fact I like it... ;) But as almkglor says, this doesn't play nicely with macros...

So what you say is to simplify it to a function 'use' that creates a table of symbols? How do you solve the references within modules? I don't see that, but otherwise it's very good. I would even remove the 'as', and make it a simple optional parameter:

  (use 'foo 'f)
  (f!bar 1 2 3)

Or maybe what (use ...) can do is simply return the table:

  (set f (use "foo"))
  (f!bar 1 2 3)

-----

5 points by almkglor 6102 days ago | link

> How do you solve the references within modules?

Easy: redefine "def" from within a 'use form (possibly using kennytilton's faux special globals) such that it keeps a table of defined functions (which is what you intend to do anyway). Replace all locals with (uniq)'ed names (in case someone uses uses a function name as a local - (def foo () nil) (def bar (foo) (foo))) ). Then for each expression in the file, replace all symbols (outside of quote-forms) that match a key in the table to references to the table itself.

-----

2 points by pau 6102 days ago | link

I have to start hacking this, so that the problems become apparent. But your solution seems about right...

-----

2 points by almkglor 6102 days ago | link

The hard part is the replacing of local variables with (uniq)'ed symbols. But I think local variables should be replaced anyway, because otherwise macros will override local variables with the same name.

-----

1 point by binx 6101 days ago | link

What if a "def" form is the result of macro expansion?

-----

1 point by almkglor 6100 days ago | link

What about it? Redefine "def", right? The def that gets referenced is still the same "def" you specially defined.

-----

1 point by bogomipz 6102 days ago | link

Doesn't that mean you have to do an assignment in the normal case as well?

  (= foo (use 'foo))

-----

3 points by bramsundar 6102 days ago | link

I don't think so. Couldn't you just have use's macroexpansion add a global variable named foo?

-----

2 points by bogomipz 6101 days ago | link

Yes, I thought of that after I wrote the comment. Avoiding repetitions like this is exactly what macros are for. I guess I'm not used to thinking in lisp yet.

pau wants 'use to be a function, though, so there would have to be a separate macro, or the other way around, or something.

-----

2 points by almkglor 6101 days ago | link

  (def function-that-assigns-one-to-a-global-symbol-of-your-choice (s)
    (eval `(= ,s 1)))

-----

2 points by vrk 6100 days ago | link

Some questions to consider before a module system is implemented:

1. Why?

I can see two different use cases for modules: a) controlled sharing and reuse; b) one more tool to encapsulate functionality inside the program. Which one is it, or is it both? Neither?

2. Should modules (module names) map directly to the file system hierarchy?

Why should they? Aren't they a different concept? Would it be reasonable to have more than one module per file, or more than one file per module? And what would happen when I wanted to move files?

Why shouldn't they? Aren't they ultimately saved in files?

3. Do we really need multiple namespaces?

Only a single namespace? Consider this: let the file or module where execution started from be the namespace. Let each imported/loaded module have an anonymous, uniq'd namespace from which symbols are imported. Repeat recursively, and you only get conflicts if you explicitly import two same symbols in a module.

-----

3 points by almkglor 6099 days ago | link

1. Reuse. Encapsulation is largely unnecessary, given with and withs (although I think we need withrec).

2. no:no:no:no:no:no:no

3. What you described is still multiple namespaces, even though we are just "working" on a single one - and besides, who says we'll be working on a single namespace when the Arc Server can serve several applications?

-----

6 points by nablaone 6102 days ago | link

@ is ugly :-(

':' is de facto standard in several languages (cl,perl, erlang, etc)

maybe pg should consider using | for function composition,

composition is just like piping in shell

(map odd:car '((1 2) (4 5) (7 9))) ->

(map car|odd '((1 2) (4 5) (7 9)))

-----

2 points by pau 6102 days ago | link

I totally agree. I looked at the keyboard and I couldn't find anything else. '$' would have been even worse...

And I think you might have to hack the MzScheme reader really deeply to make it accept '|' as a conventional character, since it is used to be able to embed special characters into symbols...

-----

1 point by sp 6101 days ago | link

Yep. My margin annotations in the tutorial notes for composition read, "Kinda like pipe..."

Unless, pipe is reserved for something else why not implement it here? Assuming an english keymap is in use both the : and | require a shiftkey to enter and both use pinky finger to insert. Is the extra bit of reach an issue? Conceptually the|seems more apropos to composition and prob. more accessible to more potential adopters than the :. For me the|is an easier visual parse when scanning multiple lines of code. For me the:isn't as easy a visual parse when scanning multiple lines of code.

-----

3 points by almkglor 6102 days ago | link

Incidentally, if you're just going to use the Arc tables to store module functions into, why not just use module!name syntax? Of course, there's the slight problem if a module needs to define macros.

-----

1 point by bogomipz 6102 days ago | link

When you do (use 'foo), is the module globally known as 'foo and in the current file accessible without prefix? This seems the most straight forward since the file is named "foo.arc" and you don't want to load it multiple times when used by different modules. But then I wonder what happens when you (useas 'foo 'fu), and why is this even useful if (use 'foo) makes sure you don't need to do foo@bar. Actually, why even have the foo@bar syntax at all?

Is each source file treated as a separate module?

How about this even simpler suggestion?

  (use foo)

Loads "foo.arc" into a symbol table named 'foo, unless this is already done, and makes all of its symbols available in the current module (the current source file, or a general module 'user or whatever if done in the repl).

  (use foo bar baz (barbeque bq))

Loads module 'foo, but only imports symbols 'bar, 'baz and 'bq locally, where the latter is originally named 'barbeque. In case you just want to rename one function to avoid a name clash, and want everything else imported straight up, this should be possible as well;

  (use foo (barbeque bq) *)

-----

2 points by pau 6102 days ago | link

In Python, if you do

  import string

you still have to access symbols within string using the prefix, and I imagined Arc's (use ...) doing the same.

Arc should register in some way that the module was loaded, since if two other modules do (use 'x), then they have to reference the same table. This is, afaik, what Python does. So 'locally' (within the current module), you access the symbol table of another package with the name you want (if you don't supply one, then it's the same as in the file), but the system remembers who is who globally.

Each source file is a module, yes.

To be able to load a list of modules, like

  (map use '(string regex parser))

I think that it is important that the first parameter to the use function be a symbol, e.g. (use 'string) instead of (use string).

I like your suggestion (specially changing the names to avoid clashes)... Yes, it could be a nice way to avoid the prefix... ;)

-----

1 point by almkglor 6102 days ago | link

  (from "some-idiot-made-a-long-named-module.arc")
  [fn]  (some-idiot-made-a-long-named-module@foo test)
   Demonstrates how having to mention the module everywhere
      just makes it harder for everyone.  Accepts two symbols,
      some-idiot-made-a-long-named-module@foo2 or
      some-idiot-made-a-long-named-module@foo3
      Example:
          (some-idiot-made-a-long-named-module@foo1
            'some-idiot-made-a-long-named-module@foo2)
      See also [[some-idiot-made-a-long-named-module@foo2]]
      [[some-idiot-made-a-long-named-module@foo3]]

There's a reason why package systems such as, say, C++'s, have some feature which allows you to not mention the module name all the time.

-----

2 points by pau 6102 days ago | link

I probably didn't make myself clear... It's like in Python:

  from long-named-module import foo1, foo2, foo3

would be

  (use* '(foo1 foo2 foo3) 'long-named-module)

With this, you use (foo1 ...), etc. without the module name.

-----

1 point by almkglor 6102 days ago | link

Sorry, I've been studiously avoiding Python. Can you give better examples, such as a module being used by another module?

-----

3 points by pau 6102 days ago | link

File 'a.arc':

  (set spam 1)
  (def myfun (x) (prn x))

File 'b.arc':

  ;; Uses 'a'
  (use 'a)
  (a@myfun "hi!")

File 'c.arc':

  ;; Uses all in 'a'
  (use* 'a)
  (myfun "hy!")

File 'd.arc':

  ;; Use 'a' as 'z'
  (useas 'z 'a)
  (z@myfun "ho!")

In Arc's repl:

  arc> (use 'a)
  t
  arc> a@spam
  1
  arc> (set a@spam 5)
  5
  arc> (use 'a)
  t
  arc> a@spam
  5
  arc> (use 'd)
  ho!
  t
  arc> (use 'b)
  hi!
  t

-----