What do you mean by "lexical"? Module-local?

In Arc, I'd recommend you avoid using global scope for anything local (natch ^_^ ). You may prefer the ability to declare private variables or something, but a big (let ...) isn't that bad. You're even using the equivalent of a big (let ...) in JavaScript.

Also, by plox do you mean http://plox.urbanup.com/4052736? :)

---

I don't know whether it's unconventional, but it can be implemented in single-threaded Arc, and I use the name "w/global" where you use "making." If your 'making plays nicely with threads and continuations, then I guess I'd consider it strictly more expressive than Arc, but it's up to you to decide if this expressiveness is helpful.

At one point Clojure supported dynamic binding of module-level variables using (binding ...). For some reason, it looks like they changed this policy so the dynamic binding would only work with variables declared (def ^:dynamic ...).[1] So it looks like Clojure programs now use dynamic scope just as explicitly as Arc programs use Racket parameters, but with a less first-class treatment. I'm not a Clojure expert, so I could be leaving out critical details here.

No, they certainly aren't. Just like you can have dynamic/lexical function scope, you can have dynamic/lexical global scope. There's no difference between global/local. There is a difference between lexical/dynamic.

Most languages (that I know of and use) have lexical scope for functions but dynamic scope for globals. And I've found that all the issues with name collisions are traced back to this global dynamic scope.

If you instead make the global scope lexical, there are no more namespace issues.

---

"Also, by plox do you mean http://plox.urbanup.com/4052736? :)"

Yes.

---

"Do you mean allowing globals to be lexically bound?"

I mean that the default binding for globals should be lexical. Arc and JavaScript do not have lexical globals. At all. Ever. It's just not possible.

Racket's modules use dynamic binding inside the module, but lexical binding outside. This is reasonable, but Racket modules are clunky and big and complicated and such. And they disallow certain dynamic features that I like, such as assigning to a variable from another module.

Racket modules are hard to implement, hard to understand (once you get down to the nitty gritty), and they're much too rigid and staticy.

I want something that's simple to implement, simple to understand, concise, flexible and dynamic, no boilerplate, and yet has all the power of Racket's modules. Using lexical global scope + a few utilities can achieve that.

---

"Is it a bad idea?"

We're arguing semantics here, but the way I see it, "lexical" means "local to a chunk of code," and "dynamic" means "local to a run time stack frame and its children." "Global," on the other hand, means "local to nothing."

While I'd put away these definitions and accept yours for the sake of argument, I'm having trouble figuring out what yours even are, and it looks like akkartik is too.

---

"Racket's modules use dynamic binding inside the module, but lexical binding outside."

Huh? If you could explain this one sentence in terms other than "dynamic" and "lexical," maybe that explanation could clarify what your definitions are all on its own. No promises though. ^_^;

---

"And they disallow certain dynamic features that I like, such as assigning to a variable from another module."

http://akkartik.name/2012-11-11-lexical-global-scope.html

Thanks to akkartik for hosting it.

---

rocketnia: "Hmm, actually, my definition of "dynamic" scope might be "not generally possible to associate a usage site with a particular definition site at compile time." I've referred to mutability as a kind of dynamic scope before."

I'm fine with that definition.

---

rocketia: "We're arguing semantics here, but the way I see it, "lexical" means "local to a chunk of code," and "dynamic" means "local to a run time stack frame and its children." "Global," on the other hand, means "local to nothing.""

I'm not a fan of those definitions, in large part because the way I'm implementing Nulan is based on first-class environments, and thus "dynamic" means "evaluated in the environment where the function is run" and "static/lexical" means "evaluated in the environment where the function is defined".

I think that's more useful conceptually, rather than bringing in things like stack frames and a distinction between global/local: Nulan treats global environments exactly the same as local environments.

---

rocketnia: "Racket modules don't provide that feature by default, but the person defining the module does have the option to provide it. This makes it easy for the author of an existing module to introduce mutability without breaking backward compatibility. Since you like immutability by default anyway, I'm surprised you take the opposite stance for module-level variables."

Yes and I don't like having to go through those extra hoops. I also dislike the inconsistency. Which is why Nulan uses purely static environments everywhere, and it lets you assign to variables defined in another module. I think this is simpler to reason about and is more powerful too.

As for immutability... that's not correct. I don't value immutability by default. I value immutability for practical reasons. All of Nulan's datatypes (cons, object, number, environment, etc.) are immutable because I've found that to be remarkably useful. But I've come to the conclusion that making variables mutable by default (like Racket and Arc do) is perfectly fine, from a practical perspective.

And even if I did decide to make variables immutable by default, I'd still need to provide some way to easily create mutable variables in the situations where they're useful. And I'd like to allow for modules to modify these mutable variables even if they're defined in a different module.

You might be thinking, "but what if I want to write a module that doesn't allow for mutation?!" My solution is very simple:

  (var foo 5)

  ... do stuff with foo ...

  (const foo foo)

In fact, you could make this even stricter and delete the variable entirely, effectively making it private to your module:

  (var foo 5)

  ... do stuff with foo ...

  (unvar foo)

  (exclude {foo}
    (var foo 5)

    ... do stuff with foo ...)

---

akkartik: "I don't understand how a global binding can have a lexical scope if it's not attached to a block. Do you want it attached to a file/module/namespace by default?"

Sorta. In this thread[1] I already explained my views, but to summarize, I think functions should only be able to refer to variables that exist at the time the function is defined. In other words, no late binding of global variables.

Now to reply to the later parts of that email exchange...

---

Me: "Pauan, when you show code examples that demonstrate how you'd deal with certain library dependency corner cases in Nulan, are you considering that the application developer can't go in and change the code of the existing libraries?"

Pauan: "Yes, definitely."

Then I'll respond to your examples individually.

Regarding "two libraries [...] use the same namespace and have name collisons," we've already started to talk about this case, so I'll skip it here and address it below.

Regarding "two libraries, each of which use different, incompatible versions of some third library," the code you posted doesn't comprise a solution. How do you keep library1 and library2 from using the same version of library3?

Regarding "they'd like the new library to be able to occupy both orignal spaces," your code is again incomplete. If the two originals are library1 and library2, an intermediate library library3 depends on library1, and the new one is library4, then how do you get library3 to start using library4 instead of library1? Keep in mind you can't modify any of the libraries.

Regarding the licensing example with "they'd like it to be a drop-in replacement [...] An application I'm working on uses two libraries, one of which requires the replacement library [...] The other [...] uses the original basically by default": I think they're asking for a mechanism that lets the application choose whether the "other" library uses the original or the replacement. I don't see how your code makes this possible, and your "this could instead import replacement" comment suggests that you'd actually modify library2.

Regarding "importing a package should only introduce the names that library explicitly exports," I agree you've got the important parts of this covered.

Regarding "conrol the names you import [...] such that [...] a new export to an existing library will not affect the meaning," it looks like you do have a good response to this. Personally, I wouldn't want there to be any imports that could possibly break that way, but I guess I'm more picky than they are. :)

---

Pauan: "That's the point of lexical scoping: they can use the same names and not collide with eachother."

You missed my point. If the two libraries really use the same namespace, your code for importing both of them will turn into something like this:

  (import library1)
  (import library1)
  ...

I'm assuming the challenge was something like this:

  # library1
  (import library3-version1)
  
  # library2
  (import library3-version2)
  
  # application
  (import library1)
  (import library2)

---

rocketnia: "Regarding "they'd like the new library to be able to occupy both orignal spaces," your code is again incomplete. If the two originals are library1 and library2, an intermediate library library3 depends on library1, and the new one is library4, then how do you get library3 to start using library4 instead of library1? Keep in mind you can't modify any of the libraries."

The challenge was the following:

  # library1
  (def foo ...)
  
  # library2
  (def bar ...)
  
  # library3 = library1 + library2
  (def foo ...)
  (def bar ...)


  # application1
  (import library1)
  
  # application2
  (import library2)
  
  # application3
  (import library3)

There's no need to change application1 and application2: they can just keep using library1 and library2. You might say that's inefficient and you'd rather them use library3, but that's up to them whether they want to switch or not. I'm also imagining the situation that library3 actually does cause some backwards compat issues, in which case it would be desirable to keep using library1/library2. The application chooses which libraries to import.

You seem to be suggesting some mechanism in which a module can actually modify what a different module imports and exports. I fail to see the benefit in that: that module's code is written under the assumption that its imports won't be changing. Could you provide an example where that would be useful?

---

rocketnia: "Regarding the licensing example with "they'd like it to be a drop-in replacement [...] An application I'm working on uses two libraries, one of which requires the replacement library [...] The other [...] uses the original basically by default": I think they're asking for a mechanism that lets the application choose whether the "other" library uses the original or the replacement. I don't see how your code makes this possible, and your "this could instead import replacement" comment suggests that you'd actually modify library2."

No I do not think that's what they're asking. I think what they're asking is: can the original library coexist at the same time as the replacement, even though they have name collisions. Notice they used the word "reconcile". This suggests reconciling the name collisions. There's no need to "reconcile" the other library to use the replacement: it can just keep using the original.

But if I'm wrong and they were in fact not talking about name collisions at all but about the orthogonal issue of replacing another module's imports... well, no, Nulan doesn't let you do that, because I don't see any useful benefit from doing so.

In any case, that's a completely different issue from avoiding name collisions, and seems to me like it'd be a part of some sort of build system or dependency resolution system, like RubyGems. That might be useful, but I don't see it as being a part of the namespace system per se.

---

rocketnia: "Regarding "conrol the names you import [...] such that [...] a new export to an existing library will not affect the meaning," it looks like you do have a good response to this. Personally, I wouldn't want there to be any imports that could possibly break that way, but I guess I'm more picky than they are. :)"

Yup, I try to be pretty laid back (at least about my code). If you're anal about it, you can use "include", but I think it's nice to have the option of just doing unqualified imports. It's up to the library/application author, in my opinion.

---

rocketnia: "You missed my point. If the two libraries really use the same namespace, your code for importing both of them will turn into something like this:"

Nope, I don't think namespace == module. You are importing the same module twice, but it's in the same namespace. And if you did this:

  (import library1)
  (import library2)

You might say this is undesirable. I agree it can be undesirable in certain situations, and I think the solution is to import each library in a separate namespace. But hyper-static scope can be implemented with a single namespace, so whether it's single-namespace or multi-namespace is an orthogonal issue.

---

rocketnia: "I often consider what would happen if two versions of a single library were both useful in a single application. If the library author didn't anticipate and avoid self-conflict, the two versions would host a train wreck of collisons on every single name, even the library name. I think that's the kind of challenge Casey McMann has in mind here."

In that case whichever one is defined last "wins", and you can use "include" and "exclude" to control which ones you want. I could also provide a way to import a library into a separate namespace to solve that case:

  (import-as library1 lib1)
  (import-as library2 lib2)

  (lib1 'bar)
  (lib2 'bar)

I'm assuming the example is supposed to look like this instead:

  ; library1 (coded with library3's version 1 in mind)
  (import library3)
  
  ; library2 (coded with library3's version 2 in mind)
  (import library3)

---

"You seem to be suggesting some mechanism in which a module can actually modify what a different module imports and exports. I fail to see the benefit in that: that module's code is written under the assumption that its imports won't be changing. Could you provide an example where that would be useful?"

Here's Casey McCann's full description of the challenge again, just for clarity's sake:

Casey McCann: "At some point, two popular libraries with related functionality get tired of duplicating effort, so they collaborate to produce a third library providing the best of the shared features. To avoid unnecessarily breaking client code with the new versions, they'd like the new library to be able to occupy both original namespaces, or otherwise be easily substituted in. How would the namespace system handle this?"

I had it wrong before, when I made the sweeping statement that only the application's code should have to change in these challenges. This challenge is a case where the library developers are trying to change their code without causing disruption for anyone else.

Ideally, the only thing an application developer should have to do to upgrade is to replace the existing two libraries with the new one. This is even something that an end user might handle using a package manager.

---

"I think what they're asking is: can the original library coexist at the same time as the replacement, even though they have name collisions. Notice they used the word "reconcile". This suggests reconciling the name collisions. There's no need to "reconcile" the other library to use the replacement: it can just keep using the original."

I think the point is that the library has been upgraded, and the library developer would like the upgrade to propagate to all code that uses the library, even though a few library users might actually prefer otherwise. This is a conflict of preferences, and the language might give tools to certain participants to "reconcile" this conflict.

I prefer to view this as a last resort. The module system should give a complete(-feeling) range of freedom to every participant, and the ranges should not overlap. I would allow upstream developers to propagate their changes only as far as downstream developers have opted in. This turns into a design for package versioning.

---

"Nope, I don't think namespace == module."

I agree. The challenges are supposed to be realistic cases where name collisions would actually occur and cause havoc, so naturally they mostly involve programs made up of parts that have been written by multiple people independently. So we should expect to consider modules here, but yeah, the modules don't have to correspond one-to-one with namespaces.

However, I think Casey McCann is assuming a namespace system built on implicit name prefixes, which may or may not incorporate a way to take a name prefix and find a module that addresses it. While this may be a fine way to think of namespaces Java, Racket, and Haskell, the challenges which arise from this assumption may not apply to Nulan. No worries. :)

That said, if you change Nulan's design in order to accommodate some of these other challenges, you may come back to this one and find you've broken it. That is to say, this is still a meaningful test case even if it passes at the moment.

---

"My point is that even if a language doesn't provide include/exclude/import-as, it'd still be better off using hyper-static rather than nothing at all."

Well, that either won't work (it would load the same "library3" in both cases), or it would work with a sufficiently smart lookup algorithm.

Basically, when importing a file, I think it should look for it in this order: the cwd, the folder (and subfolders) where the importee is defined, then lastly the folder where Nulan itself is defined.

In that case, assuming library1 and library2 are in different folders and have different versions of library3 in their path, it would work.

This implies bundling the libraries you use with your app, which you pretty much have to do anyways.

In that case, this has nothing to do with name resolution, it has to do with file resolution: finding which file to import when given a name.

You could argue that's a good case for a package manager like RubyGem, and that may be true, but I consider such a system to be in a layer above the namespace system.

---

"This challenge is a case where the library developers are trying to change their code without causing disruption for anyone else."

And Nulan's namespace system handles that just fine.

---

"I think the point is that the library has been upgraded, and the library developer would like the upgrade to propagate to all code that uses the library, even though a few library users might actually prefer otherwise. This is a conflict of preferences, and the language might give tools to certain participants to "reconcile" this conflict."

I still don't get the impression they were talking about some sort of package management or version dependency resolution system. It's certainly okay to discuss such things, but I think they're outside the scope of a namespace system. That thread (and post) seemed to me to be talking specifically about resolving naming conflicts.

I think the talk about "multiple libraries" and "multiple versions" was just to create a situation where naming conflicts occur. I don't think it was meant to imply that the goal is to be able to easily change library versions on the fly, like you were discussing.

If I wanted some sort of system like that in Nulan, it wouldn't be particularly hard to do... you'd create a new file, like say... "dependencies.nu" and you'd then put a bunch of imports into it:

  (import foo)
  (import bar)
  (import qux)

---

"However, I think Casey McCann is assuming a namespace system built on implicit name prefixes, which may or may not incorporate a way to take a name prefix and find a module that addresses it."

That is quite possible.

---

"By "nothing at all," do you mean a late-bound global scope like Arc's?"

You: "And Nulan's namespace system handles that just fine."

What I meant was that the library developer wants our applications to use their new library version without any involvement of the application developers. I seriously doubt that's what you're claiming to support.

---

"You could argue that's a good case for a package manager like RubyGem, and that may be true, but I consider such a system to be in a layer above the namespace system."

I think the only place we really disagree now is that you're content to neglect the "RubyGem" parts of these challenges for the purposes this discussion, while I consider them the hardest parts. I think if you do try to design a package manager, you may very well get most of a namespace system out of the deal, or even a whole programming language.

Unfortunately, package managers and filesystems aren't quite a full layer above the languages they're used with. Program units tend to link together using hardcoded filenames or package names, a practice that ties some of the filesystem or package manager complexity into the complexity of the language proper.

The module system I've been working on recently (ever since [1]) is meant to help resolve that, by making the non-language-specific complexity as simple and universal as possible. It's based on two notions: A program is a multiset of independently maintained extensions, and it exists in the context of an expansive historical record containing all knowledge ever discovered (or at least all knowledge we have on hand).

The extensions and units of background knowledge may ultimately be encoded as files and managed at a "layer above" using traditional package managers and filesystems. However, they may also be manipulated as first-class values in a host program, or even as visual elements in a UI.

As an argument in favor of this viewpoint, I'll quote the original challenge:

"Now I'm writing a piece of code that uses two libraries, each of which use different, incompatible versions of some third library, causing namespace collisions between the two versions. How does the namespace system handle this?"

The challenge seems to imply that the two incompatible libraries were loaded successfully: it's NOT talking about file resolution. It's assuming the file resolution proceeded successfully, and thus the only issue remaining is resolving the name collisions.

From what you quoted, technically that could be true.

---

"You keep talking about things like dependency resolution systems, swapping in different libraries, etc. That's fine, but that does not seem to be what the challenges are talking about."

Objection! :)

I don't get the impression they're saying, "we release this new library and all the users of the existing libraries automatically switch to it". I think what they're saying is, "this library can be used by existing applications, but they still need to choose to use it". That's how I interpreted the "drop-in replacement" phrase.

Technically speaking Nulan can support that... you would change library1 and library2 to just import library3:

  # library1
  (import library3)
  
  # library2
  (import library3)
  
  # library3
  (def foo ...)
  (def bar ...)

---

"I think the only place we really disagree now is that you're content to neglect the "RubyGem" parts of these challenges for the purposes this discussion, while I consider them the hardest parts. I think if you do try to design a package manager, you may very well get most of a namespace system out of the deal, or even a whole programming language."

They are the hardest parts, I just don't think it has to do with namespace support, which that entire topic was devoted to. Nor am I particularly interested in designing a package manager.

So my suggestion still stands: if you feel this strongly about package managers, make up your own semi-contrived challenges that try to demonstrate what a package manager should be capable of doing. I'm sure it will include all kinds of stuff that was left out of the namespace challenges, because package managers are much more complex and involved than namespaces.

Then you can point to those challenges and I can say, "Nulan doesn't support that stuff" rather than trying to insist that Nulan is failing the namespace challenges because you're interpreting them far larger than what they actually said. I think it's useful to have "package manager challenges", but I think it's also useful to have "namespace challenges". I don't see why we need to throw out the namespace challenges just because you view it as "not as hard" as package managers.

---

"The challenge we're talking about puts the words "different, incompatible versions" in italics. If direct substitution were unimportant, the challenge would not even need to use the word "version"; it could just say there were two separate libraries with name collisions. And yet that's precisely the scenario of the previous challenge."

Like I said, I don't get the impression they're talking about package managers. I think they used the phrase "different incompatible versions" because that's a situation that can actually occur in real life. They're using it as an example, that's all.

Oh, I can understand that point of view. Just because there were multiple examples doesn't mean each one introduced new requirements.

---

"It's obvious that you want to think in terms of package managers, and that's fine, but I really do not get the impression that's what they're talking about."

I just wanted to help you elaborate on the claims you're making about Nulan's namespace system. You originally quoted Casey McCann's examples, but I perceived them differently than you did, and I had trouble understanding your perception from your terse example code.

Now this seems to be resolved. We may not agree in our interpretations, but now I understand your claims relative to both of them. Thanks for your patience with me. ^_^

---

"Ultimately, it seems the only way to resolve this is to actually ask them what they meant."

---

akkartik: "Thanks! I find myself wishing we could put Ross's long response in the 'namespace' section of the page rather than the 'meta' section."

I was conscious of that when I sent that reply. I'm not sure how to reply to a particular "subthread" when it's email. >_<

The only time I've ever seen email become a threaded tree is in mailing list archives, and I kinda assumed that had to do with automatic detection of quotations or something.

---

akkartik: "(I'm not actually expecting you to do this, just thinking aloud about how someone could maintain hygiene when managing a forum.)"