A final, more speculative change: for loops are named, so you can break and continue multiple loops at once using the name of the loop variable. Both these fragments have identical output:
(up i 0 3
(up j 0 3
(if (> j i) (break))
(prn i " " j)))
(up i 0 3
(up j 0 3
(if (> j i) (continue-i))
(prn i " " j)))
I've made a subtle change to up and down -- following C, they are no longer inclusive of their second bound. Before:
arc> (up i 0 3 prn.i)
0
1
2
3
After:
arc> (up i 0 3 prn.i)
0
1
2
I think I've caught all their callers and corrected them, but there might be stray bugs in lib/math.arc. (It's amazing how much cleaner math.arc has gotten with the saner, C-inspired bounds.)
(mac drain (expr (o eos nil))
(w/uniq (gacc gres)
`(accum ,gacc
(whiler ,gres ,expr ,eos
(,gacc ,gres)))))
I'm still concerned about changing the semantics of loop and for, especially with the prospect of updates from pg. Fortunately it's easy to roll back these changes :)
For some reason I thought for and while were both implemented using loop. Turns out only for is so that probably led to some confusion. I guess it would make sense to implement while with xloop. I do like the idea of moving the current for to up. What I meant originally was to replace the uses of loop with xloop.
I have to agree with you that the current loop should definitely allow variable declarations as it should be the general case of up/down. The remaining cases should be covered by while (unless we want to put the update and the test in the same place which I see no reason for).
Just a question about some of your code. How is while not infinitely recursive? Are macros substituted at runtime in wart?
Edit: I just realized that we can implement the current loop in terms of xloop.
Changing the names is a separate matter. We could rename the existing loop to for. Does anybody object to that? It's been baked in since the original arc, so I'm feeling a little cautious about messing with it. Any suggestions for alternative names to the existing for which only works on ranges of numbers? One idea is to call it up since there's already a down just below it that scans downward on ranges of numbers.
On the other hand, he does have a point in the bit you quoted: there's a lot of different considerations one has to deal with, and you're often doing something wrong if your focus is on architecture.
My incomplete answer at this point is to try to always record the scenarios I considered when building, in hopes that it'll keep the code easy to rewrite later on without causing regressions. By taking on that burden I hope to drop the larger burden of worrying about architecture too prematurely.
This is why almost all my thinking lately has been around how to increase the surface area that can be covered by tests: http://akkartik.name/prose (especially http://akkartik.name/post/tracing-tests). Eventually I want us to be able to easily write tests for performance, concurrency, fault tolerance, UI layout, etc., etc.
I definitely have to agree with you that being thoughtful is important. If anything that's what programming is about. Knowing which tools of those available to you are the best for each situation.
Yeah. Technical debt[1] is a real phenomenon, and it's easy to under-estimate if you just look at nascent projects. After looking at a few short-term successes you may be tempted to assume you can just do what you think they did to get their results. But that approach is susceptible to survivor bias[2]. In reality doing anything on your own is a risky endeavor, and one way to inarguably reduce risk is to get time on your side[3]. If you take on too much technical debt time starts working against you.
The real problem is that we still have only a hazy understanding of what technical debt entails. I suspect many of the things we worry about aren't the biggest sources of technical debt or incidental complexity[4]. So yes, sometimes doing a lot of conventional 'engineering' turns out to be unnecessary. But that is a case for being more thoughtful, not less.
There's a lesson there that's easy to forget--or ignore. It's extremely difficult to be simultaneously concerned with the end-user experience of whatever it is that you're building and the architecture of the program that delivers that experience. Maybe impossible. I think the only way to pull it off is to simply not care about the latter. Write comically straightforward code, as if you just learned to program, and go out of your way avoid wearing any kind of software engineering hat--unless what you really want to be is a software engineer, and not the designer of an experience.
While this makes sense for simple, throw away projects, it becomes much harder to continuously provide a good experience if you are not worried about the design of your code. Just look at HN for an example of keeping nice code making it easy to provide a good end-user experience[0][1].
Well, you need it for cons and the generator lazy-gen and other functions that don't take a stream. Maybe the others don't, but it seemed to me that sometimes it makes sense to return a list and sometimes not. So especially since functions like firstn do something useful without changing anything, maybe we should keep that behavior around. But it's just an idea. What do you think? Since you're using them for project Euler, it'll be good to hear your experiences.
In wart the name can be nicer still -- just loop since for does what arc's loop does, in the footsteps of classical C.
Edit 20 minutes later: I ended up going with recur instead of next, to form a loop.. recur pair. I also considered with loop.. recur to strengthen the connection with with for the alternating var/val syntax. (The other pair of keywords in wart is collect.. yield in place of accum acc.) https://github.com/akkartik/wart/commit/b7b822f4fb has has an example use, which had me hankering for absz's w/afn (though with a better name; http://arclanguage.org/item?id=10125)
Edit 25 minutes later: It turns out xloop nests surprisingly cleanly. This does what you expect:
loop (a 0)
loop (b 0)
prn a " " b
if (b < 5) (recur ++b)
if (a < 5) (recur ++a)
Edit 29 minutes later: Oh, loop.. recur is precisely what clojure uses!
Ok, I've turned streams into a tagged type. I had to just support them in car and cdr and carif to get common list operations to work. However, existing operations still return regular (eager) lists when you pass them lazy streams, so I followed malisper's idea of creating lazy variants that preserve laziness in the result.
I've also added unit tests for them in lib/streams.arc.t, which is my first serious attempt at using zck's nice unit-test harness with suites.
malisper, I took some liberties with your code, such as renaming the 's' prefix to 'lazy-'. I'm not attached to these things, so feel free to revert any of my changes you don't like. Thanks for a fun exercise!
I'm not sure how you're measuring overhead, but let me know if it seems slower than before.
I'm a little surprised to see 'afnwith. Anarki also has aw's 'xloop, which is the exact same thing as 'afnwith but with the "self" anaphoric variable renamed to "next".
I don't know what I was thinking. Of couse we could just tag the streams when they are created with scons. If we really wanted to we could just extend all of the basic operations for working with lists.
There still is the problem of efficiency though. For what I have been using them for (project euler) efficiency is a big deal.
Thanks for the links! Too bad none of them show example usage.. :) But no matter, I'll add some to this version. It's looking promising so far, I'll push it later today.
Yeah, especially since one of them is destructive and one isn't!
If we can get regular car and cdr to handle lazy streams then this issue will be moot. I'm investigating this. I didn't understand your point 1 above regarding when to tag, so I'm trying to recreate the primes example from SICP 3.5 to better understand the issue.
Can you point me at any sample calls to your library, maybe things you tried on the commandline while you built it?
The simplest fix would be to overload scar/scdr based on the number of arguments. While it's a nice quick fix, I do not like the idea of using the same name for two different functions.
Hmm, I think you need to do more to redefine scar and scdr:
arc> (= x '(1 2 3))
arc> x
(1 2 3)
arc> (= car.x 4)
4
arc> x
(4 2 3)
arc> (load "lib/streams.arc")
*** redefining scar
*** redefining scdr
nil
arc> (= car.x 5)
scar: arity mismatch;
the expected number of arguments does not match the given number
expected: 1
given: 2
arguments...:
'(4 2 3 . nil)
5
The problem is that prior uses of scar and scdr (such as assignment to car[1]) will try to call your version. It's usually a bad idea to repurpose names for something unrelated. At the least we need to go through and fix callers of the old name.
Not a huge deal at the moment since your library isn't loaded by default. So we have some time to think about how to fix this.
That reminds me: I never did get around to adding your test harness to anarki like I promised. I'll do that if I get around to adding some lazy tests this weekend. Otherwise feel free to do so when you look at this again.
Land of Lisp also has a good discussion of this topic, but refers to it entirely as "lazy lists" and the like. I think it also convinced me that I want laziness to be almost transparent, so you don't need to have different functions to work with lazy data types versus strict ones. Of course, I'm not opposed to some sort of library adding them in a way that's elegant, even if that library isn't included in Arc.
I'll have to read this more this later, but if you aren't using a unit test library yet, I'd like to pimp mine again: https://bitbucket.org/zck/unit-test.arc/ . I'm willing to write some tests, but that will have to wait until next week, as I'm away this weekend.
Hmm, my interpretation of the socket complaint (that I quoted above) was that it was a qualitative rather than a quantitative argument.
Ah, I think I've found a way to reframe what Olin Shivers is saying that helps me make sense of it. The whole 80% thing is confusing. The real problem is solutions that don't work for the very first problem you try them out at. That invariably sucks. Whereas a solution that is almost certain to support the initial use case of pretty much anyone is going to sucker in users, so that later when they run into its limitations it makes more sense to fix or enhance it rather than to shop around and then NIH ("not invented here", or reinvent) a new solution for themselves. In this way it hopefully continues to receive enhancements, thereby guaranteeing that still more users are inclined to switch to it.
:) I'm using some critical-sounding language there ("sucker") just to help highlight the core dynamic. No value judgement implied.
---
I think both of us are starting with the same goal of helping hackers communicate better, share code better. But we're choosing different approaches. Your approach of fixing modularity is compatible with the Olin Shivers model, but I think both of you assume that receiving enhancements is always a good thing, and things monotonically improve. I don't buy that more and more hackers adding code and enhancements always improves things in the current world. There's a sweet spot beyond which it starts to hurt and turns away users who go off trying to NIH their own thing all over again.
My (more speculative) idea instead is to ensure more people understand the internals so that the escape hatch isn't NIH'ing something from scratch but forking from a version that is reasonable to them and also more likely to be understandable to others. By 'softening' the interface I'm hoping to make it more sticky over really long periods, more than a couple of generations. I don't think modularity will work at such long timescales.
"The number of dissatisfied hackers goes down. But won't they still continue to react to their dissatisfaction by creating new libraries that don't build on prior attempts?"
Who's saying they won't? If you're willing to believe that 100% solutions will lead to fewer dissatisfied hackers and less duplication of effort, I can't tell what other claims you're trying to challenge here.
---
To bring in my personal goals, I'm interested in using programming to improve the expressiveness of communication, so that we have less severity in our petty misunderstandings, our terminology barriers, etc.
While I'm interested in reducing duplicated effort or saving people from dissatisfaction, I'm mainly interested in these things because they go hand-in-hand with establishing better communication. If hackers are more productive, they can communicate more; and if they communicate more, they can find satisfactory tools and avoid duplicated effort.
---
I think it'll be more interesting to look the "80% solution" idea with a scenario that combines more than one solution, so that it's not a simple feedback loop anymore.
Suppose we have two completely unrelated projects A and B, with A being a 95% solution and B being an 80% solution.
Now let's say some hackers have goals in the A+B domain, so that people might suggest for them to use project A and/or B as part of the solution.
Only 76% are actually satisfied with A and satisfied with B at the same time. About 4/5 of the remaining hackers can still build on top of A, but they have to reinvent the functionality they hoped to get from B.[1]
Unfortunately, sometimes it's very difficult to combine the two dependencies A and B in a single program. If A and B involve different operating systems, different programming languages, etc., then the task of combining them may be even more difficult than reinventing the wheel. In these cases, the 76% quadrant of happy hackers must redistribute itself among the other quadrants.
How does it get distributed? Well, the user community itself is a feature of the system, so I'm assuming a 95% solution tends to have a more helpful community than the 80% solution by definition. We also haven't considered any reason that a helpful hacker would make a different technology choice than an unhelpful one, so I'll make the simplifying assumption that all communities have the same population-to-helpfulness ratio.[2] So I'd guess the 76% "+A+B" quadrant is necessarily redistributed to the 19% "+A-B" and 4% "-A+B" quadrants while roughly preserving that 19:4 ratio.
+A -A
+B 0% 17%
-B 82% 1%
This fuzzy reasoning suggests that if an 80% solution B is incompatible with a 95% solution A, approximately 83% of hackers won't use B to achieve their A+B goals. So we see, an 80% solution can leave the vast majority of hackers dissatisfied.
But 80% solutions must be acceptable in our software, or else all our software will be monolithic projects that take lots of resources to develop and maintain, and we won't be communicating very effectively.
Personally, one thing I take away from this is that the world could really use a near-100% approach to modularity, so that almost any two well-designed projects A and B can be used together. I've been working toward this.
(However, I also think this could backfire to some degree. Reusable software that doesn't bit-rot is in a way immortal, and certain immortal software may get in the way of or add noise to our communication.)
---
[1] While this 4/5 may look like it corresponds to the 80% solution, that's just a coincidence. This 4/5 roughly comes from the remaining 20% and 5% that each solution doesn't cover. We'd get a similar 4/5 result if we compared a 96% solution with a 99% solution.
[2] Some reasons this might be wrong: A big community invites people to use it as a symbol for inclusion-exclusion politics, even if that's non-sequitur with the original purpose of the community. Sometimes technology has lots of users not because its community is helpful or political, but because few people have the kind of expertise it would take to reinvent this wheel at all (not to mention how many people don't even know they have a choice).
I love rosetta code. I spent a few weeks adding samples for my arc-like dialect: http://rosettacode.org/wiki/Category:Wart. I'd meant to add samples for arc as well, but never got around to it. Glad to see someone else taking charge!
