The fact that Arc's implementation special-cases nil and () does make it quirky, but in these examples I think it's totally fine. IMO, nil should be seen as a variable that has been initialized to the symbol named "nil". (Arc actually treats it a bit more like a literal expression than a variable, and it makes an unnecessary and mostly invisible distinction between nil and ().)
Your "handles the case of quoted nils correctly" example doesn't have to do with the nil being quoted. Here's the same thing with it unquoted:
arc> (cons 'a (cons nil nil))
(a nil)
Since nil is a variable holding the symbol named "nil", the symbol named "nil" appears in the output, just as if you'd quoted it.
Now for why (coerce nil 'string) is "". A long time ago,[1], I used to think it made more sense to have (coerce nil 'string) be "nil", so that you could always get the name of a symbol without resorting to things like [string:or _ "nil"]. However, now I realize it's a pretty common operation to build a list of strings and concatenate the whole list at once, and Arc provides that operation through [coerce _ 'string]. Arc's just favoring one useful functionality over another here.
