Repository navigation
String instance is broken with non-ASCII data #28
Description
Activity
Ideally I think the Content-Encoding and Content-Length headers should be set by the HTTP implementation, which would give the String instance liberty to default to something like UTF-8 (modulo user configuration perhaps).
Alternatively, I would just delete the String instance and bump the version number. Applications that rely on this instance should really be fixed to do their own encoding, just recently I had the pleasure of figuring out what was causing package CouchDB to break - it was the broken String instance in here.
If I see a String instance like that I'll assume it's sane - as in actually handles all valid Strings appropriately.
I agree in principle that a broken String instance shouldn't exist, it's just that removing it now could have a very substantial impact and I'm scared of doing that. On the other hand there is all the time people like you will have wasted figuring out the brokenness.
As well as dealing with the sending aspect, don't we need to deal with any encoding the server chooses to return?
I'm in favour of doing
Stringproperly by looking atContent-Type, but removing support forStringand just usingByteStringor[Word8]or whatever is certainly better than the current situation.I've spent a while investigating and I currently think doing String properly is too hard :-( I guess removing the String instance is the only remaining option.
http://www.haskell.org/pipermail/libraries/2012-September/018426.html
You are right that if the library is handed content that does not have a
Content-Typewith acharsetalready, then it is rather hard. And, of course, if handed acharsetthe library does not support (for whatever reason) that is also pretty hard.Since in practice encodings are sometimes detected from the actual content itself (XML declarations, meta tags), there is in fact no sane simplifying assumption if
Content-Typelacks an encoding. While by-the-book such aContent-Typeshould indicate binary data, in practise it is used to deliver data with all sorts of character encodings (yay browsers magically auto-detecting things!). So, the only sane thing for a generic HTTP library that is not content aware to do in the case ofContent-Typewith nocharsetis to throw an error (or at least a very strong warning), which is likely not in the spirit of this library.Long live
ByteString.+1 for moving to
ByteString. Any progress on this?Another option is
Either ByteString String, where theRightis decoded properly, but only if the content is actually, unambigiously, text.I was really dismayed, on my very first HTTP attempt, to find that the
HTTPmodule blatantly ignoresContent-Type(and its charset) and assumes everything is a string. This will be confusing a lot of first timers; the additional work required to become encoding-aware will be fairly daunting. And among other things it makes it impossible to work with images.You are right that if the library is handed content that does not have a Content-Type with a charset already, then it is rather hard.
According to the MIME spec, content without
Content-Typeis considered byapplication/octet-stream, ie. a raw stream of 8-bit bytes. Text content that has aContent-Typebut no charset is assumed to be 7-bit ASCII, except when the particular MIME type defines a default.Sorry, I was hoping to do this much more promptly. It's a bit harder than I thought to do nicely because the entire library is based around using Strings :-(
Although the package works with String, and indeed encourages its use by having that be the default type in some of the API helper functions, the handling of non-ASCII data is completely broken. We should be doing encoding properly to be consistent with the Content-Type header, both when sending and receiving.
This has been reported by a couple of users so far and is clearly pretty nasty, but I don't think it's trivial to fix because presumably the receiving and sending sides can use different encodings and when sending requests we ought to be optionally giving the user control of the encoding.
The alternative of removing the String instances completely is likely to be very disruptive so I don't think it's a reasonable option.