Re: OT: character encodings

[Tilman Schmidt <tilman@imap.cc>]

Am 07.01.2007 18:06 schrieb Russell King:
> 
> $ git log | head -n 1000 | tail -n 200 > o
> $ file -i o
> o: text/plain; charset=us-ascii
> $ git log | head -n 1000 | tail -n 300 > o
> $ file -i o
> o: text/plain; charset=us-ascii
> $ git log | head -n 1000 | tail -n 400 > o
> $ file -i o
> o: text/plain; charset=utf-8
> 
> (and you know what charset the file is thought to have with all 1000
> lines in it.)

What the "file" command thinks is hardly relevant here. "file" just
attempts to guess what the contents of a file might be, by applying
a simple set of heuristics. Your results only highlight the actual
problem: "git" is apparently unable to handle character sets properly
and instead produces a mix of encodings as output.

> All on a system with LANG set to en_GB (iow ISO-8859-1).

For software with proper multilingual support, that should have been
enough to make sure that all its output would be in iso-8859-1, too.
Obviously "git" doesn't fall into that category.

>> Yes. When you stored it on disk, the character set information was lost.
> 
> The same thing actually happens when I look at it via:
> 
>   $ git log | head -n 1000 | less

The loss has happened long before you run that command, when the
data was committed into "git".

> So, I think you'll find that the contents of git _is_ an ad-hoc collection
> of character sets which people happen to have in use on their machines.

Exactly.

>> A mixed charset environment was _already_ a pain in the butt, because
>> almost nobody got labelling right. It's wrong to blame that on UTF-8.
> 
> I'm not talking about a mixed charset environment.  I'm talking about
> non-UTF-8 single charset environments being broken by programs which
> universally think the universe is UTF-8 only.

The problem is not programs thinking the universe is UTF-8 only; it's
people mixing different charsets, in conjunction with programs not
caring about charsets at all.

Specifically, your non-UTF-8 single charset environment was not broken
by git thinking everything was UTF-8, but to the contrary by some data
in the git repository actually being UTF-8 and git *not* thinking that.

And that problem is, I repeat, much older than UTF-8.

HTH
Tilman

-- 
Tilman Schmidt                          E-Mail: tilman@imap.cc
Bonn, Germany
Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
Ungeöffnet mindestens haltbar bis: (siehe Rückseite)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/