Working with UTF-8-encoded PHP files in web applications, a common, hard-to-track-down error is the following: “Headers already sent” or “Cannot modify header information“. This usually happens during a call to the function header(), which manipulates the HTTP header.
One reason for this is that the UTF-8 file starts with an invisible(!) byte order mark (BOM) consisting of the three bytes
0xEF,0xBB,0xBF. The BOM can be removed by opening the file in a suitbale text editor and unticking the Add Byte Order Mark (BOM) .option (or similar).
A more convenient way using sed is the following:
sed -i '1 s/^\xef\xbb\xbf//' utf8_file.txt
(-i enables in-place operation of sed; 1 denotes that one replacement should happen; ^ denotes the start of a line)
Let’s consider a file consisting of two lines (‘A’, ‘B’) stored with the BOM:
Investigating this file with the hex tool od, :
$ od -t c -t x1 testfile.txt
we obtain the following output:
0000000 357 273 277 A \n B \n ef bb bf 41 0a 42 0a 0000007
The three BOM bytes are clearly visible.
sed -i '1 s/^\xef\xbb\xbf//' testfile.txt
The output looks as follows, proving that the BOM is gone:
0000000 A \n B \n 41 0a 42 0a 0000004
- Original source: stackoverflow.com