str - whitespace character python
Python: splitting string by all space characters (4)
It turns out that \u200b is not technically defined as whitespace , and so python does not recognize it as matching \s even with the unicode flag on. So it must be treated as an non-whitespace character.
import re re.split(ur"[\u200b\s]+", "some string", flags=re.UNICODE)
To split strings by spaces in python, one usually uses
split method of the string without parameters:
>>> 'a\tb c\nd'.split() ['a', 'b', 'c', 'd']
split is not enough:
>>> u'a\u200bc d'.split() [u'a\u200bc', u'd']
it seems the solution suggested by
sth gererally works but depends on some OS settings or Python compilation options. It would be nice to know the reason for sure (and if the setting can be switched on in Windows).
cptphil found a great link that makes everything clear:
So I contacted the Unicode Technical Committee about the issue and received a promptly received a response back. They pointed that the ZWSP was, once upon a time considered white space but that was changed in Unicode 4.0.1
A quotation from unicode site:
Changing U+200B Zero Width Space from Zs to Cf (2003.10.27)
There have been persistent problems with usage of the U+200B Zero Width Space (ZWSP). The function of this character is to allow a line break at positions where it normally would not be allowed, and is thus functionally a format character with a general category of Cf. This behavior is well documented in the Unicode Standard, and the character not considered a Whitespace character in the Unicode Character Database. However, for historical reasons the general category is still Zs (Space Separator), which causes the character to be misused. ZWSP is also the only Zs character that is not Whitespace. The general category can cause misinterpretation of rule D13 Base character as allowing ZWSP as a base for combining marks.
The proposal is to change the general category of U+200B from Zs to Cf.
Resolution: Closed. The general category of U+200B will be changed from Zs to Cf in Unicode version 4.0.1.
The change was then reflected in Python. The result of
u'\u200B'.isspace() in Python 2.5.4 and 2.6.5 is
True, in Python 2.7.1 it is already
For other space characters regular
split is enough:
>>> u'a\u200Ac'.split() [u'a', u'c']
And if that is not enough for you, add characters one by one as
Gabi Purcaru suggests below.
Can you use something like this?
re.split(r'\s+', your_string, re.UNICODE)
You can use re.split, like this:
import re re.split(u'\s|\u200b', your_string)
You can use a regular expression with enabled Unicode matching:
>>> re.split(r'(?u)\s', u'a\u200bc d') [u'a', u'c', u'd']