Lecture 2. Tokenization and word counts - История изменений

Katya в 19:48, 5 ноября 2016

2016-11-05T19:48:54Z

← Предыдущая версия		Версия от 19:48, 5 ноября 2016
Строка 1:		Строка 1:
			Ekaterina Chernyak, Dmitry Ilvovsky

	== How many words? ==		== How many words? ==

2015-09-02T22:31:31Z

Natural Language Toolkit (NLTK)

← Предыдущая версия		Версия от 22:31, 2 сентября 2015
Строка 111:		Строка 111:

	'''NLTK tokenizers'''		'''NLTK tokenizers'''

	<code>		<code>
	In[1]: from nltk.tokenize import RegexpTokenizer,		In[1]: from nltk.tokenize import RegexpTokenizer,

2015-08-31T21:25:29Z

Heaps' law

← Предыдущая версия		Версия от 21:25, 31 августа 2015
Строка 52:		Строка 52:
	\|V\| = size of vocabulary (i.e. number of types);		\|V\| = size of vocabulary (i.e. number of types);

	''K'', ''b'' — free parameters, ~~<math>~~ K ∈ [10; 100]; b ∈ [0.4; 0.6] ~~</math>~~		''K'', ''b'' — free parameters, K ∈ [10; 100]; b ∈ [0.4; 0.6]

	== Why tokenization is difficult? ==		== Why tokenization is difficult? ==

2015-08-31T21:25:09Z

Binary classifier

← Предыдущая версия		Версия от 21:25, 31 августа 2015
Строка 95:		Строка 95:
	===Binary classifier ===		===Binary classifier ===

	A binary classifier ~~<math>~~ f : X ⇒ 0; 1 ~~</math>~~ takes input data ''X'' (a set of sentences) and decides EndOfSentence (0) or NotEndOfSentence (1).		A binary classifier f : X ⇒ 0; 1 takes input data ''X'' (a set of sentences) and decides EndOfSentence (0) or NotEndOfSentence (1).

	What can be the features for classification? I am a period, am I EndOfSentence?		What can be the features for classification? I am a period, am I EndOfSentence?

2015-08-30T20:39:07Z

Heaps' law

← Предыдущая версия		Версия от 20:39, 30 августа 2015
Строка 39:		Строка 39:

	The number of different types in a text is roughly proportional to an exponent of its size:		The number of different types in a text is roughly proportional to an exponent of its size:
	~~<math>~~ \|V\| ~~= K * N^b </math>~~		[[Файл:L2 p2.jpg\|мини\|слева]]








	''N'' = number of tokens;		''N'' = number of tokens;

2015-08-30T20:34:44Z

Zipf's law ([Gelbukh, Sidorov, 2001])

← Предыдущая версия		Версия от 20:34, 30 августа 2015
Строка 31:		Строка 31:

	''r'' — rank of a type (its position in the list of all types in order of their frequency of occurrence).		''r'' — rank of a type (its position in the list of all types in order of their frequency of occurrence).

			[[Файл:L2 p1.jpg\|мини\|Zipf's law: example]]

	== Heaps' law ==		== Heaps' law ==

2015-08-23T21:41:12Z

Exercise 1.3 Do we need all words?

← Предыдущая версия		Версия от 21:41, 23 августа 2015
Строка 194:		Строка 194:

	== Exercise 1.3 Do we need all words? ==		== Exercise 1.3 Do we need all words? ==

			Stopword is a not meaningful word: prepositions, adjunctions, pronouns, articles, etc.

			'''Stopwords'''
			<code>

			In[1]: from nltk.corpus import stopwords

			In[2]: print stopwords.words('english')

			</code>

			'''Exercise 1.3'''

			Input: Alice in Wonderland (alice.txt) or your text

			Output 1: 20 most common lemmata without stop words

			Use not in operator to exclude not stopwords in a cycle.

2015-08-23T21:40:13Z

Exercise 1.2 Word counts (continued)

← Предыдущая версия		Версия от 21:40, 23 августа 2015
Строка 184:		Строка 184:

	== Exercise 1.2 Word counts (continued) ==		== Exercise 1.2 Word counts (continued) ==

			Input: Alice in Wonderland (alice.txt) or your text

			Output 1: 20 most common lemmata

			Output 2: 20 most common stems

			Use FreqDist() to count lemmata and stems. Use FreqDist().most common() to find most common lemmata and stems.

	== Exercise 1.3 Do we need all words? ==		== Exercise 1.3 Do we need all words? ==

2015-08-23T21:39:35Z

Stemming

@@ Строка 164: / Строка 164: @@
 == Stemming ==
 == Exercise 1.2 Word counts (continued) ==
 == Exercise 1.3 Do we need all words? ==

2015-08-23T21:37:02Z

Lemmatization (Normalization)

← Предыдущая версия		Версия от 21:37, 23 августа 2015
Строка 142:		Строка 142:

	== Lemmatization (Normalization) ==		== Lemmatization (Normalization) ==

			Each word has a base form:
			* has, had, have ⇒ have
			* cats, cat, cat's ⇒ cat
			* Windows ⇒ window or Windows?

			=== Lemmatization [Jurafsky, Martin, 1999] ===
			Lemmatization (or normalization) is used to reduce in ections or variant forms to base forms. A dictionary with headwords is required.

			=== Lemmatization ===

			<code>

			In[1]: from nltk.stem import WordNetLemmatizer

			In[2]: lemmatizer = WordNetLemmatizer()

			In[3]: lemmatizer.lemmatize(t)

			</code>

	== Stemming ==		== Stemming ==