iob-lecture-tags-periods

text	O
data	O
is	O
very	O
special	O
.	O
in	O
contrast	O
to	O
the	O
data	O
captured	O
by	O
machines	O
such	O
as	O
sensors	O
text	O
data	O
is	O
produced	O
by	O
humans	O
.	O
and	O
they	O
also	O
are	O
meant	O
to	O
be	O
consumed	O
by	O
humans	O
.	O
and	O
this	O
has	O
some	O
interesting	O
consequences	O
.	O
because	O
it	O
is	O
produced	O
by	O
humans	O
it	O
tends	O
to	O
have	O
a	O
lot	O
of	O
useful	O
knowledge	O
about	O
people	O
s	O
preferences	O
people	O
s	O
opinions	O
about	O
everything	O
.	O
and	O
that	O
makes	O
it	O
possible	O
to	O
mine	O
text	O
data	O
to	O
discover	O
those	O
latent	O
prefaces	O
of	O
people	O
which	O
could	O
be	O
very	O
useful	O
to	O
build	O
an	O
intelligent	O
system	O
to	O
help	O
people	O
.	O
you	O
can	O
think	O
about	O
scientific	O
literature	O
or	O
so	O
and	O
it	O
s	O
a	O
way	O
to	O
encode	O
our	O
knowledge	O
about	O
the	O
world	O
.	O
so	O
it	O
s	O
very	O
high	O
quality	O
content	O
yet	O
we	O
have	O
difficulty	O
digesting	O
all	O
the	O
content	O
.	O
now	O
as	O
a	O
result	O
of	O
the	O
fact	O
that	O
text	O
is	O
consumed	O
by	O
we	O
humans	O
we	O
also	O
need	O
intelligent	O
software	O
tools	O
to	O
help	O
people	O
digest	O
the	O
content	O
or	O
otherwise	O
we	O
d	O
miss	O
a	O
lot	O
of	O
useful	O
content	O
.	O
this	O
slide	O
shows	O
that	O
the	O
human	O
really	O
plays	O
important	O
role	O
in	O
test	B
data	I
mining	O
.	O
we	O
have	O
to	O
consider	O
human	O
in	O
the	O
loop	O
and	O
we	O
have	O
to	O
consider	O
the	O
fact	O
that	O
the	O
text	O
is	O
generated	O
by	O
human	O
.	O
so	O
here	O
are	O
some	O
examples	O
of	O
useful	O
text	O
information	O
systems	O
.	O
this	O
is	O
by	O
no	O
means	O
a	O
complete	O
list	O
of	O
all	O
applications	O
.	O
i	O
categorize	O
them	O
into	O
different	O
categories	O
.	O
but	O
you	O
can	O
probably	O
imagine	O
other	O
kinds	O
of	O
applications	O
.	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
some	O
of	O
them	O
.	O
search	O
for	O
example	O
we	O
all	O
know	O
search	B
engines	I
is	O
special	O
.	O
web	B
search	B
engines	I
ipad	O
all	O
of	O
you	O
are	O
using	O
google	O
or	O
bing	O
or	O
another	O
web	B
search	B
engine	I
all	O
the	O
time	O
.	O
and	O
we	O
also	O
have	O
live	O
research	O
assistants	O
.	O
and	O
in	O
fact	O
wherever	O
you	O
have	O
a	O
lot	O
of	O
text	O
data	O
you	O
would	O
have	O
a	O
search	B
engine	I
.	O
so	O
for	O
example	O
you	O
might	O
have	O
a	O
search	O
box	O
on	O
your	O
laptop	O
.	O
all	O
right	O
to	O
search	O
content	O
on	O
your	O
computer	O
.	O
so	O
that	O
s	O
one	O
kind	O
of	O
application	O
systems	O
but	O
we	O
also	O
have	O
filtering	O
systems	O
or	O
recommended	O
systems	O
.	O
those	O
systems	O
can	O
push	B
information	O
to	O
users	O
.	O
they	O
can	O
recommend	O
useful	O
information	O
to	O
users	O
.	O
so	O
again	O
use	O
filters	O
spam	O
filters	O
.	O
literature	O
the	O
movie	O
recommenders	O
.	O
now	O
not	O
of	O
them	O
are	O
necessary	O
recommending	O
the	O
information	O
to	O
you	O
.	O
for	O
example	O
email	O
filter	O
spam	O
email	O
filter	O
this	O
is	O
actually	O
to	O
filter	O
out	O
the	O
spams	O
from	O
your	O
inbox	O
all	O
right	O
.	O
but	O
in	O
nature	O
these	O
are	O
similar	O
systems	O
in	O
that	O
they	O
have	O
to	O
make	O
a	O
binary	O
decision	O
regarding	O
whether	O
to	O
retain	O
a	O
particular	O
document	O
or	O
discard	O
it	O
.	O
another	O
kind	O
of	O
systems	O
are	O
categorization	O
systems	O
.	O
so	O
for	O
example	O
in	O
handling	O
emails	O
you	O
might	O
prefer	O
automatic	O
sorter	O
that	O
would	O
automatically	O
sort	O
incoming	O
emails	O
into	O
a	O
proper	O
folders	O
that	O
you	O
created	O
.	O
or	O
we	O
might	O
want	O
to	O
categorize	O
product	O
reviews	O
into	O
positive	O
or	O
negative	O
.	O
news	O
agencies	O
might	O
be	O
interested	O
in	O
categorizing	O
news	O
articles	O
into	O
all	O
kinds	O
of	O
subject	O
categories	O
.	O
those	O
are	O
all	O
categorization	O
systems	O
.	O
finally	O
there	O
are	O
also	O
systems	O
that	O
might	O
do	O
more	O
analysis	O
.	O
and	O
oh	O
you	O
can	O
say	O
mine	O
text	O
data	O
.	O
and	O
these	O
can	O
be	O
text	O
mining	O
systems	O
or	O
information	O
extraction	O
systems	O
and	O
they	O
can	O
be	O
used	O
to	O
analyze	O
text	O
data	O
in	O
more	O
detail	O
to	O
discover	O
potentially	O
useful	O
knowledge	O
.	O
for	O
example	O
companies	O
might	O
be	O
interested	O
in	O
discovering	O
major	O
complaints	O
from	O
their	O
customers	O
based	O
on	O
the	O
email	O
messages	O
that	O
the	O
they	O
have	O
received	O
from	O
the	O
customers	O
.	O
all	O
right	O
so	O
having	O
a	O
system	O
to	O
support	O
that	O
would	O
really	O
help	O
improve	O
their	O
productivity	O
and	O
the	O
customer	O
relations	O
.	O
also	O
in	O
business	O
intelligence	O
companies	O
are	O
often	O
interested	O
in	O
analyzing	O
product	O
reviews	O
to	O
understand	O
the	O
relative	O
strengths	O
of	O
their	O
own	O
products	O
in	O
comparison	O
with	O
competitors	O
.	O
and	O
and	O
so	O
these	O
are	O
all	O
examples	O
of	O
these	O
test	O
mining	O
systems	O
.	O
we	O
have	O
a	O
lot	O
of	O
data	O
in	O
particular	O
literature	O
data	O
.	O
so	O
there	O
s	O
also	O
great	O
opportunity	O
of	O
using	O
computer	O
systems	O
to	O
analyze	O
the	O
data	O
to	O
automatically	O
read	O
literature	O
and	O
to	O
gain	B
knowledge	O
and	O
to	O
help	O
biologists	O
make	O
discoveries	O
.	O
and	O
you	O
can	O
imagine	O
many	O
others	O
.	O
so	O
the	O
point	O
is	O
that	O
with	O
so	O
much	O
text	O
data	O
we	O
can	O
build	O
very	O
useful	O
systems	O
to	O
help	O
people	O
in	O
many	O
different	O
ways	O
.	O
now	O
how	O
do	O
we	O
build	O
this	O
systems	O
well	O
these	O
actually	O
are	O
the	O
main	O
technologies	O
that	O
we	O
ll	O
be	O
talking	O
about	O
in	O
this	O
course	O
and	O
the	O
other	O
course	O
that	O
i	O
m	O
teaching	O
for	O
this	O
specialization	O
.	O
the	O
main	O
techniques	O
for	O
building	O
these	O
systems	O
and	O
also	O
for	O
harnessing	O
the	O
text	O
data	O
are	O
text	B
retrieval	I
and	O
text	O
data	O
mining	O
.	O
so	O
i	O
use	O
this	O
picture	O
to	O
show	O
the	O
relation	O
between	O
these	O
two	O
some	O
of	O
the	O
different	O
techniques	O
.	O
we	O
started	O
with	O
big	O
text	O
data	O
right	O
but	O
for	O
any	O
applications	O
we	O
don	O
t	O
necessarily	O
need	O
to	O
use	O
all	O
the	O
data	O
.	O
often	O
we	O
only	O
need	O
the	O
small	O
subset	O
of	O
the	O
most	O
relevant	O
data	O
and	O
that	O
s	O
shown	O
here	O
.	O
so	O
text	B
retrieval	I
is	O
to	O
convert	O
big	O
raw	O
text	O
data	O
into	O
that	O
small	O
subset	O
of	O
most	O
relevant	O
data	O
that	O
are	O
most	O
useful	O
for	O
a	O
particular	O
application	O
.	O
and	O
this	O
is	O
usually	O
done	O
by	O
search	B
engines	I
.	O
and	O
so	O
this	O
will	O
be	O
covered	O
in	O
this	O
course	O
.	O
after	O
we	O
have	O
got	O
a	O
small	O
amount	O
of	O
relevant	O
data	O
we	O
also	O
need	O
to	O
further	O
analyze	O
the	O
data	O
to	O
help	O
people	O
digest	O
the	O
data	O
or	O
to	O
turn	O
the	O
data	O
into	O
actionable	O
knowledge	O
.	O
and	O
this	O
step	O
is	O
called	O
text	O
mining	O
where	O
we	O
use	O
a	O
number	O
of	O
techniques	O
to	O
mine	O
the	O
data	O
to	O
get	O
useful	O
knowledge	O
or	O
pairings	O
.	O
and	O
the	O
knowledge	O
can	O
then	O
be	O
used	O
in	O
many	O
different	O
applications	O
.	O
and	O
this	O
part	O
text	O
mining	O
will	O
be	O
covered	O
in	O
the	O
other	O
course	O
that	O
i	O
m	O
teaching	O
called	O
text	O
mining	O
and	O
analytics	O
.	O
the	O
emphasis	O
of	O
this	O
course	O
is	O
on	O
basic	O
concepts	O
and	O
practical	O
techniques	O
in	O
text	B
retrieval	I
.	O
more	O
specifically	O
we	O
will	O
cover	O
how	O
search	B
engines	I
work	O
.	O
how	O
to	O
implement	O
a	O
search	B
engine	I
.	O
how	O
to	O
evaluate	O
a	O
search	B
engine	I
so	O
that	O
you	O
know	O
one	O
search	B
engine	I
is	O
better	O
than	O
another	O
or	O
one	O
method	O
is	O
better	O
than	O
another	O
.	O
how	O
to	O
improve	O
and	O
optimize	O
a	O
search	B
engine	I
system	O
.	O
and	O
how	O
to	O
build	O
a	O
recommender	B
system	I
.	O
we	O
also	O
hope	O
to	O
provide	O
a	O
hands	O
on	O
experience	O
on	O
multiple	O
aspects	O
.	O
one	O
is	O
to	O
create	O
a	O
test	B
collection	I
for	O
evaluating	O
search	B
engines	I
.	O
this	O
is	O
very	O
important	O
for	O
knowing	O
which	O
technique	O
actually	O
worked	O
well	O
.	O
and	O
whether	O
your	O
search	B
engine	I
system	O
is	O
really	O
good	O
for	O
your	O
application	O
.	O
the	O
other	O
aspect	O
is	O
to	O
experiment	O
with	O
search	B
engine	I
algorithms	O
.	O
in	O
practice	O
you	O
will	O
have	O
to	O
face	O
choices	O
of	O
different	O
algorithms	O
.	O
so	O
it	O
s	O
important	O
to	O
know	O
how	O
to	O
compare	O
them	O
and	O
to	O
figure	O
out	O
how	O
they	O
work	O
or	O
maybe	O
potentially	O
how	O
to	O
improve	O
them	O
.	O
and	O
finally	O
we	O
ll	O
provide	O
a	O
platform	O
for	O
you	O
to	O
do	O
search	B
engine	I
competition	O
.	O
where	O
you	O
can	O
compare	O
your	O
different	O
ideas	O
to	O
see	O
which	O
idea	O
works	O
better	O
on	O
some	O
data	O
set	O
.	O
the	O
prerequisites	O
for	O
this	O
course	O
are	O
minimum	O
.	O
basically	O
we	O
hope	O
you	O
have	O
some	O
basic	O
concepts	O
of	O
computer	O
science	O
for	O
example	O
data	O
structures	O
.	O
and	O
we	O
hope	O
you	O
will	O
be	O
comfortable	O
with	O
programming	O
especially	O
in	O
c	O
	O
.	O
because	O
that	O
s	O
the	O
language	O
that	O
we	O
ll	O
use	O
for	O
some	O
of	O
the	O
programming	O
assignments	O
.	O
the	O
format	O
is	O
lectures	O
plus	O
quizzes	O
as	O
often	O
happens	O
in	O
moocs	O
.	O
and	O
we	O
also	O
will	O
provide	O
a	O
program	O
assignments	O
for	O
those	O
of	O
you	O
that	O
have	O
the	O
resources	O
to	O
do	O
that	O
.	O
we	O
don	O
t	O
really	O
have	O
any	O
required	O
readings	O
for	O
this	O
course	O
.	O
that	O
just	O
means	O
if	O
you	O
follow	O
all	O
the	O
lecture	O
videos	O
carefully	O
and	O
you	O
re	O
suppose	O
to	O
know	O
all	O
the	O
basic	O
concepts	O
and	O
the	O
basic	O
techniques	O
.	O
but	O
it	O
s	O
always	O
useful	O
to	O
read	O
more	O
so	O
here	O
we	O
provide	O
a	O
list	O
of	O
some	O
useful	O
reference	O
books	O
.	O
and	O
this	O
in	O
time	O
order	O
and	O
that	O
also	O
includes	O
a	O
book	O
that	O
and	O
i	O
are	O
co-authoring	O
now	O
and	O
we	O
make	O
some	O
draft	O
chapters	O
available	O
on	O
this	O
website	O
.	O
and	O
we	O
can	O
find	O
more	O
readings	O
and	O
reference	O
books	O
on	O
this	O
website	O
.	O
finally	O
and	O
this	O
is	O
the	O
course	O
schedule	O
.	O
that	O
s	O
just	O
the	O
top	O
of	O
the	O
map	B
for	O
the	O
rest	O
of	O
the	O
course	O
and	O
it	O
shows	O
the	O
topics	O
that	O
we	O
will	O
cover	O
in	O
the	O
remaining	O
lectures	O
.	O
this	O
picture	O
also	O
shows	O
basic	O
flow	O
of	O
information	O
in	O
a	O
text	O
information	O
system	O
.	O
so	O
starting	O
from	O
the	O
big	O
text	O
data	O
the	O
first	O
step	O
is	O
to	O
do	O
some	O
natural	B
language	I
content	O
analysis	O
because	O
text	O
data	O
is	O
in	O
the	O
form	O
of	O
natural	B
language	I
text	O
.	O
so	O
we	O
need	O
to	O
understand	O
the	O
text	O
to	O
some	O
extent	O
in	O
order	O
to	O
do	O
something	O
useful	O
for	O
the	O
users	O
.	O
so	O
this	O
is	O
the	O
first	O
topic	O
that	O
we	O
will	O
cover	O
.	O
and	O
then	O
on	O
top	O
of	O
that	O
as	O
you	O
can	O
see	O
there	O
are	O
two	O
boxes	O
here	O
.	O
those	O
are	O
two	O
types	O
of	O
systems	O
that	O
can	O
be	O
used	O
to	O
help	O
people	O
get	O
access	O
to	O
the	O
most	O
relevant	O
data	O
.	O
or	O
in	O
other	O
words	O
those	O
are	O
the	O
two	O
kinds	O
of	O
systems	O
that	O
will	O
convert	O
big	O
text	O
data	O
into	O
small	O
relevant	O
text	O
data	O
.	O
search	B
engines	I
are	O
helping	O
users	O
to	O
search	O
or	O
to	O
query	O
the	O
data	O
to	O
get	O
the	O
most	O
relevant	B
documents	I
out	O
.	O
recommender	B
systems	I
are	O
to	O
recommend	O
information	O
to	O
users	O
to	O
push	B
information	O
to	O
users	O
.	O
so	O
those	O
are	O
two	O
complementary	O
was	O
of	O
getting	O
users	O
connected	O
to	O
the	O
most	O
relevant	O
data	O
at	O
the	O
right	O
time	O
.	O
so	O
this	O
part	O
is	O
called	O
text	O
access	O
and	O
this	O
will	O
be	O
the	O
next	O
topic	O
.	O
and	O
after	O
we	O
cover	O
that	O
we	O
are	O
going	O
to	O
cover	O
a	O
number	O
of	O
topics	O
all	O
about	O
the	O
search	B
engines	I
.	O
now	O
the	O
text	O
access	O
topic	O
is	O
a	O
brief	O
topic	O
a	O
brief	O
coverage	O
of	O
the	O
two	O
kinds	O
of	O
systems	O
.	O
in	O
the	O
remaining	O
topics	O
we	O
ll	O
cover	O
search	B
engines	I
in	O
much	O
more	O
detail	O
.	O
that	O
includes	O
text	B
retrieval	I
problem	O
text	B
retrieval	B
methods	I
how	O
to	O
evaluate	O
these	O
methods	O
implementation	O
of	O
the	O
system	O
and	O
web	B
search	I
applications	O
.	O
and	O
after	O
these	O
we	O
re	O
going	O
to	O
go	O
cover	O
the	O
recommender	B
system	I
.	O
so	O
this	O
is	O
what	O
you	O
expect	O
in	O
the	O
rest	O
of	O
this	O
course	O
.	O
thanks	O
.	O
this	O
lecture	O
is	O
about	O
natural	B
language	I
content	O
analysis	O
.	O
as	O
you	O
see	O
from	O
this	O
picture	O
this	O
is	O
really	O
the	O
first	O
step	O
to	O
process	O
any	O
text	O
data	O
.	O
text	O
data	O
are	O
in	O
natural	B
languages	I
.	O
so	O
computers	O
have	O
to	O
understand	O
natural	B
languages	I
to	O
some	O
extent	O
in	O
order	O
to	O
make	O
use	O
of	O
the	O
data	O
so	O
that	O
s	O
the	O
topic	O
of	O
this	O
lecture	O
.	O
we	O
re	O
going	O
to	O
cover	O
three	O
things	O
.	O
first	O
what	O
is	O
natural	B
language	I
processing	I
which	O
is	O
a	O
main	O
technique	O
for	O
processing	O
natural	B
language	I
to	O
obtain	O
understanding	O
the	O
second	O
is	O
the	O
state	O
of	O
the	O
art	O
in	O
nlp	B
which	O
stands	O
for	O
natural	B
language	I
processing	I
.	O
finally	O
we	O
re	O
going	O
to	O
cover	O
the	O
relation	O
between	O
natural	B
language	I
processing	I
and	O
text	B
retrieval	I
.	O
first	O
what	O
is	O
nlp	B
well	O
the	O
best	O
way	O
to	O
explain	O
it	O
is	O
to	O
think	O
about	O
if	O
you	O
see	O
a	O
text	O
in	O
a	O
foreign	O
language	O
that	O
you	O
can	O
t	O
understand	O
.	O
now	O
what	O
you	O
have	O
to	O
do	O
in	O
order	O
to	O
understand	O
that	O
text	O
this	O
is	O
basically	O
what	O
computers	O
are	O
facing	O
.	O
right	O
so	O
looking	O
at	O
the	O
simple	O
sentence	O
like	O
a	O
dog	O
is	O
chasing	O
a	O
boy	O
on	O
the	O
playground	O
.	O
we	O
don	O
t	O
have	O
any	O
problems	O
understanding	O
this	O
sentence	O
but	O
imagine	O
what	O
the	O
computer	O
would	O
have	O
to	O
do	O
in	O
order	O
to	O
understand	O
it	O
.	O
for	O
in	O
general	O
it	O
would	O
have	O
to	O
do	O
the	O
following	O
.	O
first	O
it	O
would	O
have	O
to	O
know	O
dog	O
is	O
a	O
noun	O
chasing	O
s	O
a	O
verb	O
et	O
cetera	O
.	O
so	O
this	O
is	O
a	O
code	O
lexile	O
analysis	O
or	O
part	B
of	I
speech	I
tagging	I
.	O
and	O
we	O
need	O
to	O
pick	O
out	O
the	O
the	O
syntaxing	O
categories	O
of	O
those	O
words	O
.	O
so	O
that	O
s	O
a	O
first	O
step	O
.	O
after	O
that	O
we	O
re	O
going	O
to	O
figure	O
out	O
the	O
structure	O
of	O
the	O
sentence	O
.	O
so	O
for	O
example	O
here	O
it	O
shows	O
that	O
a	O
and	O
dog	O
would	O
go	O
together	O
to	O
form	O
a	O
noun	O
phrase	O
.	O
and	O
we	O
won	O
t	O
have	O
dog	O
and	O
is	O
to	O
go	O
first	O
right	O
.	O
and	O
there	O
are	O
some	O
structures	O
that	O
are	O
not	O
just	O
right	O
.	O
but	O
this	O
structure	O
shows	O
what	O
we	O
might	O
get	O
if	O
we	O
look	O
at	O
the	O
sentence	O
and	O
try	O
to	O
interpret	O
the	O
sentence	O
.	O
some	O
words	O
would	O
go	O
together	O
first	O
and	O
then	O
they	O
will	O
go	O
together	O
with	O
other	O
words	O
.	O
so	O
here	O
we	O
show	O
we	O
have	O
noun	O
phrases	O
as	O
intermediate	O
components	O
and	O
then	O
verb	O
phrases	O
.	O
finally	O
we	O
have	O
a	O
sentence	O
.	O
and	O
you	O
get	O
this	O
structure	O
we	O
need	O
to	O
do	O
something	O
called	O
a	O
syntactic	B
analysis	I
or	O
parsing	B
.	O
and	O
we	O
may	O
have	O
a	O
parser	O
a	O
computer	O
program	O
that	O
would	O
automatically	O
create	O
this	O
structure	O
.	O
at	O
this	O
point	O
you	O
would	O
know	O
the	O
structure	O
of	O
this	O
sentence	O
but	O
still	O
you	O
don	O
t	O
know	O
the	O
meaning	O
of	O
the	O
sentence	O
.	O
so	O
we	O
have	O
to	O
go	O
further	O
through	O
semantic	B
analysis	I
.	O
in	O
our	O
mind	O
we	O
usually	O
can	O
map	B
such	O
a	O
sentence	O
to	O
what	O
we	O
already	O
know	O
in	O
our	O
knowledge	O
base	O
.	O
and	O
for	O
example	O
you	O
might	O
imagine	O
a	O
dog	O
that	O
looks	O
like	O
that	O
there	O
s	O
a	O
boy	O
and	O
there	O
s	O
some	O
activity	O
here	O
.	O
but	O
for	O
computer	O
will	O
have	O
to	O
use	O
symbols	O
to	O
denote	O
that	O
.	O
all	O
right	O
.	O
so	O
we	O
would	O
use	O
the	O
symbol	O
d	O
to	O
denote	O
a	O
dog	O
.	O
and	O
b	O
to	O
denote	O
a	O
boy	O
and	O
then	O
p	O
to	O
denote	O
the	O
playground	O
playground	O
.	O
now	O
there	O
is	O
also	O
a	O
chasing	O
activity	O
that	O
s	O
happening	O
here	O
so	O
we	O
have	O
the	O
relation	O
chasing	O
here	O
that	O
connects	O
all	O
these	O
symbols	O
.	O
so	O
this	O
is	O
how	O
a	O
computer	O
would	O
obtain	O
some	O
understanding	O
of	O
this	O
sentence	O
.	O
now	O
from	O
this	O
representation	O
we	O
could	O
also	O
further	O
infer	O
some	O
other	O
things	O
and	O
we	O
might	O
indeed	O
naturally	O
think	O
of	O
something	O
else	O
when	O
we	O
read	O
text	O
.	O
and	O
this	O
is	O
call	O
inference	B
.	O
so	O
for	O
example	O
if	O
you	O
believe	O
that	O
if	O
someone	O
s	O
being	O
chased	O
and	O
this	O
person	O
might	O
be	O
scared	O
.	O
all	O
right	O
.	O
with	O
this	O
rule	O
you	O
can	O
see	O
computers	O
could	O
also	O
infer	O
that	O
this	O
boy	O
may	O
be	O
scared	O
.	O
so	O
this	O
is	O
some	O
extra	O
knowledge	O
that	O
you	O
would	O
infer	O
based	O
on	O
some	O
understanding	O
of	O
the	O
text	O
.	O
you	O
can	O
even	O
go	O
further	O
to	O
understand	O
the	O
why	O
the	O
person	O
said	O
this	O
sentence	O
.	O
so	O
this	O
has	O
to	O
do	O
with	O
the	O
use	O
of	O
language	O
.	O
all	O
right	O
.	O
this	O
is	O
called	O
pragmatic	O
analysis	O
.	O
in	O
order	O
to	O
understand	O
the	O
speech	O
actor	O
of	O
a	O
sentence	O
all	O
right	O
we	O
say	O
something	O
to	O
basically	O
achieve	O
some	O
goal	O
.	O
there	O
s	O
some	O
purpose	O
there	O
and	O
this	O
has	O
to	O
do	O
with	O
the	O
use	O
of	O
language	O
.	O
in	O
this	O
case	O
the	O
person	O
who	O
said	O
the	O
sentence	O
might	O
be	O
reminding	O
another	O
person	O
to	O
bring	O
back	O
the	O
dog	O
.	O
that	O
could	O
be	O
one	O
possible	O
intent	O
.	O
to	O
reach	O
this	O
level	O
of	O
understanding	O
we	O
would	O
require	O
all	O
these	O
steps	O
.	O
and	O
a	O
computer	O
would	O
have	O
to	O
go	O
through	O
all	O
these	O
steps	O
in	O
order	O
to	O
completely	O
understand	O
this	O
sentence	O
.	O
yet	O
we	O
humans	O
have	O
no	O
trouble	O
with	O
understand	O
that	O
.	O
we	O
instantly	O
will	O
get	O
everything	O
and	O
there	O
is	O
a	O
reason	O
for	O
that	O
.	O
that	O
s	O
because	O
we	O
have	O
a	O
large	O
knowledge	O
base	O
in	O
our	O
brain	O
and	O
we	O
use	O
common	O
sense	O
knowledge	O
to	O
help	O
interpret	O
the	O
sentence	O
.	O
computers	O
unfortunately	O
are	O
hard	O
to	O
obtain	O
such	O
understanding	O
.	O
they	O
don	O
t	O
have	O
such	O
a	O
knowledge	O
base	O
.	O
they	O
are	O
still	O
incapable	O
of	O
doing	O
reasoning	O
and	O
uncertainties	O
.	O
so	O
that	O
makes	O
natural	B
language	I
processing	I
difficult	O
for	O
computers	O
.	O
but	O
the	O
fundamental	O
reason	O
why	O
the	O
natural	B
language	I
processing	I
is	O
difficult	O
for	O
computers	O
is	O
simple	O
because	O
natural	B
language	I
has	O
not	O
been	O
designed	O
for	O
computers	O
.	O
they	O
they	O
natural	B
languages	I
are	O
designed	O
for	O
us	O
to	O
communicate	O
.	O
there	O
are	O
other	O
languages	O
designed	O
for	O
computers	O
.	O
for	O
example	O
program	O
languages	O
.	O
those	O
are	O
harder	O
for	O
us	O
right	O
.	O
so	O
natural	B
languages	I
is	O
designed	O
to	O
make	O
our	O
communication	O
efficient	O
.	O
as	O
a	O
result	O
we	O
omit	O
a	O
lot	O
of	O
common	O
sense	O
knowledge	O
because	O
we	O
assume	O
everyone	O
knows	O
about	O
that	O
.	O
we	O
also	O
keep	O
a	O
lot	O
of	O
ambiguities	O
because	O
we	O
assume	O
the	O
receiver	O
or	O
the	O
hearer	O
could	O
know	O
how	O
to	O
discern	O
an	O
ambiguous	O
word	O
based	O
on	O
the	O
knowledge	O
or	O
the	O
context	O
.	O
there	O
s	O
no	O
need	O
to	O
invent	O
a	O
different	O
word	O
for	O
different	O
meanings	O
.	O
we	O
could	O
overload	O
the	O
same	O
word	O
with	O
different	O
meanings	O
without	O
the	O
problem	O
.	O
because	O
of	O
these	O
reasons	O
this	O
makes	O
every	O
step	O
in	O
natural	B
language	I
of	O
processing	O
difficult	O
for	O
computers	O
.	O
ambiguity	O
s	O
the	O
main	O
difficulty	O
and	O
common	O
sense	O
reasoning	O
is	O
often	O
required	O
that	O
s	O
also	O
hard	O
.	O
so	O
let	O
me	O
give	O
you	O
some	O
examples	O
of	O
challenges	O
here	O
.	O
conceded	O
the	O
word-level	O
ambiguities	O
.	O
the	O
same	O
word	O
can	O
have	O
different	O
syntactical	O
categories	O
.	O
for	O
example	O
design	O
can	O
be	O
a	O
noun	O
or	O
a	O
verb	O
.	O
the	O
word	O
root	O
may	O
have	O
multiple	O
meanings	O
.	O
so	O
square	O
root	O
in	O
math	O
sense	O
or	O
the	O
root	O
of	O
a	O
plant	O
.	O
you	O
might	O
be	O
able	O
to	O
think	O
of	O
other	O
meanings	O
.	O
there	O
are	O
also	O
syntactical	O
ambiguities	O
.	O
for	O
example	O
the	O
main	O
topic	O
of	O
this	O
lecture	O
natural	B
language	I
processing	I
can	O
actually	O
be	O
interpreted	O
in	O
two	O
ways	O
in	O
terms	O
of	O
the	O
structure	O
.	O
think	O
for	O
a	O
moment	O
and	O
see	O
if	O
you	O
can	O
figure	O
that	O
out	O
.	O
we	O
usually	O
think	O
of	O
this	O
as	O
processing	O
of	O
natural	B
languages	I
but	O
you	O
could	O
also	O
think	O
of	O
this	O
as	O
you	O
say	O
language	O
process	O
is	O
natural	O
.	O
right	O
.	O
so	O
this	O
is	O
example	O
of	O
syntatic	O
ambiguity	O
.	O
where	O
we	O
have	O
different	O
structures	O
that	O
can	O
be	O
applied	O
to	O
the	O
same	O
sequence	O
of	O
words	O
.	O
another	O
example	O
of	O
ambiguous	O
sentence	O
is	O
the	O
following	O
a	O
man	O
saw	O
a	O
boy	O
with	O
a	O
telescope	O
.	O
now	O
in	O
this	O
case	O
the	O
question	O
is	O
who	O
had	O
the	O
telescope	O
all	O
right	O
this	O
is	O
called	O
a	O
prepositional	O
phrase	O
attachment	O
ambiguity	O
or	O
pp	O
attachment	O
ambiguity	O
.	O
now	O
we	O
generally	O
don	O
t	O
have	O
a	O
problem	O
with	O
these	O
ambiguities	O
because	O
we	O
have	O
a	O
lot	O
of	O
background	O
knowledge	O
to	O
help	O
us	O
disintegrate	O
the	O
ambiguity	O
.	O
another	O
example	O
of	O
difficulty	O
is	O
anaphora	O
resolution	O
.	O
so	O
think	O
about	O
the	O
sentence	O
like	O
john	O
persuaded	O
bill	O
to	O
buy	O
a	O
tv	O
for	O
himself	O
.	O
the	O
question	O
here	O
is	O
does	O
himself	O
refer	O
to	O
john	O
or	O
bill	O
so	O
again	O
this	O
is	O
something	O
that	O
you	O
have	O
to	O
use	O
some	O
background	O
or	O
the	O
context	O
to	O
figure	O
out	O
.	O
finally	O
presupposition	O
is	O
another	O
problem	O
.	O
consider	O
the	O
sentence	O
he	O
has	O
quit	O
smoking	O
.	O
now	O
this	O
obviously	O
implies	O
he	O
smoked	O
before	O
.	O
so	O
imagine	O
a	O
computer	O
wants	O
to	O
understand	O
all	O
the	O
subtle	O
differences	O
and	O
meanings	O
.	O
they	O
would	O
have	O
to	O
use	O
a	O
lot	O
of	O
knowledge	O
to	O
figure	O
that	O
out	O
.	O
it	O
also	O
would	O
have	O
to	O
maintain	O
a	O
large	O
knowl	O
knowledge	O
base	O
of	O
odd	O
meanings	O
of	O
words	O
and	O
how	O
they	O
are	O
connected	O
to	O
our	O
common	O
sense	O
knowledge	O
of	O
the	O
word	O
.	O
so	O
this	O
is	O
why	O
it	O
s	O
very	O
difficult	O
.	O
so	O
as	O
a	O
result	O
we	O
are	O
still	O
not	O
perfect	O
.	O
in	O
fact	O
far	O
from	O
perfect	O
in	O
understanding	O
natural	B
languages	I
using	O
computers	O
.	O
so	O
this	O
slide	O
sort	O
of	O
gives	O
a	O
simplified	O
view	O
of	O
state	O
of	O
the	O
art	O
technologies	O
.	O
we	O
can	O
do	O
part	B
of	I
speech	I
tagging	I
pretty	O
well	O
.	O
so	O
i	O
showed	O
minus	O
accuracy	O
here	O
.	O
now	O
this	O
number	O
is	O
obviously	O
based	O
on	O
a	O
certain	O
data	O
set	O
so	O
don	O
t	O
take	O
this	O
literally	O
.	O
all	O
right	O
this	O
just	O
shows	O
that	O
we	O
could	O
do	O
it	O
pretty	O
well	O
.	O
but	O
it	O
s	O
still	O
not	O
perfect	O
.	O
in	O
terms	O
of	O
parsing	B
we	O
can	O
do	O
partial	B
parsing	B
pretty	O
well	O
.	O
that	O
means	O
we	O
can	O
get	O
noun	O
phrase	O
structures	O
or	O
verb	O
phrase	O
structure	O
or	O
some	O
segment	O
of	O
the	O
sentence	O
understood	O
correctly	O
in	O
terms	O
of	O
the	O
structure	O
.	O
and	O
in	O
some	O
evaluation	O
results	O
we	O
have	O
seen	O
about	O
accuracy	O
in	O
terms	O
of	O
partial	B
parsing	B
of	O
sentences	O
.	O
again	O
i	O
have	O
to	O
say	O
these	O
numbers	O
are	O
relative	O
to	O
the	O
data	O
set	O
.	O
in	O
some	O
other	O
data	O
sets	O
the	O
numbers	O
might	O
be	O
lower	O
.	O
most	O
of	O
existing	O
work	O
has	O
been	O
evaluated	O
using	O
news	O
data	O
set	O
.	O
and	O
so	O
a	O
lot	O
of	O
these	O
numbers	O
are	O
more	O
or	O
less	O
biased	O
towards	O
news	O
data	O
.	O
think	O
about	O
social	O
media	O
data	O
.	O
the	O
accuracy	O
likely	O
is	O
lower	O
.	O
in	O
terms	O
of	O
semantic	B
analysis	I
we	O
are	O
far	O
from	O
being	O
able	O
to	O
do	O
a	O
complete	O
understanding	O
of	O
a	O
sentence	O
.	O
but	O
we	O
have	O
some	O
techniques	O
that	O
would	O
allow	O
us	O
to	O
do	O
partial	O
understanding	O
of	O
the	O
sentence	O
.	O
so	O
i	O
could	O
mention	O
some	O
of	O
them	O
.	O
for	O
example	O
we	O
have	O
techniques	O
that	O
can	O
allow	O
us	O
to	O
extract	O
the	O
entities	O
and	O
relations	O
mentioned	O
in	O
text	O
or	O
articles	O
.	O
for	O
example	O
recognizing	O
the	O
mentions	O
of	O
people	O
locations	O
organizations	O
et	O
cetera	O
in	O
text	O
.	O
right	O
so	O
this	O
is	O
called	O
entity	O
extraction	O
.	O
we	O
may	O
be	O
able	O
to	O
recognize	O
the	O
relations	O
.	O
for	O
example	O
this	O
person	O
visited	O
that	O
per	O
that	O
place	O
.	O
or	O
this	O
person	O
met	O
that	O
person	O
or	O
this	O
company	O
acquired	O
another	O
company	O
.	O
such	O
relations	O
can	O
be	O
extracted	O
by	O
using	O
the	O
current	O
and	O
natural	O
languaging	O
processing	O
techniques	O
.	O
they	O
are	O
not	O
perfect	O
but	O
they	O
can	O
do	O
well	O
for	O
some	O
entities	O
.	O
some	O
entities	O
are	O
harder	O
than	O
others	O
.	O
we	O
can	O
also	O
do	O
word	O
sentence	O
disintegration	O
to	O
some	O
extent	O
.	O
we	O
have	O
to	O
figure	O
out	O
whether	O
this	O
word	O
in	O
this	O
sentence	O
would	O
have	O
certain	O
meaning	O
and	O
in	O
another	O
context	O
the	O
computer	O
could	O
figure	O
out	O
that	O
it	O
has	O
a	O
different	O
meaning	O
.	O
again	O
it	O
s	O
not	O
perfect	O
but	O
you	O
can	O
do	O
something	O
in	O
that	O
direction	O
.	O
we	O
can	O
also	O
do	O
sentiment	O
analysis	O
meaning	O
to	O
figure	O
out	O
whether	O
sentence	O
is	O
positive	O
or	O
negative	O
.	O
this	O
is	O
a	O
special	O
use	O
for	O
for	O
review	O
analysis	O
for	O
example	O
.	O
so	O
these	O
examples	O
of	O
semantic	B
analysis	I
.	O
and	O
they	O
help	O
us	O
to	O
obtain	O
partial	O
understanding	O
of	O
the	O
sentences	O
.	O
right	O
it	O
s	O
not	O
giving	O
us	O
a	O
complete	O
understanding	O
as	O
i	O
showed	O
before	O
for	O
the	O
sentence	O
but	O
it	O
will	O
still	O
help	O
us	O
gain	B
understanding	O
of	O
the	O
content	O
and	O
these	O
can	O
be	O
useful	O
.	O
in	O
terms	O
of	O
inference	B
we	O
are	O
not	O
yet	O
there	O
probably	O
because	O
of	O
the	O
general	O
difficulty	O
of	O
inference	B
and	O
uncertainties	O
.	O
this	O
is	O
a	O
general	O
challenge	O
in	O
artificial	O
intelligence	O
.	O
that	O
s	O
probably	O
also	O
because	O
we	O
don	O
t	O
have	O
complete	O
semantic	O
reimplementation	O
for	O
natural	B
language	I
text	O
.	O
so	O
this	O
is	O
hard	O
.	O
yet	O
in	O
some	O
domains	O
perhaps	O
in	O
limited	O
domains	O
when	O
you	O
have	O
a	O
lot	O
of	O
restrictions	O
on	O
the	O
world	O
of	O
users	O
you	O
may	O
be	O
to	O
may	O
be	O
able	O
to	O
perform	O
inference	B
to	O
some	O
extent	O
but	O
in	O
general	O
we	O
cannot	O
really	O
do	O
that	O
reliably	O
.	O
speech	O
act	O
analysis	O
is	O
also	O
far	O
from	O
being	O
done	O
and	O
we	O
can	O
only	O
do	O
that	O
analysis	O
for	O
very	O
special	O
cases	O
.	O
so	O
this	O
roughly	O
gives	O
you	O
some	O
idea	O
about	O
the	O
state	O
of	O
the	O
art	O
.	O
and	O
let	O
me	O
also	O
talk	O
a	O
little	O
bit	O
about	O
what	O
we	O
can	O
t	O
do	O
.	O
and	O
and	O
so	O
we	O
can	O
t	O
even	O
do	O
part	B
of	I
speech	I
tagging	I
.	O
this	O
looks	O
like	O
a	O
simple	O
task	O
but	O
think	O
about	O
the	O
example	O
here	O
the	O
two	O
uses	O
of	O
off	O
may	O
have	O
different	O
syntactic	B
categories	I
if	O
you	O
try	O
to	O
make	O
a	O
fine	O
grain	O
distinctions	O
.	O
it	O
s	O
not	O
that	O
easy	O
to	O
figure	O
out	O
such	O
differences	O
.	O
it	O
s	O
also	O
hard	O
to	O
do	O
general	O
complete	O
the	O
parsing	B
.	O
and	O
again	O
this	O
same	O
sentence	O
that	O
you	O
saw	O
before	O
is	O
example	O
.	O
this	O
this	O
ambiguity	O
can	O
be	O
very	O
hard	O
to	O
disambiguate	O
.	O
and	O
you	O
can	O
imagine	O
example	O
where	O
you	O
have	O
to	O
use	O
a	O
lot	O
of	O
knowledge	O
i	O
in	O
the	O
context	O
of	O
the	O
sentence	O
or	O
from	O
the	O
background	O
in	O
order	O
to	O
figure	O
out	O
the	O
who	O
actually	O
had	O
the	O
telescope	O
.	O
so	O
is	O
i	O
although	O
sentence	O
looks	O
very	O
simple	O
it	O
actually	O
is	O
pretty	O
hard	O
.	O
and	O
in	O
cases	O
when	O
the	O
sentence	O
is	O
very	O
long	O
imagine	O
it	O
has	O
four	O
or	O
five	O
prepositional	O
phrases	O
then	O
there	O
are	O
even	O
more	O
possibilities	O
to	O
figure	O
out	O
.	O
it	O
s	O
also	O
harder	O
to	O
precise	O
deep	O
semantic	B
analysis	I
.	O
so	O
here	O
s	O
example	O
.	O
in	O
this	O
sentence	O
john	O
owns	O
a	O
restaurant	O
how	O
do	O
we	O
define	O
owns	O
exactly	O
the	O
word	O
own	O
you	O
know	O
is	O
something	O
that	O
we	O
can	O
understand	O
but	O
it	O
s	O
very	O
hard	O
to	O
precisely	O
describe	O
the	O
meaning	O
of	O
own	O
for	O
computers	O
.	O
so	O
as	O
a	O
result	O
we	O
have	O
robust	O
and	O
general	O
natural	B
language	I
processing	I
techniques	I
that	O
can	O
process	O
a	O
lot	O
of	O
text	O
data	O
in	O
a	O
shallow	O
way	O
meaning	O
we	O
only	O
do	O
superficial	O
analysis	O
.	O
for	O
example	O
part	O
of	O
s	O
of	O
speech	O
tagging	O
or	O
partial	B
parsing	B
or	O
recognizing	O
sentiment	O
.	O
and	O
those	O
are	O
not	O
deep	O
understanding	O
because	O
we	O
re	O
not	O
really	O
understanding	O
the	O
exact	O
meaning	O
of	O
the	O
sentence	O
.	O
on	O
the	O
other	O
hand	O
the	O
deep	O
understanding	O
techniques	O
tend	O
not	O
to	O
scale	O
up	O
well	O
meaning	O
that	O
they	O
would	O
fail	O
on	O
some	O
unrestricted	O
text	O
.	O
and	O
if	O
you	O
don	O
t	O
restrict	O
the	O
text	O
domain	O
or	O
the	O
use	O
of	O
words	O
then	O
these	O
techniques	O
tend	O
not	O
to	O
work	O
well	O
.	O
they	O
may	O
work	O
well	O
based	O
on	O
machine	B
learning	I
techniques	O
on	O
the	O
data	O
that	O
are	O
similar	O
to	O
the	O
training	O
data	O
that	O
the	O
program	O
has	O
been	O
trained	O
on	O
.	O
but	O
they	O
generally	O
wouldn	O
t	O
work	O
well	O
on	O
the	O
data	O
that	O
are	O
very	O
different	O
from	O
the	O
training	O
data	O
.	O
so	O
this	O
pretty	O
much	O
summarizes	O
the	O
state	O
of	O
the	O
art	O
of	O
natural	B
language	I
processing	I
.	O
of	O
course	O
within	O
such	O
a	O
short	O
amount	O
of	O
time	O
we	O
can	O
t	O
really	O
give	O
you	O
a	O
a	O
complete	O
view	O
of	O
any	O
of	O
it	O
which	O
is	O
a	O
big	O
field	O
and	O
either	O
expect	O
that	O
to	O
have	O
to	O
see	O
multiple	O
courses	O
on	O
natural	B
language	I
processing	I
topic	O
itself	O
.	O
but	O
because	O
of	O
it	O
s	O
relevance	O
to	O
the	O
topic	O
that	O
we	O
talked	O
about	O
it	O
s	O
useful	O
for	O
you	O
to	O
know	O
the	O
background	O
in	O
case	O
you	O
haven	O
t	O
been	O
exposed	O
to	O
that	O
.	O
so	O
what	O
does	O
that	O
mean	O
for	O
text	B
retrieval	I
well	O
in	O
text	B
retrieval	I
we	O
are	O
dealing	O
with	O
all	O
kinds	O
of	O
text	O
.	O
it	O
s	O
very	O
hard	O
to	O
restrict	O
the	O
text	O
to	O
a	O
certain	O
domain	O
.	O
and	O
we	O
also	O
are	O
often	O
dealing	O
with	O
a	O
lot	O
of	O
text	O
data	O
so	O
that	O
means	O
.	O
the	O
nlp	B
techniques	O
must	O
be	O
general	O
robust	O
and	O
efficient	O
and	O
that	O
just	O
implies	O
today	O
we	O
can	O
only	O
use	O
fairly	O
shallow	B
nlp	B
techniques	O
for	O
text	B
retrieval	I
.	O
in	O
fact	O
most	O
search	B
engines	I
today	O
use	O
something	O
called	O
a	O
bag	B
of	I
words	I
representation	I
.	O
now	O
this	O
is	O
probably	O
the	O
simplest	O
representation	O
you	O
can	O
probably	O
think	O
of	O
.	O
that	O
is	O
to	O
turn	O
text	O
data	O
into	O
simply	O
a	O
bag	B
of	I
words	I
.	O
meaning	O
we	O
will	O
keep	O
the	O
individual	O
words	O
but	O
we	O
ll	O
ignore	O
all	O
the	O
orders	O
of	O
words	O
.	O
and	O
we	O
ll	O
keep	O
duplicated	O
occurrences	O
of	O
words	O
.	O
so	O
this	O
is	O
called	O
a	O
bag	B
of	I
words	I
representation	I
.	O
when	O
you	O
represent	O
the	O
text	O
in	O
this	O
way	O
you	O
ignore	O
a	O
lot	O
about	O
the	O
information	O
and	O
that	O
just	O
makes	O
it	O
harder	O
to	O
understand	O
the	O
exact	O
meaning	O
of	O
a	O
sentence	O
because	O
we	O
ve	O
lost	O
the	O
order	O
.	O
but	O
yet	O
this	O
representation	O
tends	O
to	O
actually	O
work	O
pretty	O
well	O
for	O
most	O
search	B
tasks	I
.	O
and	O
this	O
is	O
partly	O
because	O
the	O
search	B
task	I
is	O
not	O
all	O
that	O
difficult	O
.	O
if	O
you	O
see	O
matching	O
of	O
some	O
of	O
the	O
query	B
words	I
in	O
a	O
text	O
document	O
chances	O
are	O
that	O
that	O
document	O
is	O
about	O
the	O
topic	O
although	O
there	O
are	O
exceptions	O
right	O
so	O
in	O
comparison	O
some	O
other	O
tasks	O
for	O
example	O
machine	O
translation	O
would	O
require	O
you	O
to	O
understand	O
the	O
language	O
accurately	O
otherwise	O
the	O
translation	O
would	O
be	O
wrong	O
.	O
so	O
in	O
comparison	O
search	B
tasks	I
are	O
solved	O
relatively	O
easy	O
such	O
a	O
representation	O
is	O
often	O
sufficient	O
.	O
and	O
that	O
s	O
also	O
the	O
representation	O
that	O
the	O
major	B
search	B
engines	I
today	O
like	O
google	O
or	O
bing	O
are	O
using	O
.	O
of	O
course	O
i	O
put	O
in	O
in	O
parentheses	O
but	O
not	O
all	O
.	O
of	O
course	O
there	O
are	O
many	O
queries	O
that	O
are	O
not	O
answered	O
well	O
by	O
the	O
current	O
search	B
engines	I
and	O
they	O
do	O
require	O
a	O
representation	O
that	O
would	O
go	O
beyond	O
bag	B
of	I
words	I
representation	I
.	O
that	O
would	O
require	O
more	O
natural	B
language	I
processing	I
to	O
be	O
done	O
.	O
there	O
is	O
another	O
reason	O
why	O
we	O
have	O
not	O
used	O
the	O
sophisticated	O
nlp	B
techniques	O
in	O
modern	O
search	B
engines	I
and	O
that	O
s	O
because	O
some	O
retrieval	B
techniques	I
actually	O
naturally	O
solve	O
the	O
problem	O
of	O
nlp	B
.	O
so	O
one	O
example	O
is	O
word	O
sense	O
disambiguation	O
.	O
think	O
about	O
a	O
word	O
like	O
java	O
.	O
it	O
could	O
mean	O
coffee	O
or	O
it	O
could	O
mean	O
program	O
language	O
.	O
if	O
you	O
look	O
at	O
the	O
word	O
alone	O
it	O
would	O
be	O
ambiguous	O
.	O
but	O
when	O
the	O
user	O
uses	O
the	O
water	O
in	O
the	O
query	O
usually	O
there	O
are	O
other	O
words	O
.	O
for	O
example	O
i	O
m	O
looking	O
for	O
usage	O
of	O
java	O
applet	O
.	O
when	O
i	O
have	O
applet	O
there	O
that	O
implies	O
java	O
means	O
program	O
language	O
.	O
and	O
that	O
context	O
can	O
help	O
us	O
naturally	O
prefer	O
documents	O
where	O
java	O
is	O
referring	O
to	O
program	O
language	O
because	O
those	O
documents	O
would	O
probably	O
match	O
applet	O
as	O
well	O
.	O
if	O
java	O
occurs	O
in	O
the	O
document	O
in	O
a	O
way	O
that	O
means	O
coffee	O
then	O
you	O
would	O
never	O
match	O
applet	O
or	O
with	O
very	O
small	O
probability	O
.	O
right	O
.	O
so	O
this	O
is	O
a	O
case	O
when	O
some	O
retrieval	B
techniques	I
naturally	O
achieve	O
the	O
goal	O
of	O
word	O
sense	O
disambiguation	O
.	O
another	O
example	O
is	O
some	O
technique	O
called	O
feedback	O
which	O
we	O
will	O
talk	O
about	O
later	O
in	O
some	O
of	O
the	O
lectures	O
.	O
this	O
tech	O
technique	O
would	O
allow	O
us	O
to	O
add	O
additional	O
words	O
to	O
the	O
query	O
.	O
and	O
those	O
additional	O
words	O
could	O
be	O
related	O
to	O
the	O
query	B
words	I
.	O
and	O
these	O
words	O
can	O
help	O
match	O
documents	O
where	O
the	O
original	O
query	B
words	I
have	O
not	O
occurred	O
.	O
so	O
this	O
achieves	O
to	O
some	O
extent	O
semantic	B
matching	I
of	O
terms	O
.	O
so	O
those	O
techniques	O
also	O
helped	O
us	O
bypass	O
some	O
of	O
the	O
difficulties	O
in	O
natural	B
language	I
processing	I
.	O
however	O
in	O
the	O
long	O
run	O
we	O
still	O
need	O
deeper	O
natural	B
language	I
processing	I
techniques	I
in	O
order	O
to	O
improve	O
the	O
accuracy	O
of	O
the	O
current	O
search	B
engines	I
.	O
and	O
it	O
s	O
particularly	O
needed	O
for	O
complex	O
search	B
tasks	I
or	O
for	O
question	O
answering	O
.	O
google	O
has	O
recently	O
launched	O
a	O
knowledge	O
graph	O
.	O
and	O
this	O
is	O
one	O
step	O
toward	O
that	O
goal	O
because	O
knowledge	O
graph	O
would	O
contain	O
entities	O
and	O
their	O
relations	O
.	O
and	O
this	O
goes	O
beyond	O
the	O
simple	O
bag	B
of	I
words	I
representation	I
.	O
and	O
such	O
technique	O
should	O
help	O
us	O
improve	O
the	O
search	B
engine	I
utility	O
significantly	O
although	O
this	O
is	O
a	O
still	O
open	O
topic	O
for	O
research	O
and	O
exploration	O
.	O
in	O
sum	O
in	O
this	O
lecture	O
we	O
ll	O
talk	O
about	O
what	O
is	O
nlp	B
and	O
we	O
ve	O
talked	O
about	O
the	O
state	O
of	O
the	O
art	O
techniques	O
what	O
we	O
can	O
do	O
what	O
we	O
cannot	O
do	O
.	O
and	O
finally	O
we	O
also	O
explained	O
why	O
bag	B
of	I
words	I
representation	I
remains	O
the	O
dominant	O
representation	O
used	O
in	O
modern	O
search	B
engines	I
even	O
though	O
deeper	O
nlp	B
would	O
be	O
needed	O
for	O
future	O
search	B
engines	I
.	O
if	O
you	O
want	O
to	O
know	O
more	O
you	O
can	O
take	O
a	O
look	O
at	O
some	O
additional	O
readings	O
.	O
i	O
only	O
cited	O
one	O
here	O
.	O
and	O
that	O
s	O
a	O
good	O
starting	O
point	O
though	O
.	O
thanks	O
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
text	O
access	O
.	O
in	O
the	O
previously	O
lecture	O
we	O
talked	O
about	O
natural	B
language	I
content	O
analysis	O
.	O
we	O
explained	O
that	O
the	O
state	O
of	O
the	O
art	O
natural	B
language	I
processing	I
techniques	I
are	O
still	O
not	O
good	O
enough	O
to	O
process	O
a	O
lot	O
of	O
unrestricted	O
text	O
data	O
in	O
a	O
robust	O
manner	O
.	O
as	O
a	O
result	O
bag	B
of	I
words	I
representation	I
remains	O
very	O
popular	O
in	O
applications	O
like	O
search	B
engines	I
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
some	O
high	O
level	O
strategies	O
to	O
help	O
users	O
get	O
access	O
to	O
the	O
text	O
data	O
.	O
this	O
is	O
also	O
important	O
step	O
to	O
convert	O
raw	O
big	O
text	O
data	O
into	O
small	O
relevant	O
data	O
that	O
are	O
actually	O
needed	O
in	O
a	O
specific	O
application	O
.	O
so	O
the	O
main	O
question	O
we	O
address	O
here	O
is	O
how	O
can	O
a	O
text	O
information	O
system	O
help	O
users	O
get	O
access	O
to	O
the	O
relevant	O
text	O
data	O
we	O
re	O
going	O
to	O
cover	O
two	O
complementary	O
strategies	O
push	B
vs	O
pull	O
.	O
and	O
then	O
we	O
re	O
going	O
to	O
talk	O
about	O
two	O
ways	O
to	O
implement	O
the	O
pull	O
mode	O
querying	B
vs	O
browsing	O
.	O
so	O
first	O
push	B
vs	O
pull	O
.	O
these	O
are	O
two	O
different	O
ways	O
to	O
connect	O
users	O
with	O
the	O
right	O
information	O
at	O
the	O
right	O
time	O
.	O
the	O
difference	O
is	O
which	O
takes	O
the	O
initiative	O
which	O
party	O
it	O
takes	O
in	O
the	O
initiative	O
.	O
in	O
the	O
pull	O
mode	O
the	O
users	O
would	O
take	O
the	O
initiative	O
to	O
start	O
the	O
information	B
access	I
process	O
.	O
and	O
in	O
this	O
case	O
a	O
user	O
typically	O
would	O
use	O
a	O
search	B
engine	I
to	O
fulfill	O
the	O
goal	O
.	O
for	O
example	O
the	O
user	O
may	O
type	O
in	O
a	O
query	O
and	O
then	O
browse	O
results	O
to	O
find	O
the	O
relevant	B
information	I
.	O
so	O
this	O
is	O
usually	O
appropriate	O
for	O
satisfying	O
a	O
user	O
s	O
ad	O
hoc	O
information	O
need	O
.	O
an	O
ad	O
hoc	O
information	O
need	O
is	O
a	O
temporary	O
information	O
need	O
.	O
for	O
example	O
you	O
want	O
to	O
buy	O
a	O
product	O
so	O
you	O
suddenly	O
have	O
a	O
need	O
to	O
read	O
reviews	O
about	O
related	O
products	O
.	O
but	O
after	O
you	O
have	O
collected	O
information	O
you	O
have	O
purchased	O
your	O
product	O
you	O
generally	O
no	O
longer	O
need	O
such	O
information	O
.	O
so	O
it	O
s	O
a	O
temporary	O
information	O
need	O
.	O
in	O
such	O
a	O
case	O
it	O
s	O
very	O
hard	O
for	O
a	O
system	O
to	O
predict	O
your	O
need	O
and	O
it	O
s	O
more	O
appropriate	O
for	O
the	O
users	O
to	O
take	O
the	O
initiative	O
.	O
and	O
that	O
s	O
why	O
search	B
engines	I
are	O
very	O
useful	O
today	O
because	O
many	O
people	O
have	O
many	O
ad	O
hoc	O
information	O
needs	O
all	O
the	O
time	O
.	O
so	O
as	O
we	O
are	O
speaking	O
google	O
probably	O
is	O
processing	O
many	O
queries	O
from	O
this	O
and	O
those	O
are	O
all	O
or	O
mostly	O
ad	O
hoc	O
information	O
needs	O
.	O
so	O
this	O
is	O
a	O
pull	O
mode	O
.	O
in	O
contrast	O
in	O
the	O
push	B
mode	O
the	O
system	O
will	O
take	O
the	O
initiative	O
to	O
push	B
the	O
information	O
to	O
the	O
user	O
or	O
to	O
recommend	O
the	O
information	O
to	O
the	O
user	O
.	O
so	O
in	O
this	O
case	O
this	O
is	O
usually	O
supported	O
by	O
a	O
recommender	B
system	I
.	O
now	O
this	O
would	O
be	O
appropriate	O
if	O
the	O
user	O
has	O
a	O
stable	O
information	O
need	O
.	O
for	O
example	O
you	O
may	O
have	O
a	O
research	O
interest	O
in	O
some	O
topic	O
and	O
that	O
interest	O
tends	O
to	O
stay	O
for	O
a	O
while	O
so	O
it	O
s	O
relatively	O
stable	O
.	O
your	O
hobby	O
is	O
another	O
example	O
of	O
a	O
stable	O
information	O
need	O
.	O
in	O
such	O
a	O
case	O
the	O
system	O
can	O
interact	O
with	O
you	O
and	O
can	O
learn	O
your	O
interest	O
and	O
then	O
can	O
monitor	O
the	O
information	O
stream	O
.	O
if	O
it	O
is	O
the	O
system	O
hasn	O
t	O
seen	O
any	O
relevant	B
items	I
to	O
your	O
interest	O
the	O
system	O
could	O
then	O
take	O
the	O
initiative	O
to	O
recommend	O
information	O
to	O
you	O
.	O
so	O
for	O
example	O
a	O
news	O
filter	O
or	O
news	O
recommender	B
system	I
could	O
monitor	O
the	O
news	O
stream	O
and	O
identify	O
interest	O
in	O
news	O
to	O
you	O
and	O
simply	O
push	B
the	O
news	O
articles	O
to	O
you	O
.	O
this	O
mode	O
of	O
information	B
access	I
may	O
be	O
also	O
appropriate	O
when	O
the	O
system	O
has	O
a	O
good	O
knowledge	O
about	O
the	O
user	O
s	O
need	O
.	O
and	O
this	O
happens	O
in	O
the	O
search	O
context	O
.	O
so	O
for	O
example	O
when	O
you	O
search	O
for	O
information	O
on	O
the	O
web	O
a	O
search	B
engine	I
might	O
infer	O
you	O
might	O
be	O
also	O
interested	O
in	O
some	O
related	O
information	O
.	O
and	O
they	O
would	O
recommend	O
the	O
information	O
to	O
you	O
.	O
so	O
that	O
should	O
remind	O
you	O
for	O
example	O
advertisement	O
placed	O
on	O
a	O
search	O
page	O
.	O
so	O
this	O
is	O
about	O
the	O
the	O
two	O
high	O
level	O
strategies	O
or	O
two	O
modes	O
of	O
text	O
access	O
.	O
now	O
let	O
s	O
look	O
at	O
the	O
pull	O
mode	O
in	O
more	O
detail	O
.	O
in	O
the	O
pull	O
mode	O
we	O
can	O
further	O
this	O
in	O
usually	O
two	O
ways	O
to	O
help	O
users	O
querying	B
vs	O
browsing	O
.	O
in	O
querying	B
a	O
user	O
would	O
just	O
enter	O
a	O
query	O
typically	O
a	O
keyword	O
query	O
and	O
the	O
search	B
engine	I
system	O
would	O
return	O
relevant	B
documents	I
to	O
users	O
.	O
and	O
this	O
works	O
well	O
when	O
the	O
user	O
knows	O
what	O
exactly	O
key	O
are	O
the	O
keywords	O
to	O
be	O
used	O
.	O
so	O
if	O
you	O
know	O
exactly	O
what	O
you	O
re	O
looking	O
for	O
you	O
tend	O
to	O
know	O
the	O
right	O
keywords	O
and	O
then	O
query	O
would	O
work	O
very	O
well	O
.	O
and	O
we	O
do	O
that	O
all	O
the	O
time	O
.	O
but	O
we	O
also	O
know	O
that	O
sometimes	O
it	O
doesn	O
t	O
work	O
so	O
well	O
when	O
you	O
don	O
t	O
know	O
the	O
right	O
keywords	O
to	O
use	O
in	O
the	O
query	O
or	O
you	O
want	O
to	O
browse	O
information	O
in	O
some	O
topic	O
area	O
.	O
in	O
this	O
case	O
browsing	O
would	O
be	O
more	O
useful	O
.	O
so	O
in	O
this	O
case	O
in	O
the	O
case	O
of	O
browsing	O
the	O
users	O
would	O
simply	O
navigate	O
into	O
the	O
relevant	B
information	I
by	O
following	O
the	O
path	O
that	O
s	O
supported	O
by	O
the	O
structures	O
on	O
the	O
documents	O
.	O
so	O
the	O
system	O
would	O
maintain	O
some	O
kind	O
of	O
structures	O
and	O
then	O
the	O
user	O
could	O
follow	O
these	O
structures	O
to	O
navigate	O
.	O
so	O
this	O
strategy	O
works	O
well	O
when	O
the	O
user	O
wants	O
to	O
explore	O
information	O
space	O
or	O
the	O
user	O
doesn	O
t	O
know	O
what	O
are	O
the	O
keywords	O
to	O
use	O
in	O
the	O
query	O
.	O
or	O
simply	O
because	O
the	O
user	O
finds	O
it	O
inconvenient	O
to	O
type	O
in	O
the	O
query	O
.	O
so	O
even	O
if	O
a	O
user	O
knows	O
what	O
query	O
to	O
type	O
in	O
if	O
the	O
user	O
is	O
using	O
a	O
cell	O
phone	O
to	O
search	O
for	O
information	O
then	O
it	O
s	O
still	O
hard	O
to	O
enter	O
the	O
query	O
.	O
in	O
such	O
a	O
case	O
again	O
browsing	O
tends	O
to	O
be	O
more	O
convenient	O
.	O
the	O
relationship	O
between	O
browsing	O
and	O
the	O
query	O
is	O
best	O
understood	O
by	O
making	O
an	O
analogy	O
to	O
sightseeing	O
.	O
imagine	O
if	O
you	O
are	O
touring	O
a	O
city	O
.	O
now	O
if	O
you	O
know	O
the	O
exact	O
address	O
of	O
a	O
attraction	O
then	O
taking	O
a	O
taxi	O
there	O
is	O
perhaps	O
the	O
fastest	O
way	O
you	O
can	O
go	O
directly	O
to	O
the	O
site	O
.	O
but	O
if	O
you	O
don	O
t	O
know	O
the	O
exact	O
address	O
you	O
may	O
need	O
to	O
walk	O
around	O
or	O
you	O
can	O
take	O
a	O
taxi	O
to	O
a	O
nearby	O
place	O
and	O
then	O
walk	O
around	O
.	O
it	O
turns	O
out	O
that	O
we	O
do	O
exactly	O
the	O
same	O
in	O
the	O
information	O
space	O
.	O
if	O
you	O
know	O
exactly	O
what	O
you	O
are	O
looking	O
for	O
then	O
you	O
can	O
use	O
the	O
right	O
keywords	O
in	O
your	O
query	O
to	O
find	O
the	O
information	O
directly	O
.	O
that	O
s	O
usually	O
the	O
fastest	O
way	O
to	O
do	O
find	O
information	O
.	O
but	O
what	O
if	O
you	O
don	O
t	O
know	O
the	O
exact	O
keywords	O
to	O
use	O
well	O
your	O
query	O
probably	O
won	O
t	O
work	O
so	O
well	O
and	O
you	O
will	O
land	O
on	O
some	O
related	O
pages	O
and	O
then	O
you	O
need	O
to	O
also	O
walk	O
around	O
in	O
the	O
information	O
space	O
.	O
meaning	O
by	O
following	O
the	O
links	O
or	O
by	O
browsing	O
you	O
can	O
then	O
finally	O
get	O
into	O
the	O
relevant	O
page	O
.	O
if	O
you	O
want	O
to	O
learn	O
about	O
a	O
topic	O
again	O
you	O
you	O
will	O
likely	O
do	O
a	O
lot	O
of	O
browsing	O
.	O
so	O
just	O
like	O
you	O
are	O
looking	O
around	O
in	O
some	O
area	O
and	O
you	O
want	O
to	O
see	O
some	O
interesting	O
attractions	O
in	O
a	O
related	O
in	O
the	O
same	O
region	O
.	O
so	O
this	O
analogy	O
also	O
tells	O
us	O
that	O
today	O
we	O
have	O
very	O
good	O
support	O
for	O
querying	B
but	O
we	O
don	O
t	O
really	O
have	O
good	O
support	O
for	O
browsing	O
.	O
and	O
this	O
is	O
because	O
in	O
order	O
to	O
browse	O
effectively	O
we	O
need	O
a	O
a	O
map	B
to	O
guide	O
us	O
just	O
like	O
you	O
need	O
a	O
map	B
of	O
chicago	O
to	O
tour	O
the	O
city	O
of	O
chicago	O
.	O
you	O
need	O
a	O
topical	O
map	B
to	O
tour	O
the	O
information	O
space	O
.	O
so	O
how	O
to	O
construct	O
such	O
a	O
topical	O
map	B
is	O
in	O
fact	O
a	O
very	O
interesting	O
research	O
question	O
that	O
likely	O
will	O
bring	O
us	O
more	O
interesting	O
browsing	O
experience	O
on	O
the	O
web	O
or	O
in	O
other	O
applications	O
.	O
so	O
to	O
summarize	O
this	O
lecture	O
we	O
have	O
talked	O
about	O
two	O
high	O
level	O
strategies	O
for	O
text	O
access	O
push	B
and	I
pull	I
.	O
push	B
tends	O
to	O
be	O
supported	O
by	O
a	O
recommender	B
system	I
and	O
pull	O
tends	O
to	O
be	O
supported	O
by	O
a	O
search	B
engine	I
.	O
of	O
course	O
in	O
the	O
sophisticated	O
intent	O
in	O
the	O
information	O
system	O
we	O
should	O
combine	O
the	O
two	O
.	O
in	O
the	O
pull	O
mode	O
we	O
have	O
further	O
distinguished	O
querying	B
and	O
browsing	O
.	O
again	O
we	O
generally	O
want	O
to	O
combine	O
the	O
two	O
ways	O
to	O
help	O
users	O
so	O
that	O
you	O
can	O
support	O
both	O
querying	B
and	O
browsing	O
.	O
if	O
you	O
want	O
to	O
know	O
more	O
about	O
the	O
relationship	O
between	O
pull	B
and	I
push	B
you	O
can	O
read	O
this	O
article	O
.	O
this	O
gives	O
a	O
excellent	O
discussion	O
of	O
the	O
relationship	O
between	O
information	B
filtering	I
and	O
information	B
retrieval	I
.	O
here	O
information	B
filtering	I
is	O
similar	O
to	O
information	O
recommendation	O
or	O
the	O
push	B
mode	O
of	O
information	B
access	I
.	O
this	O
lecture	O
is	O
about	O
the	O
text	B
retrieval	I
problem	O
.	O
this	O
picture	O
shows	O
our	O
overall	O
plan	O
for	O
lectures	O
.	O
in	O
the	O
last	O
lecture	O
we	O
talked	O
about	O
the	O
high	O
level	O
strategies	O
for	O
text	O
access	O
.	O
we	O
talked	O
about	O
push	B
versus	O
pull	O
.	O
search	B
engines	I
are	O
the	O
main	O
tools	O
for	O
supporting	O
the	O
pull	O
mode	O
.	O
starting	O
from	O
this	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
how	O
search	B
engines	I
work	O
in	O
detail	O
.	O
so	O
first	O
it	O
s	O
about	O
the	O
text	B
retrieval	I
problem	O
.	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
three	O
things	O
in	O
this	O
lecture	O
.	O
first	O
we	O
ll	O
define	O
text	B
retrieval	I
.	O
second	O
we	O
re	O
going	O
to	O
make	O
a	O
comparison	O
between	O
text	B
retrieval	I
and	O
the	O
related	O
task	O
database	O
retrieval	O
.	O
finally	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
document	O
selection	O
versus	O
document	O
ranking	O
as	O
two	O
strategies	O
for	O
responding	O
to	O
a	O
user	O
s	O
query	O
.	O
so	O
what	O
is	O
text	B
retrieval	I
it	O
should	O
be	O
a	O
task	O
that	O
s	O
familiar	O
to	O
most	O
of	O
us	O
because	O
we	O
re	O
using	O
web	B
search	B
engines	I
all	O
the	O
time	O
.	O
so	O
text	B
retrieval	I
is	O
basically	O
a	O
task	O
where	O
the	O
system	O
would	O
respond	O
to	O
a	O
user	O
s	O
query	O
with	O
relevant	O
lock-ins	O
basically	O
through	O
supported	O
querying	B
as	O
one	O
way	O
to	O
implement	O
the	O
poor	O
mold	O
of	O
information	B
access	I
.	O
so	O
the	O
scenario	O
s	O
the	O
following	O
.	O
you	O
have	O
a	O
collection	O
of	O
text	O
documents	O
.	O
these	O
documents	O
could	O
be	O
all	O
the	O
web	O
pages	O
on	O
the	O
web	O
.	O
or	O
all	O
the	O
literature	O
articles	O
in	O
the	O
digital	O
library	O
or	O
maybe	O
all	O
the	O
text	O
files	O
in	O
your	O
computer	O
.	O
a	O
user	O
will	O
typically	O
give	O
a	O
query	O
to	O
the	O
system	O
to	O
express	O
the	O
information	O
need	O
.	O
and	O
then	O
the	O
system	O
would	O
return	O
relevant	B
documents	I
to	O
users	O
.	O
relevant	B
documents	I
refer	O
to	O
those	O
documents	O
that	O
are	O
useful	O
to	O
the	O
user	O
who	O
typed	O
in	O
the	O
query	O
.	O
now	O
this	O
task	O
is	O
a	O
often	O
called	O
information	B
retrieval	I
.	O
but	O
literally	O
information	B
retrieval	I
would	O
broadly	O
include	O
the	O
retrieval	O
of	O
other	O
non-textual	O
information	O
as	O
well	O
.	O
for	O
example	O
audio	O
video	O
et	O
cetera	O
.	O
it	O
s	O
worth	O
noting	O
that	O
text	B
retrieval	I
is	O
at	O
the	O
core	O
of	O
information	B
retrieval	I
in	O
the	O
sense	O
that	O
other	O
medias	O
such	O
as	O
video	O
can	O
be	O
retrieved	O
by	O
exploiting	O
the	O
companion	O
text	O
data	O
.	O
so	O
for	O
example	O
can	O
the	O
image	O
search	B
engines	I
actually	O
match	O
a	O
user	O
s	O
query	O
with	O
the	O
companion	O
text	O
data	O
of	O
the	O
image	O
this	O
problem	O
is	O
also	O
called	O
the	O
the	O
search	O
problem	O
and	O
the	O
technology	O
is	O
often	O
called	O
search	O
technology	O
in	O
industry	O
.	O
if	O
you	O
ever	O
take	O
on	O
course	O
in	O
databases	O
it	O
ll	O
be	O
useful	O
to	O
pause	O
the	O
lecture	O
at	O
this	O
point	O
and	O
think	O
about	O
the	O
differences	O
between	O
text	B
retrieval	I
and	O
database	O
retrieval	O
.	O
now	O
these	O
two	O
tasks	O
are	O
similar	O
in	O
many	O
ways	O
.	O
but	O
there	O
are	O
some	O
important	O
differences	O
.	O
so	O
spend	O
a	O
moment	O
to	O
think	O
about	O
the	O
differences	O
between	O
the	O
two	O
.	O
think	O
about	O
the	O
data	O
and	O
information	O
managed	O
by	O
a	O
search	B
engine	I
versus	O
those	O
that	O
are	O
man	O
managed	O
by	O
a	O
database	O
system	O
.	O
think	O
about	O
the	O
difference	O
between	O
the	O
queries	O
that	O
you	O
typically	O
specify	O
for	O
a	O
database	O
system	O
versus	O
the	O
queries	O
that	O
typed	O
in	O
by	O
users	O
on	O
the	O
search	B
engine	I
.	O
and	O
then	O
finally	O
think	O
about	O
the	O
answers	O
.	O
what	O
s	O
the	O
difference	O
between	O
the	O
two	O
okay	O
so	O
if	O
we	O
think	O
probably	O
the	O
information	O
out	O
there	O
are	O
managed	O
by	O
the	O
two	O
systems	O
.	O
we	O
will	O
see	O
that	O
in	O
text	B
retrieval	I
the	O
data	O
is	O
unstructured	O
it	O
s	O
free	O
text	O
.	O
but	O
in	O
databases	O
they	O
are	O
structured	O
data	O
where	O
there	O
is	O
a	O
clear	O
defined	O
schema	O
to	O
tell	O
you	O
this	O
column	O
is	O
the	O
names	O
of	O
people	O
and	O
that	O
column	O
is	O
ages	O
et	O
cetera	O
.	O
in	O
unstructured	O
text	O
it	O
s	O
not	O
obvious	O
what	O
are	O
the	O
names	O
of	O
people	O
mentioned	O
in	O
the	O
text	O
.	O
because	O
of	O
this	O
difference	O
we	O
can	O
also	O
see	O
that	O
text	O
information	O
tends	O
to	O
be	O
more	O
ambiguous	O
.	O
and	O
we	O
ll	O
talk	O
about	O
that	O
in	O
the	O
natural	B
language	I
processing	I
lecture	O
.	O
whereas	O
in	O
databases	O
the	O
data	O
tend	O
to	O
have	O
well-defined	O
semantics	O
.	O
there	O
is	O
also	O
important	O
difference	O
in	O
the	O
queries	O
and	O
this	O
is	O
partly	O
due	O
to	O
the	O
difference	O
in	O
the	O
information	O
or	O
data	O
.	O
so	O
text	O
queries	O
tend	O
to	O
be	O
ambiguous	O
whereas	O
in	O
their	O
research	O
the	O
queries	O
are	O
particularly	O
well-defined	O
.	O
think	O
about	O
the	O
sql	O
query	O
that	O
would	O
clear	O
the	O
specify	O
what	O
records	O
to	O
be	O
returned	O
.	O
so	O
it	O
has	O
very	O
well	O
defined	O
semantics	O
.	O
queue	O
all	O
queries	O
or	O
naturally	O
ending	O
queries	O
tend	O
to	O
be	O
incomplete	O
.	O
also	O
in	O
that	O
it	O
doesn	O
t	O
really	O
fully	O
specify	O
what	O
documents	O
should	O
be	O
retrieved	O
.	O
whereas	O
in	O
the	O
database	O
search	O
the	O
sql	O
query	O
can	O
be	O
regarded	O
as	O
a	O
computer	O
specification	O
for	O
what	O
should	O
be	O
returned	O
.	O
and	O
because	O
of	O
these	O
differences	O
the	O
answers	O
would	O
be	O
also	O
different	O
.	O
in	O
the	O
case	O
of	O
text	B
retrieval	I
we	O
re	O
looking	O
for	O
relevant	B
documents	I
.	O
in	O
the	O
database	O
search	O
we	O
are	O
retrieving	O
records	O
or	O
matched	O
records	O
with	O
the	O
sql	O
query	O
more	O
precisely	O
.	O
now	O
in	O
the	O
case	O
of	O
text	B
retrieval	I
what	O
should	O
be	O
the	O
right	O
answers	O
to	O
a	O
query	O
is	O
not	O
very	O
well	O
specified	O
as	O
we	O
just	O
discussed	O
.	O
so	O
it	O
s	O
unclear	O
what	O
should	O
be	O
the	O
right	O
answers	O
to	O
a	O
query	O
.	O
and	O
this	O
has	O
very	O
important	O
consequences	O
and	O
that	O
is	O
text	B
retrieval	I
is	O
an	O
empirically	O
defined	O
problem	O
.	O
and	O
so	O
this	O
a	O
problem	O
because	O
if	O
it	O
s	O
empirically	O
defined	O
then	O
we	O
cannot	O
mathematically	O
prove	O
one	O
method	O
is	O
better	O
than	O
another	O
method	O
.	O
that	O
also	O
means	O
we	O
must	O
rely	O
on	O
emperical	O
evaluation	O
more	O
than	O
users	O
to	O
know	O
which	O
method	O
works	O
better	O
.	O
and	O
that	O
s	O
why	O
we	O
have	O
one	O
lecture	O
actually	O
more	O
than	O
one	O
lectures	O
to	O
cover	O
the	O
issue	O
of	O
evaluation	O
.	O
because	O
this	O
is	O
a	O
very	O
important	O
topic	O
for	O
search	B
engines	I
.	O
without	O
knowing	O
how	O
to	O
evaluate	O
an	O
algorithm	O
appropriately	O
there	O
s	O
no	O
way	O
to	O
tell	O
whether	O
we	O
have	O
got	O
the	O
better	O
algorithm	O
or	O
whether	O
one	O
system	O
is	O
better	O
than	O
another	O
.	O
so	O
now	O
let	O
s	O
look	O
at	O
the	O
problem	O
in	O
a	O
formal	O
way	O
.	O
so	O
this	O
slide	O
shows	O
a	O
formal	O
formulation	O
of	O
the	O
text	B
retrieval	I
problem	O
.	O
first	O
we	O
have	O
our	O
vocabulary	O
set	O
which	O
is	O
just	O
a	O
set	O
of	O
words	O
in	O
a	O
language	O
.	O
now	O
here	O
we	O
re	O
considering	O
just	O
one	O
language	O
but	O
in	O
reality	O
on	O
the	O
web	O
there	O
might	O
be	O
multiple	O
natural	B
languages	I
.	O
we	O
have	O
text	O
that	O
are	O
in	O
all	O
kinds	O
of	O
languages	O
.	O
but	O
here	O
for	O
simplicity	O
we	O
just	O
assume	O
there	O
is	O
one	O
kind	O
of	O
language	O
.	O
as	O
the	O
techniques	O
used	O
for	O
retrieving	O
data	O
from	O
multiple	O
languages	O
	O
.	O
are	O
more	O
or	O
less	O
similar	O
to	O
the	O
techniques	O
used	O
for	O
retrieving	O
documents	O
in	O
one	O
language	O
.	O
although	O
there	O
is	O
important	O
difference	O
the	O
principles	O
and	O
methods	O
are	O
very	O
similar	O
.	O
next	O
we	O
have	O
the	O
query	O
which	O
is	O
a	O
sequence	O
of	O
words	O
.	O
and	O
so	O
here	O
you	O
can	O
see	O
the	O
query	O
is	O
defined	O
as	O
a	O
sequence	O
of	O
words	O
.	O
each	O
q	O
sub	O
i	O
is	O
a	O
word	O
in	O
the	O
vocabulary	O
.	O
a	O
document	O
is	O
defined	O
in	O
the	O
same	O
way	O
.	O
so	O
it	O
s	O
also	O
a	O
sequence	O
of	O
words	O
.	O
and	O
here	O
d	O
sub	O
ij	O
is	O
also	O
a	O
word	O
in	O
the	O
vocabulary	O
.	O
now	O
typically	O
the	O
documents	O
are	O
much	O
longer	O
than	O
queries	O
.	O
but	O
there	O
are	O
also	O
cases	O
where	O
the	O
documents	O
may	O
be	O
very	O
short	O
.	O
so	O
you	O
can	O
think	O
about	O
the	O
what	O
might	O
be	O
a	O
example	O
of	O
that	O
case	O
.	O
i	O
hope	O
you	O
can	O
think	O
of	O
of	O
twitter	O
search	O
all	O
right	O
tweets	O
are	O
very	O
short	O
.	O
but	O
in	O
general	O
documents	O
are	O
longer	O
then	O
the	O
queries	O
.	O
now	O
then	O
we	O
have	O
a	O
collection	O
of	O
documents	O
.	O
and	O
this	O
collection	O
can	O
be	O
very	O
large	O
.	O
so	O
think	O
about	O
the	O
web	O
.	O
it	O
could	O
could	O
be	O
very	O
large	O
.	O
and	O
then	O
the	O
goal	O
of	O
text	B
retrieval	I
is	O
you	O
ll	O
find	O
the	O
set	O
of	O
relevant	B
documents	I
which	O
we	O
denote	O
by	O
r	O
of	O
q	O
because	O
it	O
depends	O
on	O
the	O
query	O
.	O
and	O
this	O
is	O
in	O
general	O
a	O
subset	O
of	O
all	O
the	O
documents	O
in	O
the	O
collection	O
.	O
unfortunately	O
this	O
set	O
of	O
random	O
documents	O
is	O
generally	O
unknown	O
and	O
usually	O
depend	O
in	O
the	O
sense	O
that	O
for	O
the	O
same	O
query	O
typed	O
in	O
by	O
different	O
users	O
the	O
expected	O
relevant	B
documents	I
may	O
be	O
different	O
.	O
the	O
query	O
given	O
to	O
us	O
by	O
the	O
user	O
is	O
only	O
a	O
hint	O
on	O
which	O
document	O
should	O
be	O
in	O
this	O
set	O
.	O
and	O
indeed	O
the	O
user	O
is	O
generally	O
unable	O
to	O
specify	O
what	O
exactly	O
should	O
be	O
in	O
the	O
set	O
especially	O
in	O
the	O
case	O
of	O
a	O
web	B
search	I
where	O
the	O
collection	O
is	O
so	O
large	O
.	O
the	O
user	O
doesn	O
t	O
have	O
complete	O
knowledge	O
about	O
the	O
whole	O
collection	O
.	O
so	O
the	O
best	O
a	O
search	O
system	O
can	O
do	O
is	O
to	O
compute	O
an	O
approximation	O
of	O
this	O
relevent	O
document	O
set	O
.	O
so	O
we	O
denote	O
it	O
by	O
r	O
prime	O
of	O
q	O
.	O
so	O
formally	O
we	O
can	O
see	O
the	O
task	O
is	O
to	O
compute	O
this	O
r	O
prime	O
of	O
q	O
an	O
approximation	O
of	O
the	O
relevant	B
documents	I
.	O
so	O
how	O
can	O
we	O
do	O
that	O
now	O
imagine	O
if	O
you	O
are	O
now	O
asked	O
to	O
write	O
a	O
program	O
to	O
do	O
this	O
.	O
what	O
would	O
you	O
do	O
now	O
think	O
for	O
a	O
moment	O
.	O
right	O
so	O
these	O
are	O
your	O
input	O
.	O
with	O
the	O
query	O
the	O
documents	O
and	O
then	O
you	O
will	O
have	O
computed	O
the	O
answers	O
to	O
this	O
query	O
which	O
is	O
set	O
of	O
documents	O
that	O
would	O
be	O
useful	O
to	O
the	O
user	O
.	O
so	O
how	O
would	O
you	O
solve	O
the	O
problem	O
in	O
general	O
there	O
are	O
two	O
strategies	O
that	O
we	O
can	O
use	O
.	O
all	O
right	O
the	O
first	O
strategy	O
is	O
to	O
do	O
document	O
selection	O
.	O
and	O
that	O
is	O
we	O
re	O
going	O
to	O
have	O
a	O
binary	O
classification	O
function	O
or	O
binary	O
classified	O
.	O
that	O
s	O
a	O
function	O
that	O
will	O
take	O
a	O
document	O
and	O
query	O
as	O
input	O
and	O
then	O
give	O
a	O
zero	O
or	O
one	O
as	O
output	O
to	O
indicate	O
whether	O
this	O
document	O
is	O
relevant	O
to	O
the	O
query	O
or	O
not	O
.	O
so	O
in	O
this	O
case	O
you	O
can	O
see	O
the	O
document	O
.	O
the	O
the	O
relevant	O
document	O
set	O
is	O
defined	O
as	O
follows	O
.	O
it	O
basically	O
all	O
the	O
documents	O
that	O
have	O
a	O
value	O
of	O
one	O
by	O
this	O
function	O
.	O
and	O
so	O
in	O
this	O
case	O
you	O
can	O
see	O
the	O
system	O
must	O
have	O
decided	O
if	O
a	O
document	O
is	O
relevant	O
or	O
not	O
.	O
basically	O
that	O
has	O
to	O
say	O
whether	O
it	O
s	O
one	O
or	O
zero	O
.	O
and	O
this	O
is	O
called	O
absolute	O
relevance	O
.	O
basically	O
it	O
needs	O
to	O
know	O
exactly	O
whether	O
it	O
s	O
going	O
to	O
be	O
useful	O
to	O
the	O
user	O
.	O
alternatively	O
there	O
s	O
another	O
strategy	O
called	O
document	O
ranking	O
.	O
now	O
in	O
this	O
case	O
the	O
system	O
is	O
not	O
going	O
to	O
make	O
a	O
call	O
whether	O
a	O
document	O
is	O
relevant	O
or	O
not	O
.	O
rather	O
the	O
system	O
s	O
going	O
to	O
use	O
a	O
real	O
value	O
function	O
f	O
here	O
that	O
would	O
simply	O
give	O
us	O
a	O
value	O
.	O
that	O
would	O
indicate	O
which	O
document	O
is	O
more	O
likely	O
relevant	O
.	O
so	O
it	O
s	O
not	O
going	O
to	O
make	O
a	O
call	O
whether	O
this	O
document	O
is	O
relevant	O
or	O
not	O
but	O
rather	O
it	O
would	O
say	O
which	O
document	O
is	O
more	O
likely	O
relevant	O
.	O
so	O
this	O
function	O
then	O
can	O
be	O
used	O
to	O
rank	O
the	O
documents	O
.	O
and	O
then	O
we	O
re	O
going	O
to	O
let	O
the	O
user	O
decide	O
where	O
to	O
stop	O
when	O
the	O
user	O
looks	O
at	O
the	O
documents	O
.	O
so	O
we	O
have	O
a	O
threshold	O
theta	O
here	O
to	O
determine	O
what	O
documents	O
should	O
be	O
in	O
this	O
approximation	O
set	O
.	O
and	O
we	O
re	O
going	O
to	O
assume	O
that	O
all	O
the	O
documents	O
that	O
are	O
ranked	O
above	O
this	O
threshold	O
are	O
in	O
the	O
set	O
.	O
because	O
in	O
effect	O
these	O
are	O
the	O
documents	O
that	O
we	O
delivered	O
to	O
the	O
user	O
.	O
and	O
theta	O
is	O
a	O
cutoff	O
determined	O
by	O
the	O
user	O
.	O
so	O
here	O
we	O
ve	O
got	O
some	O
collaboration	O
from	O
the	O
user	O
in	O
some	O
sense	O
because	O
we	O
don	O
t	O
really	O
make	O
a	O
cutoff	O
and	O
the	O
user	O
kind	O
of	O
helped	O
the	O
system	O
make	O
a	O
cutoff	O
.	O
so	O
in	O
this	O
case	O
the	O
system	O
only	O
needs	O
to	O
decide	O
if	O
one	O
document	O
is	O
more	O
likely	O
relevant	O
than	O
another	O
.	O
and	O
that	O
is	O
it	O
only	O
needs	O
for	O
determined	O
relative	O
relevance	O
as	O
opposed	O
to	O
absolute	O
relevance	O
.	O
now	O
you	O
can	O
probably	O
already	O
sense	O
that	O
relevant	O
relative	O
relevance	O
would	O
be	O
easier	O
to	O
determine	O
the	O
absolute	O
relevance	O
.	O
because	O
in	O
the	O
first	O
case	O
we	O
have	O
to	O
say	O
exactly	O
whether	O
a	O
document	O
is	O
relevant	O
or	O
not	O
right	O
and	O
it	O
turns	O
out	O
that	O
ranking	O
is	O
indeed	O
generally	O
preferred	O
to	O
document	O
selection	O
.	O
so	O
let	O
s	O
look	O
this	O
these	O
two	O
strategies	O
in	O
more	O
detail	O
.	O
so	O
this	O
pictures	O
shows	O
how	O
it	O
works	O
.	O
so	O
on	O
the	O
left	O
side	O
we	O
see	O
these	O
documents	O
.	O
and	O
we	O
use	O
the	O
pluses	O
to	O
indicate	O
the	O
relevant	B
documents	I
.	O
so	O
we	O
can	O
see	O
the	O
true	O
relevant	B
documents	I
here	O
consists	O
this	O
set	O
of	O
true	O
relevant	B
documents	I
consists	O
of	O
these	O
pluses	O
these	O
documents	O
.	O
and	O
with	O
the	O
document	O
selection	O
function	O
we	O
can	O
do	O
basically	O
classify	O
them	O
into	O
two	O
groups	O
relevant	B
documents	I
and	O
non-relevant	O
ones	O
.	O
of	O
course	O
the	O
classifier	O
will	O
not	O
be	O
perfect	O
so	O
it	O
will	O
make	O
mistakes	O
.	O
so	O
here	O
we	O
can	O
see	O
in	O
the	O
approximation	O
of	O
the	O
relevant	B
documents	I
we	O
have	O
got	O
some	O
non-relevant	O
documents	O
.	O
and	O
similarly	O
there	O
s	O
a	O
relevant	O
document	O
that	O
that	O
s	O
misclassified	O
as	O
non-relevant	O
.	O
in	O
the	O
case	O
of	O
document	O
ranking	O
we	O
can	O
see	O
the	O
system	O
seems	O
like	O
simply	O
ranks	O
all	O
the	O
documents	O
in	O
the	O
descending	O
order	O
of	O
the	O
scores	O
.	O
and	O
we	O
re	O
going	O
to	O
let	O
the	O
user	O
stop	O
wherever	O
the	O
user	O
wants	O
to	O
stop	O
.	O
so	O
if	O
a	O
user	O
wants	O
to	O
examine	O
more	O
documents	O
then	O
the	O
user	O
will	O
go	O
down	O
the	O
list	O
to	O
examine	O
more	O
and	O
stop	O
at	O
the	O
lower	O
position	O
.	O
but	O
if	O
the	O
user	O
only	O
wants	O
to	O
read	O
a	O
few	O
random	O
documents	O
the	O
user	O
might	O
stop	O
at	O
the	O
top	O
position	O
.	O
so	O
in	O
this	O
case	O
the	O
user	O
stops	O
at	O
d	O
so	O
the	O
effect	O
we	O
have	O
delivered	O
these	O
four	O
documents	O
to	O
our	O
user	O
.	O
so	O
as	O
i	O
said	O
ranking	O
is	O
generally	O
preferred	O
.	O
and	O
one	O
of	O
the	O
reasons	O
is	O
because	O
the	O
classifier	O
in	O
the	O
case	O
of	O
document	O
selection	O
is	O
unlikely	O
accurate	O
.	O
why	O
because	O
the	O
only	O
clue	O
is	O
usually	O
the	O
query	O
.	O
but	O
the	O
query	O
may	O
not	O
be	O
accurate	O
in	O
the	O
sense	O
that	O
it	O
could	O
be	O
overly	O
constrained	O
.	O
for	O
example	O
you	O
might	O
expect	O
the	O
relevant	B
documents	I
to	O
talk	O
about	O
all	O
these	O
topics	O
you	O
by	O
using	O
specific	O
vocabulary	O
and	O
as	O
a	O
result	O
you	O
might	O
match	O
no	O
random	O
documents	O
because	O
in	O
the	O
collection	O
no	O
others	O
have	O
discussed	O
the	O
topic	O
using	O
these	O
vocabularies	O
.	O
all	O
right	O
.	O
so	O
in	O
this	O
case	O
we	O
ll	O
see	O
there	O
is	O
this	O
problem	O
of	O
no	O
relevant	B
documents	I
to	O
return	O
in	O
the	O
case	O
of	O
overly	O
constrained	O
query	O
.	O
on	O
the	O
other	O
hand	O
if	O
the	O
query	O
is	O
under	O
constrained	O
for	O
example	O
if	O
the	O
query	O
does	O
not	O
have	O
sufficient	O
discriminating	O
words	O
you	O
ll	O
find	O
in	O
relevant	B
documents	I
you	O
may	O
actually	O
end	O
up	O
having	O
.	O
over	O
delivery	O
.	O
and	O
this	O
is	O
when	O
you	O
thought	O
these	O
words	O
might	O
be	O
sufficient	O
to	O
help	O
you	O
find	O
the	O
relevant	B
documents	I
but	O
it	O
turns	O
out	O
that	O
they	O
re	O
not	O
sufficient	O
.	O
and	O
there	O
are	O
many	O
distraction	O
documents	O
using	O
similar	O
words	O
.	O
and	O
so	O
this	O
is	O
the	O
case	O
of	O
over	O
delivery	O
.	O
unfortunately	O
it	O
s	O
very	O
hard	O
to	O
find	O
the	O
right	O
position	O
between	O
these	O
two	O
extremes	O
.	O
why	O
because	O
when	O
the	O
users	O
looking	O
for	O
the	O
information	O
in	O
general	O
the	O
user	O
does	O
not	O
have	O
a	O
good	O
knowledge	O
about	O
the	O
the	O
information	O
to	O
be	O
found	O
.	O
and	O
in	O
that	O
case	O
the	O
user	O
does	O
not	O
have	O
a	O
good	O
knowledge	O
about	O
what	O
vocabularies	O
will	O
be	O
used	O
in	O
those	O
random	O
documents	O
.	O
so	O
it	O
s	O
very	O
hard	O
for	O
a	O
user	O
to	O
pre-specify	O
the	O
right	O
level	O
of	O
of	O
constraints	O
.	O
even	O
if	O
the	O
class	O
file	O
is	O
accurate	O
we	O
also	O
still	O
want	O
to	O
rank	O
these	O
relevant	B
documents	I
because	O
they	O
are	O
generally	O
not	O
equally	O
relevant	O
.	O
relevance	O
is	O
often	O
a	O
matter	O
of	O
degree	O
.	O
so	O
we	O
must	O
prioritize	O
these	O
documents	O
for	O
user	O
to	O
exam	O
.	O
and	O
this	O
note	O
that	O
this	O
prioritization	O
is	O
very	O
important	O
because	O
a	O
user	O
cannot	O
digest	O
all	O
the	O
contents	O
at	O
once	O
.	O
the	O
user	O
generally	O
would	O
have	O
to	O
look	O
at	O
each	O
document	O
sequentially	O
.	O
and	O
therefore	O
it	O
would	O
make	O
sense	O
to	O
feed	O
users	O
with	O
the	O
most	O
relevant	B
documents	I
and	O
that	O
s	O
what	O
ranking	O
is	O
doing	O
.	O
so	O
for	O
these	O
reasons	O
ranking	O
is	O
generally	O
preferred	O
.	O
now	O
this	O
preference	O
also	O
has	O
a	O
theoretical	O
justification	O
and	O
this	O
is	O
given	O
by	O
the	O
probability	B
ranking	I
principle	I
.	O
in	O
the	O
end	O
of	O
this	O
lecture	O
there	O
is	O
a	O
reference	O
for	O
this	O
.	O
this	O
principal	O
says	O
returning	O
a	O
ranked	O
list	O
of	O
documents	O
in	O
descending	O
order	O
of	O
probability	O
that	O
a	O
document	O
is	O
relevant	O
to	O
the	O
query	O
is	O
the	O
optimal	O
strategy	O
under	O
the	O
following	O
two	O
assumptions	O
.	O
first	O
the	O
utility	O
of	O
a	O
document	O
to	O
a	O
user	O
is	O
independent	O
of	O
the	O
utility	O
of	O
any	O
other	O
document	O
.	O
second	O
a	O
user	O
would	O
be	O
assumed	O
to	O
browse	O
the	O
results	O
sequentially	O
.	O
now	O
it	O
s	O
easy	O
to	O
understand	O
why	O
these	O
two	O
assumptions	O
are	O
needed	O
in	O
order	O
to	O
justify	O
for	O
the	O
ranking	B
strategy	I
.	O
because	O
if	O
the	O
documents	O
are	O
independent	O
then	O
we	O
can	O
evaluate	O
the	O
utility	O
of	O
each	O
document	O
that	O
s	O
separate	O
.	O
and	O
this	O
would	O
allow	O
us	O
to	O
compute	O
a	O
score	O
for	O
each	O
document	O
independently	O
.	O
and	O
then	O
we	O
re	O
going	O
to	O
rank	O
these	O
documents	O
based	O
on	O
those	O
scores	O
.	O
the	O
second	O
assumption	O
is	O
to	O
say	O
that	O
the	O
user	O
would	O
indeed	O
follow	O
the	O
rank	O
list	O
.	O
if	O
the	O
user	O
is	O
not	O
going	O
to	O
follow	O
the	O
ranked	O
list	O
is	O
not	O
going	O
to	O
examine	O
the	O
documents	O
sequentially	O
then	O
obviously	O
the	O
ordering	O
would	O
not	O
be	O
optimal	O
.	O
so	O
under	O
these	O
two	O
assumptions	O
we	O
can	O
theoretically	O
justify	O
the	O
ranking	B
strategy	I
is	O
in	O
fact	O
the	O
best	O
that	O
you	O
could	O
do	O
.	O
now	O
i	O
ve	O
put	O
one	O
question	O
here	O
.	O
do	O
these	O
assumptions	O
hold	O
now	O
i	O
suggest	O
you	O
to	O
pause	O
the	O
lecture	O
for	O
a	O
moment	O
to	O
think	O
about	O
these	O
.	O
now	O
can	O
you	O
think	O
of	O
some	O
examples	O
that	O
would	O
suggest	O
these	O
assumptions	O
aren	O
t	O
necessarily	O
true	O
now	O
if	O
you	O
think	O
for	O
a	O
moment	O
you	O
may	O
realize	O
none	O
of	O
the	O
assumptions	O
is	O
actually	O
true	O
.	O
for	O
example	O
in	O
the	O
case	O
of	O
independence	O
assumption	O
we	O
might	O
have	O
identical	O
documents	O
that	O
have	O
similar	O
content	O
or	O
exactly	O
the	O
same	O
content	O
.	O
if	O
you	O
look	O
at	O
each	O
of	O
them	O
alone	O
each	O
is	O
relevant	O
.	O
but	O
if	O
the	O
user	O
has	O
already	O
seen	O
one	O
of	O
them	O
we	O
assume	O
it	O
s	O
generally	O
not	O
very	O
useful	O
for	O
the	O
user	O
to	O
see	O
another	O
similar	O
or	O
duplicate	O
one	O
.	O
so	O
clearly	O
the	O
utility	O
of	O
a	O
document	O
is	O
dependent	O
on	O
other	O
documents	O
that	O
the	O
user	O
has	O
seen	O
.	O
in	O
some	O
other	O
cases	O
you	O
might	O
see	O
a	O
scenario	O
where	O
one	O
document	O
that	O
may	O
not	O
be	O
useful	O
to	O
the	O
user	O
but	O
when	O
three	O
particular	O
documents	O
are	O
put	O
together	O
they	O
provide	O
answer	O
to	O
the	O
user	O
s	O
question	O
.	O
so	O
this	O
is	O
collective	O
relevance	O
.	O
and	O
that	O
also	O
suggests	O
that	O
the	O
value	O
of	O
the	O
document	O
might	O
depend	O
on	O
other	O
documents	O
.	O
sequential	O
browsing	O
generally	O
would	O
make	O
sense	O
if	O
you	O
have	O
a	O
ranked	O
list	O
there	O
.	O
but	O
even	O
if	O
you	O
have	O
a	O
run	O
list	O
there	O
is	O
evidence	O
showing	O
that	O
users	O
don	O
t	O
always	O
just	O
go	O
strictly	O
sequentially	O
through	O
the	O
entire	O
list	O
.	O
they	O
sometimes	O
would	O
look	O
at	O
the	O
bottom	O
for	O
example	O
or	O
skip	O
some	O
.	O
and	O
if	O
you	O
think	O
about	O
the	O
more	O
complicated	O
interfaces	O
that	O
would	O
possibly	O
use	O
like	O
two	O
dimensional	O
interface	O
where	O
you	O
can	O
put	O
additional	O
information	O
on	O
the	O
screen	O
then	O
sequential	O
browsing	O
is	O
a	O
very	O
restrictive	O
assumption	O
.	O
so	O
the	O
point	O
here	O
is	O
that	O
none	O
of	O
these	O
assumptions	O
is	O
really	O
true	O
but	O
nevertheless	O
the	O
probability	B
ranking	I
principle	I
establishes	O
some	O
solid	O
foundation	O
for	O
ranking	O
as	O
a	O
primary	O
task	O
for	O
search	B
engines	I
.	O
and	O
this	O
has	O
actually	O
been	O
the	O
basis	O
for	O
a	O
lot	O
of	O
research	O
work	O
in	O
information	B
retrieval	I
.	O
and	O
many	O
algorithms	O
have	O
been	O
designed	O
based	O
on	O
this	O
assumption	O
.	O
despite	O
that	O
the	O
assumptions	O
aren	O
t	O
necessarily	O
true	O
.	O
and	O
we	O
can	O
address	O
this	O
problem	O
by	O
doing	O
post	O
processing	O
of	O
a	O
ranked	O
list	O
.	O
for	O
example	O
to	O
remove	O
redundancy	O
.	O
so	O
to	O
summarize	O
this	O
lecture	O
the	O
main	O
points	O
that	O
you	O
can	O
take	O
away	O
are	O
the	O
following	O
.	O
first	O
text	B
retrieval	I
is	O
an	O
empirically	O
defined	O
problem	O
.	O
and	O
that	O
means	O
which	O
algorithm	O
is	O
better	O
must	O
be	O
judged	O
by	O
the	O
users	O
.	O
second	O
document	O
ranking	O
is	O
generally	O
prefer	O
and	O
this	O
is	O
will	O
help	O
users	O
prioritize	O
examination	O
of	O
search	O
results	O
.	O
and	O
this	O
is	O
also	O
to	O
bypass	O
the	O
difficulty	O
in	O
determining	O
absolute	O
relevance	O
because	O
we	O
can	O
get	O
some	O
help	O
from	O
users	O
in	O
determining	O
where	O
to	O
make	O
the	O
cut	O
off	O
.	O
it	O
s	O
more	O
flexible	O
.	O
so	O
this	O
further	O
suggests	O
that	O
the	O
main	O
technical	O
challenge	O
in	O
designing	O
the	O
search	B
engine	I
is	O
with	O
designing	O
effective	O
ranking	B
function	I
.	O
in	O
other	O
words	O
we	O
need	O
to	O
define	O
what	O
is	O
the	O
value	O
of	O
this	O
function	O
f	O
on	O
the	O
query	O
and	O
document	O
pair	O
.	O
now	O
how	O
to	O
design	O
such	O
a	O
function	O
is	O
a	O
main	O
topic	O
in	O
the	O
following	O
lectures	O
.	O
there	O
are	O
two	O
suggested	O
additional	O
readings	O
.	O
the	O
first	O
is	O
the	O
classic	O
paper	O
on	O
probability	B
ranking	I
principle	I
.	O
the	O
second	O
is	O
a	O
must	O
read	O
for	O
anyone	O
doing	O
research	O
information	B
retrieval	I
.	O
it	O
s	O
classical	O
ir	O
book	O
which	O
has	O
excellent	O
coverage	O
of	O
the	O
main	O
research	O
results	O
in	O
early	O
days	O
up	O
to	O
the	O
time	O
when	O
the	O
book	O
was	O
written	O
.	O
chapter	O
six	O
of	O
this	O
book	O
has	O
an	O
in	O
depth	O
discussion	O
of	O
the	O
probability	O
of	O
the	O
ranking	O
principal	O
and	O
the	O
probabilistic	B
retrieval	B
models	I
in	O
general	O
.	O
this	O
lecture	O
is	O
a	O
overview	O
of	O
text	B
retrieval	B
methods	I
.	O
in	O
the	O
previous	O
lecture	O
we	O
introduced	O
you	O
to	O
the	O
problem	O
of	O
text	B
retrieval	I
.	O
we	O
explained	O
that	O
the	O
main	O
problem	O
is	O
to	O
design	O
a	O
ranking	B
function	I
to	O
rank	O
documents	O
for	O
a	O
query	O
.	O
in	O
this	O
lecture	O
we	O
will	O
give	O
a	O
overview	O
of	O
different	O
ways	O
of	O
designing	O
this	O
ranking	B
function	I
.	O
so	O
the	O
problem	O
is	O
the	O
following	O
.	O
we	O
have	O
a	O
query	O
that	O
has	O
a	O
sequence	O
of	O
words	O
and	O
a	O
document	O
that	O
that	O
s	O
also	O
a	O
sequence	O
of	O
words	O
and	O
we	O
hope	O
to	O
define	O
the	O
function	O
f	O
that	O
can	O
compute	O
a	O
score	O
based	O
on	O
the	O
query	O
and	O
document	O
.	O
so	O
the	O
main	O
challenge	O
you	O
here	O
is	O
with	O
designing	O
a	O
good	O
ranking	B
function	I
that	O
can	O
rank	O
all	O
the	O
relevant	B
documents	I
on	O
top	O
of	O
all	O
the	O
non-relevant	O
ones	O
.	O
now	O
clearly	O
this	O
means	O
our	O
function	O
must	O
be	O
able	O
to	O
measure	O
the	O
likelihood	O
that	O
a	O
document	O
d	O
is	O
relevant	O
to	O
a	O
query	O
q	O
.	O
that	O
also	O
means	O
we	O
have	O
to	O
have	O
some	O
way	O
to	O
define	O
relevance	O
.	O
in	O
particular	O
in	O
order	O
to	O
implement	O
the	O
program	O
to	O
do	O
that	O
we	O
have	O
to	O
have	O
a	O
computational	O
definition	O
of	O
relevance	O
and	O
we	O
achieve	O
this	O
goal	O
by	O
designing	O
a	O
retrieval	B
model	I
which	O
gives	O
us	O
a	O
formalization	O
of	O
relevance	O
.	O
now	O
over	O
many	O
decades	O
researchers	O
have	O
designed	O
many	O
different	O
kinds	O
of	O
retrieval	B
models	I
and	O
they	O
fall	O
into	O
different	O
categories	O
.	O
first	O
one	O
fair	O
many	O
of	O
the	O
models	O
are	O
based	O
on	O
the	O
similarity	O
idea	O
.	O
basically	O
we	O
assume	O
that	O
if	O
a	O
document	O
is	O
more	O
similar	O
to	O
the	O
query	O
than	O
another	O
document	O
is	O
then	O
we	O
would	O
say	O
the	O
first	O
document	O
is	O
more	O
relevant	O
than	O
the	O
second	O
one	O
.	O
so	O
in	O
this	O
case	O
the	O
ranking	B
function	I
is	O
defined	O
as	O
the	O
similarity	O
between	O
the	O
query	O
and	O
the	O
document	O
.	O
one	O
well	O
known	O
example	O
in	O
this	O
case	O
is	O
vector	B
space	I
model	I
which	O
we	O
will	O
cover	O
more	O
in	O
detail	O
later	O
in	O
the	O
lecture	O
.	O
the	O
second	O
kind	O
of	O
models	O
are	O
called	O
probabilistic	B
models	I
.	O
in	O
this	O
family	O
of	O
models	O
we	O
follow	O
a	O
very	O
different	O
strategy	O
.	O
while	O
we	O
assume	O
that	O
queries	O
and	O
documents	O
are	O
all	O
observations	O
from	O
random	O
variables	O
and	O
we	O
assume	O
there	O
is	O
a	O
binary	O
random	O
variable	O
called	O
r	O
here	O
to	O
indicate	O
whether	O
a	O
document	O
is	O
relevant	O
to	O
a	O
query	O
.	O
we	O
then	O
define	O
the	O
score	O
of	O
document	O
with	O
respect	O
to	O
a	O
query	O
as	O
is	O
a	O
probability	O
that	O
this	O
random	O
variable	O
r	O
is	O
equal	O
to	O
given	O
a	O
particular	O
document	O
and	O
query	O
.	O
there	O
are	O
different	O
cases	O
of	O
such	O
a	O
general	O
idea	O
.	O
one	O
is	O
classic	O
probabilistic	B
model	I
another	O
is	O
language	B
model	I
yet	O
another	O
is	O
divergence-from-randomness	O
model	O
.	O
in	O
a	O
later	O
lecture	O
we	O
will	O
talk	O
more	O
about	O
the	O
one	O
case	O
which	O
is	O
language	B
model	I
.	O
the	O
third	O
kind	O
of	O
models	O
of	O
this	O
is	O
probabilistic	O
inference	B
.	O
so	O
here	O
the	O
idea	O
is	O
to	O
associate	O
uncertainty	O
to	O
inference	B
rules	I
.	O
and	O
we	O
can	O
then	O
quantify	O
the	O
probability	O
that	O
we	O
can	O
show	O
that	O
the	O
query	O
follows	O
from	O
the	O
document	O
.	O
finally	O
there	O
is	O
also	O
a	O
family	O
of	O
models	O
that	O
are	O
using	O
axiomatic	O
thinking	O
.	O
here	O
the	O
idea	O
is	O
to	O
define	O
a	O
set	O
of	O
constraints	O
that	O
we	O
hope	O
a	O
good	O
retrieval	O
function	O
to	O
satisfy	O
.	O
so	O
in	O
this	O
case	O
the	O
problem	O
is	O
you	O
seek	O
a	O
good	O
ranking	B
function	I
that	O
can	O
satisfy	O
all	O
the	O
desired	O
constraints	O
.	O
interestingly	O
although	O
these	O
different	O
models	O
are	O
based	O
on	O
different	O
thinking	O
in	O
the	O
end	O
the	O
retrieval	O
function	O
tends	O
to	O
be	O
very	O
similar	O
.	O
and	O
these	O
functions	O
tend	O
to	O
also	O
involve	O
similar	O
variables	O
.	O
so	O
now	O
let	O
s	O
take	O
a	O
look	O
at	O
the	O
the	O
common	O
form	O
of	O
a	O
state	O
of	O
that	O
retrieval	B
model	I
and	O
examine	O
some	O
of	O
the	O
common	O
ideas	O
used	O
in	O
all	O
these	O
models	O
.	O
first	O
these	O
models	O
are	O
all	O
based	O
on	O
the	O
assumption	O
of	O
using	O
bag	B
of	I
words	I
for	O
representing	O
text	O
.	O
and	O
we	O
explained	O
this	O
in	O
the	O
natural	B
language	I
processing	I
lecture	O
.	O
bag	B
of	I
words	I
representation	I
remains	O
the	O
main	O
representation	O
used	O
in	O
all	O
the	O
search	B
engines	I
.	O
so	O
with	O
this	O
assumption	O
the	O
score	O
of	O
a	O
query	O
like	O
a	O
presidential	O
campaign	O
news	O
with	O
respect	O
to	O
a	O
document	O
d	O
here	O
would	O
be	O
based	O
on	O
scores	O
computed	O
at	O
based	O
on	O
each	O
individual	O
word	O
.	O
and	O
that	O
means	O
the	O
score	O
would	O
depend	O
on	O
the	O
score	O
of	O
each	O
word	O
such	O
as	O
presidential	O
campaign	O
and	O
news	O
.	O
here	O
we	O
can	O
see	O
there	O
are	O
three	O
different	O
components	O
each	O
corresponding	O
to	O
how	O
well	O
the	O
document	O
matches	O
each	O
of	O
the	O
query	B
words	I
.	O
inside	O
of	O
these	O
functions	O
we	O
see	O
a	O
number	O
of	O
heuristics	O
views	O
.	O
so	O
for	O
example	O
one	O
factor	O
that	O
affects	O
the	O
function	O
g	O
here	O
is	O
how	O
many	O
times	O
does	O
the	O
word	O
presidential	O
occur	O
in	O
the	O
document	O
this	O
is	O
called	O
a	O
term	B
frequency	I
or	O
tf	B
.	O
we	O
might	O
also	O
denote	O
as	O
c	O
of	O
presidential	O
and	O
d	O
.	O
in	O
general	O
if	O
the	O
word	O
occurs	O
more	O
frequently	O
in	O
the	O
document	O
then	O
the	O
value	O
of	O
this	O
function	O
would	O
be	O
larger	O
.	O
another	O
factor	O
is	O
how	O
long	O
is	O
the	O
document	O
and	O
this	O
is	O
so	O
to	O
use	O
the	O
document	O
length	O
for	O
score	O
.	O
in	O
general	O
if	O
a	O
term	O
occurs	O
in	O
a	O
long	O
document	O
that	O
many	O
times	O
it	O
s	O
not	O
as	O
significant	O
as	O
if	O
it	O
occurred	O
the	O
same	O
number	O
of	O
times	O
in	O
a	O
short	O
document	O
.	O
because	O
in	O
the	O
long	O
document	O
any	O
term	O
is	O
expected	O
to	O
occur	O
more	O
frequently	O
.	O
finally	O
there	O
is	O
this	O
factor	O
called	O
a	O
document	B
frequency	I
and	O
that	O
is	O
we	O
also	O
want	O
to	O
look	O
at	O
how	O
often	O
presidential	O
occurs	O
in	O
the	O
entire	O
collection	O
.	O
and	O
we	O
call	O
this	O
document	B
frequency	I
or	O
df	B
of	O
presidential	O
.	O
and	O
in	O
some	O
other	O
models	O
we	O
might	O
also	O
use	O
a	O
probability	O
to	O
characterize	O
this	O
information	O
.	O
so	O
here	O
i	O
show	O
the	O
probability	O
of	O
presidential	O
in	O
the	O
collection	O
.	O
so	O
all	O
these	O
are	O
trying	O
to	O
characterize	O
the	O
popularity	O
of	O
the	O
term	O
in	O
the	O
collection	O
.	O
in	O
general	O
matching	O
a	O
rare	O
term	O
in	O
the	O
collection	O
is	O
contributing	O
more	O
to	O
the	O
overall	O
score	O
then	O
matching	O
a	O
common	O
term	O
.	O
so	O
this	O
captures	O
some	O
of	O
the	O
main	O
ideas	O
used	O
in	O
pretty	O
much	O
all	O
the	O
state	O
of	O
the	O
art	O
retrieval	B
models	I
.	O
so	O
now	O
a	O
natural	O
question	O
is	O
which	O
model	O
works	O
the	O
best	O
now	O
it	O
turns	O
out	O
that	O
many	O
models	O
work	O
equally	O
well	O
so	O
here	O
i	O
listed	O
the	O
four	O
major	O
models	O
that	O
are	O
generally	O
regarded	O
as	O
a	O
state	O
of	O
the	O
art	O
retrieval	B
models	I
.	O
pivoted	O
length	B
normalization	I
bm	O
query	O
likelihood	O
pl	O
	O
.	O
when	O
optimized	O
these	O
models	O
tend	O
to	O
perform	O
similarly	O
and	O
this	O
was	O
discussed	O
in	O
detail	O
in	O
this	O
reference	O
at	O
the	O
end	O
of	O
this	O
lecture	O
.	O
among	O
all	O
these	O
bm	O
is	O
probably	O
the	O
most	O
popular	O
.	O
it	O
s	O
most	O
likely	O
that	O
this	O
has	O
been	O
used	O
in	O
virtually	O
all	O
the	O
search	B
engines	I
and	O
you	O
will	O
also	O
often	O
see	O
this	O
method	O
discussed	O
in	O
research	O
papers	O
.	O
and	O
we	O
ll	O
talk	O
more	O
about	O
this	O
method	O
later	O
in	O
some	O
other	O
lectures	O
.	O
so	O
to	O
summarize	O
the	O
main	O
points	O
made	O
in	O
this	O
lecture	O
are	O
first	O
the	O
design	O
of	O
a	O
good	O
ranking	B
function	I
pre-requires	O
a	O
computational	O
definition	O
of	O
relevance	O
and	O
we	O
achieve	O
this	O
goal	O
by	O
designing	O
a	O
proper	O
retrieval	B
model	I
.	O
second	O
many	O
models	O
are	O
equally	O
effective	O
but	O
we	O
don	O
t	O
have	O
a	O
single	O
winner	O
here	O
.	O
researchers	O
are	O
still	O
actively	O
working	O
on	O
this	O
problem	O
trying	O
to	O
find	O
a	O
truly	O
optimal	O
retrieval	B
model	I
.	O
finally	O
the	O
state	O
of	O
the	O
art	O
ranking	B
functions	I
tend	O
to	O
rely	O
on	O
the	O
following	O
ideas	O
.	O
first	O
bag	B
of	I
words	I
representation	I
.	O
second	O
tf	B
and	O
the	O
document	B
frequency	I
of	O
words	O
.	O
such	O
information	O
is	O
used	O
when	O
ranking	B
function	I
to	O
determine	O
the	O
overall	O
contribution	O
of	O
matching	O
a	O
word	O
and	O
document	O
length	O
.	O
these	O
are	O
often	O
combined	O
in	O
interesting	O
ways	O
and	O
we	O
ll	O
discuss	O
how	O
exactly	O
they	O
are	O
combined	O
to	O
rank	O
documents	O
in	O
the	O
lectures	O
later	O
.	O
there	O
are	O
two	O
suggested	O
additional	O
readings	O
if	O
you	O
have	O
time	O
.	O
the	O
first	O
is	O
a	O
paper	O
where	O
you	O
can	O
find	O
a	O
detailed	O
discussion	O
and	O
comparison	O
of	O
multiple	O
state	O
of	O
the	O
art	O
models	O
.	O
the	O
second	O
is	O
a	O
book	O
with	O
a	O
chapter	O
that	O
gives	O
a	O
broad	O
review	O
of	O
different	O
retrieval	B
models	I
.	O
	O
.	O
this	O
lecture	O
is	O
about	O
the	O
vector	B
space	I
retrieval	B
model	I
.	O
we	O
re	O
going	O
to	O
give	O
an	O
introduction	O
to	O
its	O
basic	O
idea	O
.	O
in	O
the	O
last	O
lecture	O
we	O
talked	O
about	O
the	O
different	O
ways	O
of	O
designing	O
a	O
retrieval	B
model	I
which	O
would	O
give	O
us	O
a	O
different	O
the	O
ranking	B
function	I
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
the	O
specific	O
way	O
of	O
design	O
the	O
ramping	O
function	O
called	O
a	O
vector	B
space	I
mutual	O
model	O
.	O
and	O
we	O
re	O
going	O
to	O
give	O
a	O
brief	O
introduction	O
to	O
the	O
basic	O
idea	O
.	O
vector	B
space	I
model	I
is	O
a	O
special	O
case	O
of	O
similarity	O
based	O
models	O
as	O
we	O
discussed	O
before	O
.	O
which	O
means	O
we	O
assume	O
relevance	O
is	O
roughly	O
similarity	O
between	O
a	O
document	O
and	O
a	O
query	O
.	O
now	O
whether	O
this	O
assumption	O
is	O
true	O
is	O
actually	O
a	O
question	O
.	O
but	O
in	O
order	O
to	O
solve	O
our	O
search	O
problem	O
we	O
have	O
to	O
convert	O
the	O
vague	O
notion	O
of	O
relevance	O
into	O
a	O
more	O
precise	O
definition	O
that	O
can	O
be	O
implemented	O
with	O
the	O
programming	O
language	O
.	O
so	O
in	O
this	O
process	O
we	O
have	O
to	O
make	O
a	O
number	O
of	O
assumptions	O
.	O
this	O
is	O
the	O
first	O
assumption	O
that	O
we	O
make	O
here	O
.	O
basically	O
we	O
assume	O
that	O
if	O
a	O
document	O
is	O
more	O
similar	O
to	O
a	O
query	O
than	O
another	O
document	O
then	O
the	O
first	O
document	O
would	O
be	O
assumed	O
to	O
be	O
more	O
relevant	O
than	O
the	O
second	O
one	O
.	O
and	O
this	O
is	O
the	O
basis	O
for	O
ranking	O
documents	O
in	O
this	O
approach	O
.	O
again	O
it	O
s	O
questionable	O
whether	O
this	O
is	O
really	O
the	O
best	O
definition	O
for	O
relevance	O
.	O
as	O
we	O
will	O
see	O
later	O
there	O
are	O
other	O
ways	O
to	O
model	O
relevance	O
.	O
the	O
first	O
idea	O
of	O
vector	B
space	I
retrieval	B
model	I
is	O
actually	O
very	O
easy	O
to	O
understand	O
.	O
imagine	O
a	O
high	O
dimensional	O
space	O
where	O
each	O
dimension	O
corresponds	O
to	O
a	O
term	O
.	O
so	O
here	O
i	O
show	O
a	O
three	O
dimensional	O
space	O
with	O
three	O
words	O
programming	O
library	O
and	O
presidential	O
.	O
so	O
each	O
term	O
here	O
defines	O
one	O
dimension	O
.	O
now	O
we	O
can	O
consider	O
vectors	O
in	O
this	O
three	O
dimensional	O
space	O
.	O
and	O
we	O
re	O
going	O
to	O
assume	O
all	O
our	O
documents	O
and	O
the	O
query	O
will	O
be	O
placed	O
in	O
this	O
vector	B
space	I
.	O
so	O
for	O
example	O
one	O
document	O
that	O
might	O
be	O
represented	O
at	O
by	O
this	O
vector	O
d	O
	O
.	O
now	O
this	O
means	O
this	O
document	O
probably	O
covers	O
library	O
and	O
presidential	O
.	O
but	O
it	O
doesn	O
t	O
really	O
talk	O
about	O
programming	O
.	O
all	O
right	O
what	O
does	O
this	O
mean	O
in	O
terms	O
of	O
presentation	O
of	O
document	O
that	O
just	O
means	O
we	O
re	O
going	O
to	O
look	O
at	O
our	O
document	O
from	O
the	O
perspective	O
of	O
this	O
vector	O
.	O
we	O
re	O
going	O
to	O
ignore	O
everything	O
else	O
.	O
basically	O
what	O
we	O
see	O
here	O
is	O
only	O
the	O
vector	O
of	O
the	O
document	O
.	O
of	O
course	O
the	O
document	O
has	O
other	O
information	O
.	O
for	O
example	O
the	O
orders	O
of	O
words	O
are	O
simply	O
ignored	O
and	O
that	O
s	O
because	O
we	O
re	O
assume	O
that	O
the	O
words	O
.	O
so	O
with	O
this	O
representation	O
you	O
have	O
already	O
seen	O
d	O
seems	O
to	O
suggest	O
a	O
topic	O
in	O
either	O
presidential	O
library	O
.	O
now	O
this	O
is	O
different	O
from	O
another	O
document	O
.	O
which	O
might	O
be	O
represented	O
as	O
a	O
different	O
vector	O
d	O
here	O
.	O
now	O
in	O
this	O
case	O
the	O
document	O
that	O
covers	O
programming	O
and	O
library	O
but	O
it	O
doesn	O
t	O
talk	O
about	O
presidential	O
.	O
so	O
what	O
does	O
this	O
remind	O
you	O
well	O
you	O
can	O
probably	O
guess	O
the	O
topic	O
is	O
likely	O
about	O
program	O
language	O
and	O
the	O
library	O
is	O
software	O
library	O
library	O
.	O
so	O
this	O
shows	O
that	O
by	O
using	O
this	O
vector	B
space	I
representation	O
we	O
can	O
actually	O
capture	O
the	O
differences	O
between	O
topics	O
of	O
documents	O
.	O
now	O
you	O
can	O
also	O
imagine	O
there	O
are	O
other	O
vectors	O
.	O
for	O
example	O
d	O
is	O
pointing	O
in	O
that	O
direction	O
that	O
might	O
be	O
about	O
presidential	O
programming	O
.	O
and	O
in	O
fact	O
we	O
re	O
going	O
to	O
place	O
all	O
the	O
documents	O
in	O
this	O
vector	B
space	I
.	O
and	O
they	O
will	O
be	O
pointing	O
to	O
all	O
kinds	O
of	O
directions	O
.	O
and	O
similarly	O
we	O
re	O
going	O
to	O
place	O
our	O
query	O
also	O
in	O
this	O
space	O
as	O
another	O
vector	O
.	O
and	O
then	O
we	O
re	O
going	O
to	O
measure	O
the	O
similarity	O
between	O
the	O
query	O
vector	O
and	O
every	O
document	O
vector	O
.	O
so	O
in	O
this	O
case	O
for	O
example	O
we	O
can	O
easily	O
see	O
d	O
seems	O
to	O
be	O
the	O
closest	O
of	O
to	O
this	O
query	O
factor	O
and	O
therefore	O
d	O
will	O
be	O
ranked	O
above	O
others	O
.	O
so	O
this	O
was	O
a	O
basically	O
the	O
main	O
idea	O
of	O
the	O
the	O
vector	B
space	I
model	I
.	O
so	O
to	O
be	O
more	O
pri	O
precise	O
be	O
more	O
precise	O
.	O
vector	B
space	I
model	I
is	O
a	O
framework	O
.	O
in	O
this	O
framework	O
we	O
make	O
the	O
following	O
assumptions	O
.	O
first	O
we	O
represent	O
a	O
document	O
and	O
query	O
by	O
a	O
term	O
vector	O
.	O
so	O
here	O
a	O
term	O
can	O
be	O
any	O
basic	O
concept	O
.	O
for	O
example	O
a	O
word	O
or	O
a	O
phrase	O
or	O
even	O
enneagram	O
of	O
characters	O
.	O
those	O
are	O
a	O
sequence	O
of	O
characters	O
inside	O
a	O
word	O
.	O
each	O
term	O
is	O
assumed	O
to	O
define	O
one	O
dimension	O
.	O
therefore	O
n	O
terms	O
.	O
in	O
our	O
vocabulary	O
we	O
define	O
n-dimensional	O
space	O
.	O
a	O
query	O
vector	O
would	O
consist	O
of	O
a	O
number	O
of	O
elements	O
corresponding	O
to	O
the	O
weights	O
of	O
different	O
terms	O
.	O
each	O
document	O
vector	O
is	O
also	O
similar	O
.	O
it	O
has	O
a	O
number	O
of	O
elements	O
and	O
each	O
value	O
of	O
each	O
element	O
is	O
indicating	O
that	O
weight	O
of	O
the	O
corresponding	O
term	O
.	O
here	O
you	O
can	O
see	O
we	O
have	O
seen	O
there	O
are	O
n	O
dimensions	O
.	O
therefore	O
there	O
are	O
n	O
elements	O
each	O
corresponding	O
to	O
the	O
weight	O
on	O
the	O
particular	O
term	O
.	O
so	O
the	O
relevance	O
in	O
this	O
case	O
would	O
be	O
assume	O
to	O
be	O
the	O
similarity	O
between	O
the	O
two	O
vectors	O
therefore	O
our	O
range	O
in	O
function	O
is	O
also	O
defined	O
as	O
the	O
similarity	O
between	O
the	O
query	O
vector	O
and	O
document	O
vector	O
.	O
now	O
if	O
i	O
ask	O
you	O
to	O
write	O
the	O
program	O
to	O
the	O
internet	O
this	O
approach	O
in	O
the	O
search	B
engine	I
.	O
you	O
would	O
realize	O
that	O
this	O
was	O
far	O
from	O
clear	O
right	O
we	O
haven	O
t	O
seen	O
a	O
lot	O
of	O
things	O
in	O
detail	O
therefore	O
it	O
s	O
impossible	O
to	O
actually	O
write	O
the	O
program	O
to	O
implement	O
this	O
.	O
that	O
s	O
why	O
i	O
said	O
this	O
is	O
a	O
framework	O
.	O
and	O
this	O
has	O
to	O
be	O
refined	O
in	O
order	O
to	O
actually	O
suggest	O
a	O
particular	O
function	O
that	O
you	O
can	O
implement	O
on	O
the	O
computer	O
.	O
so	O
what	O
does	O
this	O
framework	O
not	O
serve	O
well	O
it	O
actually	O
hasn	O
t	O
set	O
many	O
things	O
that	O
would	O
be	O
required	O
in	O
order	O
to	O
implement	O
this	O
function	O
.	O
first	O
it	O
did	O
not	O
say	O
how	O
we	O
should	O
define	O
or	O
select	O
the	O
basic	O
concepts	O
exactly	O
.	O
we	O
clearly	O
assume	O
the	O
concepts	O
are	O
orthogonal	O
otherwise	O
there	O
will	O
be	O
redundancy	O
.	O
for	O
example	O
if	O
two	O
synonyms	O
are	O
somehow	O
distinguished	O
as	O
two	O
different	O
concepts	O
.	O
then	O
they	O
would	O
be	O
defined	O
in	O
two	O
different	O
dimensions	O
.	O
and	O
then	O
that	O
would	O
clearly	O
cause	O
a	O
redundancy	O
here	O
.	O
or	O
overemphasizing	O
of	O
matching	O
this	O
concept	O
.	O
because	O
it	O
would	O
be	O
as	O
if	O
you	O
matched	O
the	O
two	O
dimensions	O
when	O
you	O
actually	O
matched	O
one	O
semantic	O
concept	O
.	O
secondly	O
it	O
did	O
not	O
say	O
how	O
we	O
exactly	O
should	O
place	O
documents	O
and	O
query	O
in	O
this	O
space	O
.	O
basically	O
i	O
show	O
you	O
some	O
examples	O
of	O
query	O
and	O
document	O
vectors	O
.	O
but	O
where	O
exactly	O
should	O
the	O
vector	O
for	O
a	O
particular	O
document	O
point	O
to	O
so	O
this	O
is	O
equivalent	O
to	O
how	O
to	O
define	O
the	O
term	O
weights	O
.	O
how	O
do	O
you	O
computer	O
use	O
element	O
values	O
in	O
those	O
vectors	O
this	O
is	O
a	O
very	O
important	O
question	O
because	O
term	O
weight	O
in	O
the	O
query	O
vector	O
indicates	O
the	O
importance	O
of	O
term	O
.	O
so	O
depending	O
on	O
how	O
you	O
assign	O
the	O
weight	O
you	O
might	O
prefer	O
some	O
terms	O
to	O
be	O
matched	O
over	O
others	O
.	O
similarly	O
term	O
weight	O
in	O
the	O
document	O
is	O
also	O
very	O
meaningful	O
.	O
it	O
indicates	O
how	O
well	O
the	O
term	O
characterizes	O
the	O
document	O
.	O
if	O
you	O
got	O
it	O
wrong	O
then	O
you	O
clearly	O
don	O
t	O
represent	O
this	O
document	O
accurately	O
.	O
finally	O
how	O
we	O
define	O
the	O
similarity	O
measure	O
is	O
also	O
not	O
clear	O
.	O
so	O
these	O
questions	O
must	O
be	O
addressed	O
before	O
we	O
can	O
have	O
an	O
operational	O
function	O
that	O
we	O
can	O
actually	O
implement	O
using	O
a	O
program	O
language	O
.	O
so	O
how	O
do	O
we	O
solve	O
these	O
problems	O
is	O
the	O
main	O
topic	O
of	O
the	O
next	O
lecture	O
.	O
	O
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
how	O
to	O
instantiate	O
a	O
vector	B
space	I
model	I
so	O
that	O
we	O
can	O
get	O
a	O
very	O
specific	O
ranking	B
function	I
.	O
so	O
this	O
is	O
the	O
to	O
continue	O
the	O
discussion	O
of	O
the	O
vector	B
space	I
model	I
.	O
which	O
is	O
one	O
particular	O
approach	O
to	O
design	O
ranking	B
function	I
.	O
and	O
we	O
are	O
going	O
to	O
talk	O
about	O
how	O
we	O
use	O
the	O
general	O
framework	O
of	O
the	O
the	O
vector	B
space	I
model	I
.	O
as	O
a	O
guidance	O
to	O
instantiate	O
the	O
framework	O
to	O
derive	O
a	O
specific	O
ranking	B
function	I
.	O
and	O
we	O
re	O
going	O
to	O
cover	O
the	O
simplest	O
instantiation	O
of	O
the	O
framework	O
.	O
so	O
as	O
we	O
discussed	O
in	O
the	O
previous	O
lecture	O
.	O
the	O
vector	B
space	I
model	I
is	O
really	O
a	O
framework	O
.	O
it	O
isn	O
t	O
didn	O
t	O
say	O
.	O
as	O
we	O
discussed	O
in	O
the	O
previous	O
lecture	O
vector	B
space	I
model	I
is	O
really	O
a	O
framework	O
.	O
it	O
doesn	O
t	O
say	O
many	O
things	O
.	O
so	O
for	O
example	O
here	O
it	O
shows	O
that	O
it	O
did	O
not	O
say	O
how	O
we	O
should	O
define	O
the	O
dimension	O
.	O
it	O
also	O
did	O
not	O
say	O
how	O
we	O
place	O
a	O
documented	O
vector	O
in	O
this	O
space	O
.	O
it	O
did	O
not	O
say	O
how	O
we	O
place	O
a	O
query	O
vector	O
in	O
this	O
vector	B
space	I
.	O
and	O
finally	O
it	O
did	O
not	O
say	O
how	O
we	O
should	O
match	O
a	O
similarity	O
between	O
the	O
query	O
vector	O
and	O
the	O
document	O
vector	O
.	O
so	O
you	O
can	O
imagine	O
in	O
order	O
to	O
implement	O
this	O
model	O
.	O
we	O
have	O
to	O
see	O
specifically	O
how	O
we	O
are	O
computing	O
these	O
vectors	O
.	O
what	O
is	O
exactly	O
xi	O
and	O
what	O
is	O
exactly	O
yi	O
this	O
will	O
determine	O
where	O
we	O
place	O
the	O
document	O
vector	O
.	O
where	O
we	O
place	O
a	O
query	O
vector	O
.	O
and	O
of	O
course	O
we	O
also	O
need	O
to	O
say	O
exactly	O
what	O
will	O
be	O
the	O
similarity	O
function	O
.	O
so	O
if	O
we	O
can	O
provide	O
a	O
definition	O
of	O
the	O
concepts	O
that	O
would	O
define	O
the	O
dimensions	O
and	O
these	O
xi	O
s	O
or	O
yi	O
s	O
.	O
and	O
then	O
the	O
waits	O
of	O
terms	O
for	O
query	O
and	O
document	O
.	O
then	O
we	O
will	O
be	O
able	O
to	O
place	O
document	O
vectors	O
and	O
query	O
vector	O
in	O
this	O
well	O
defined	O
space	O
.	O
and	O
then	O
if	O
we	O
also	O
specify	O
similarity	O
function	O
then	O
we	O
ll	O
have	O
well	O
defined	O
ranking	B
function	I
.	O
so	O
let	O
s	O
see	O
how	O
we	O
can	O
do	O
that	O
.	O
and	O
think	O
about	O
the	O
the	O
simpliciter	O
instantiation	O
.	O
actually	O
i	O
would	O
suggest	O
you	O
to	O
pause	O
the	O
lecture	O
at	O
this	O
point	O
spend	O
a	O
couple	O
of	O
minute	O
to	O
think	O
about	O
.	O
suppose	O
you	O
are	O
asked	O
to	O
implement	O
this	O
idea	O
.	O
you	O
ve	O
come	O
up	O
with	O
the	O
idea	O
of	O
vector	B
space	I
model	I
.	O
but	O
you	O
still	O
haven	O
t	O
figured	O
out	O
how	O
to	O
compute	O
this	O
vector	O
exactly	O
how	O
to	O
define	O
this	O
similarity	O
function	O
.	O
what	O
would	O
you	O
do	O
so	O
think	O
for	O
a	O
couple	O
of	O
minutes	O
and	O
then	O
proceed	O
.	O
so	O
let	O
s	O
think	O
about	O
some	O
simplest	O
ways	O
of	O
instantiating	O
this	O
vector	B
space	I
model	I
.	O
first	O
how	O
do	O
we	O
define	O
a	O
dimension	O
.	O
well	O
the	O
obvious	O
choice	O
is	O
we	O
use	O
each	O
word	O
in	O
our	O
vocabulary	O
to	O
define	O
a	O
dimension	O
.	O
and	O
a	O
whole	O
issue	O
that	O
there	O
are	O
n	O
words	O
in	O
our	O
vocabulary	O
therefore	O
there	O
are	O
n	O
dimensions	O
.	O
each	O
word	O
defines	O
one	O
dimension	O
.	O
and	O
this	O
is	O
basically	O
the	O
bag	B
of	I
words	I
instantiation	O
.	O
now	O
let	O
s	O
look	O
at	O
how	O
we	O
place	O
vectors	O
in	O
this	O
space	O
.	O
again	O
here	O
the	O
simplest	O
of	O
strategy	O
is	O
to	O
use	O
a	O
bit	B
vector	I
to	O
represent	O
both	O
a	O
query	O
and	O
a	O
document	O
.	O
and	O
that	O
means	O
each	O
element	O
xi	O
and	O
yi	O
would	O
be	O
taking	O
a	O
value	O
of	O
either	O
zero	O
or	O
one	O
.	O
when	O
it	O
s	O
one	O
it	O
means	O
the	O
corresponding	O
word	O
is	O
present	O
in	O
the	O
document	O
or	O
in	O
the	O
query	O
.	O
when	O
it	O
s	O
zero	O
it	O
s	O
going	O
to	O
mean	O
that	O
it	O
s	O
absent	O
.	O
so	O
you	O
can	O
imagine	O
if	O
the	O
user	O
types	O
in	O
a	O
few	O
word	O
in	O
your	O
query	O
.	O
then	O
the	O
query	O
vector	O
we	O
only	O
have	O
a	O
few	O
ones	O
many	O
many	O
zeros	O
.	O
the	O
document	O
vector	O
in	O
general	O
we	O
have	O
more	O
ones	O
of	O
course	O
but	O
we	O
also	O
have	O
many	O
zeros	O
.	O
so	O
it	O
seems	O
the	O
vocabulary	O
is	O
generally	O
very	O
large	O
.	O
many	O
words	O
don	O
t	O
really	O
occur	O
in	O
a	O
document	O
.	O
many	O
words	O
will	O
only	O
occasionally	O
occur	O
in	O
the	O
document	O
.	O
a	O
lot	O
of	O
words	O
will	O
be	O
absent	O
in	O
a	O
particular	O
document	O
.	O
so	O
now	O
we	O
have	O
placed	O
the	O
documents	O
and	O
the	O
query	O
in	O
the	O
vector	B
space	I
.	O
let	O
s	O
look	O
at	O
how	O
we	O
match	O
up	O
the	O
similarity	O
.	O
so	O
a	O
commonly	O
used	O
similarity	O
measure	O
here	O
is	O
dot	B
product	I
.	O
the	O
dot	B
product	I
of	O
two	O
vectors	O
is	O
simply	O
defined	O
as	O
the	O
sum	O
of	O
the	O
products	O
of	O
the	O
corresponding	O
elements	O
of	O
the	O
two	O
vectors	O
.	O
so	O
here	O
we	O
see	O
that	O
it	O
s	O
the	O
product	O
of	O
x	O
and	O
the	O
y	O
	O
.	O
so	O
here	O
.	O
and	O
then	O
x	O
multiplied	O
by	O
y	O
	O
.	O
and	O
then	O
finally	O
xn	O
multiplied	O
by	O
yn	O
.	O
and	O
then	O
we	O
take	O
a	O
sum	O
here	O
.	O
so	O
that	O
s	O
the	O
dot	B
product	I
.	O
now	O
we	O
can	O
represent	O
this	O
in	O
a	O
more	O
general	O
way	O
using	O
a	O
sum	O
here	O
.	O
so	O
this	O
only	O
one	O
of	O
the	O
many	O
different	O
ways	O
of	O
matching	O
the	O
similarity	O
.	O
so	O
now	O
we	O
see	O
that	O
we	O
have	O
defined	O
the	O
the	O
dimensions	O
.	O
we	O
have	O
defined	O
the	O
the	O
vectors	O
.	O
and	O
we	O
have	O
also	O
defined	O
the	O
similarity	O
function	O
.	O
so	O
now	O
we	O
finally	O
have	O
the	O
simplest	O
vector	B
space	I
model	I
.	O
which	O
is	O
based	O
on	O
the	O
bit	B
vector	I
representation	O
dot	B
product	I
similarity	O
and	O
bag	B
of	I
words	I
instantiation	O
.	O
and	O
the	O
formula	O
looks	O
like	O
this	O
.	O
so	O
this	O
is	O
our	O
formula	O
.	O
and	O
that	O
s	O
actually	O
a	O
particular	O
retrieval	O
function	O
a	O
ranking	B
function	I
all	O
right	O
now	O
we	O
can	O
finally	O
implement	O
this	O
function	O
using	O
a	O
program	O
language	O
and	O
then	O
rank	O
documents	O
for	O
query	O
.	O
now	O
at	O
this	O
point	O
you	O
should	O
again	O
pause	O
the	O
lecture	O
to	O
think	O
about	O
how	O
we	O
can	O
interpret	O
this	O
score	O
.	O
so	O
we	O
have	O
gone	O
through	O
the	O
process	O
of	O
modeling	O
the	O
retrieval	O
problem	O
using	O
a	O
vector	B
space	I
model	I
.	O
and	O
then	O
we	O
make	O
assumptions	O
.	O
about	O
how	O
we	O
place	O
vectors	O
in	O
the	O
vector	B
space	I
and	O
how	O
we	O
define	O
the	O
similarity	O
.	O
so	O
in	O
the	O
end	O
we	O
ve	O
got	O
a	O
specific	O
retrieval	O
function	O
shown	O
here	O
.	O
now	O
the	O
next	O
step	O
is	O
to	O
think	O
about	O
what	O
of	O
this	O
individual	O
function	O
actually	O
makes	O
sense	O
i	O
can	O
we	O
expect	O
this	O
function	O
to	O
actually	O
perform	O
well	O
where	O
we	O
use	O
it	O
to	O
ramp	O
it	O
up	O
for	O
use	O
in	O
query	O
.	O
so	O
it	O
s	O
worth	O
thinking	O
about	O
what	O
is	O
this	O
value	O
that	O
we	O
are	O
calculating	O
so	O
in	O
the	O
end	O
we	O
ve	O
got	O
a	O
number	O
but	O
what	O
does	O
this	O
number	O
mean	O
is	O
it	O
meaningful	O
so	O
spend	O
a	O
couple	O
minutes	O
to	O
think	O
about	O
that	O
.	O
and	O
of	O
course	O
the	O
general	O
question	O
here	O
is	O
do	O
you	O
believe	O
this	O
is	O
a	O
good	O
ranking	B
function	I
would	O
it	O
actually	O
work	O
well	O
so	O
again	O
think	O
about	O
how	O
to	O
interpret	O
this	O
value	O
.	O
is	O
it	O
actually	O
meaningful	O
does	O
it	O
mean	O
something	O
so	O
related	O
to	O
how	O
well	O
that	O
document	O
matches	O
the	O
query	O
.	O
so	O
in	O
order	O
to	O
assess	O
whether	O
this	O
simplest	O
vector	B
space	I
model	I
actually	O
works	O
well	O
let	O
s	O
look	O
at	O
the	O
example	O
.	O
so	O
here	O
i	O
show	O
some	O
sample	O
documents	O
and	O
a	O
simple	O
query	O
.	O
the	O
query	O
is	O
news	O
about	O
the	O
presidential	O
campaign	O
.	O
and	O
we	O
have	O
five	O
documents	O
here	O
.	O
they	O
cover	O
different	O
terms	O
in	O
the	O
query	O
.	O
and	O
if	O
you	O
look	O
at	O
the	O
these	O
documents	O
for	O
a	O
moment	O
.	O
you	O
may	O
realize	O
that	O
some	O
documents	O
are	O
probably	O
relevant	O
in	O
some	O
cases	O
or	O
probably	O
not	O
relevant	O
.	O
now	O
if	O
i	O
ask	O
you	O
to	O
rank	O
these	O
documents	O
how	O
would	O
you	O
rank	O
them	O
this	O
is	O
basically	O
our	O
ideal	O
ranking	O
.	O
right	O
.	O
when	O
humans	O
can	O
examine	O
the	O
documents	O
and	O
then	O
try	O
to	O
rank	O
them	O
.	O
now	O
so	O
think	O
for	O
a	O
moment	O
and	O
take	O
a	O
look	O
at	O
this	O
slide	O
.	O
and	O
perhaps	O
by	O
pausing	O
the	O
lecture	O
.	O
so	O
i	O
think	O
most	O
of	O
you	O
would	O
agree	O
that	O
d	O
and	O
d	O
are	O
probably	O
better	O
than	O
others	O
.	O
because	O
they	O
really	O
cover	O
the	O
query	O
well	O
.	O
they	O
match	O
news	O
presidential	O
and	O
campaign	O
.	O
so	O
it	O
looks	O
like	O
that	O
these	O
two	O
documents	O
are	O
probably	O
better	O
than	O
the	O
others	O
.	O
they	O
should	O
be	O
ranked	O
on	O
top	O
.	O
and	O
the	O
other	O
three	O
d	O
d	O
and	O
d	O
are	O
really	O
non-relavant	O
.	O
so	O
we	O
can	O
also	O
say	O
d	O
and	O
d	O
are	O
relevent	O
documents	O
and	O
d	O
d	O
and	O
d	O
are	O
non-relevant	O
.	O
so	O
now	O
lets	O
see	O
if	O
our	O
vector	B
space	I
model	I
could	O
do	O
the	O
same	O
or	O
could	O
do	O
something	O
closer	O
.	O
so	O
let	O
s	O
first	O
think	O
about	O
how	O
we	O
actually	O
use	O
this	O
model	O
to	O
score	O
documents	O
.	O
right	O
here	O
i	O
show	O
two	O
documents	O
d	O
and	O
d	O
and	O
we	O
have	O
the	O
query	O
also	O
here	O
.	O
in	O
the	O
vector	B
space	I
model	I
of	O
course	O
we	O
want	O
to	O
first	O
compute	O
the	O
vectors	O
for	O
these	O
documents	O
and	O
the	O
query	O
.	O
now	O
i	O
issue	O
with	O
the	O
vocabulary	O
here	O
as	O
well	O
so	O
these	O
are	O
the	O
n	O
dimensions	O
that	O
we	O
ll	O
be	O
thinking	O
about	O
.	O
so	O
what	O
do	O
you	O
think	O
is	O
the	O
vector	O
representation	O
for	O
the	O
query	O
note	O
that	O
we	O
are	O
assuming	O
that	O
we	O
only	O
use	O
zero	O
and	O
one	O
to	O
indicate	O
whether	O
a	O
term	O
is	O
absent	O
or	O
present	O
in	O
the	O
query	O
or	O
in	O
the	O
document	O
.	O
so	O
these	O
are	O
zero	O
one	O
bit	B
vectors	I
.	O
so	O
what	O
do	O
you	O
think	O
is	O
the	O
query	O
vector	O
well	O
the	O
query	O
has	O
four	O
words	O
here	O
.	O
so	O
for	O
these	O
four	O
words	O
there	O
would	O
be	O
a	O
one	O
and	O
for	O
the	O
rest	O
there	O
will	O
be	O
zeros	O
.	O
now	O
what	O
about	O
the	O
documents	O
it	O
s	O
the	O
same	O
.	O
so	O
d	O
has	O
two	O
rows	O
news	O
and	O
about	O
.	O
so	O
there	O
are	O
two	O
ones	O
here	O
and	O
the	O
rest	O
are	O
zeros	O
.	O
similarly	O
so	O
now	O
that	O
we	O
have	O
the	O
two	O
vectors	O
let	O
s	O
compute	O
the	O
similarity	O
.	O
and	O
we	O
re	O
going	O
to	O
use	O
dot	B
product	I
.	O
so	O
you	O
can	O
see	O
when	O
we	O
use	O
dot	B
product	I
we	O
just	O
multiply	O
the	O
corresponding	O
elements	O
.	O
right	O
.	O
so	O
these	O
two	O
would	O
be	O
form	O
a	O
be	O
forming	O
a	O
product	O
.	O
and	O
these	O
two	O
will	O
generate	O
another	O
product	O
.	O
and	O
these	O
two	O
would	O
generate	O
yet	O
another	O
product	O
.	O
and	O
so	O
on	O
and	O
so	O
forth	O
.	O
now	O
you	O
can	O
you	O
need	O
to	O
see	O
if	O
we	O
do	O
that	O
.	O
we	O
actually	O
don	O
t	O
have	O
to	O
care	O
about	O
these	O
zeroes	O
because	O
if	O
whenever	O
we	O
have	O
a	O
zero	O
the	O
product	O
will	O
be	O
zero	O
.	O
so	O
when	O
we	O
take	O
a	O
sum	O
over	O
all	O
these	O
pairs	O
then	O
the	O
zero	O
entries	O
will	O
be	O
gone	O
.	O
as	O
long	O
as	O
you	O
have	O
one	O
zero	O
then	O
the	O
product	O
would	O
be	O
zero	O
.	O
so	O
in	O
the	O
fact	O
we	O
re	O
just	O
counting	O
how	O
many	O
pairs	O
of	O
one	O
and	O
one	O
right	O
in	O
this	O
case	O
we	O
have	O
seen	O
two	O
.	O
so	O
the	O
result	O
will	O
be	O
two	O
.	O
so	O
what	O
does	O
that	O
mean	O
well	O
that	O
means	O
this	O
number	O
or	O
the	O
value	O
of	O
this	O
scoring	O
function	O
.	O
is	O
simply	O
the	O
count	O
of	O
how	O
many	O
unique	O
query	B
terms	I
are	O
matched	O
in	O
the	O
document	O
.	O
because	O
if	O
a	O
document	O
if	O
a	O
term	O
is	O
matched	O
in	O
the	O
document	O
then	O
there	O
will	O
be	O
two	O
ones	O
.	O
if	O
it	O
s	O
not	O
then	O
there	O
will	O
be	O
zero	O
on	O
the	O
document	O
side	O
.	O
similarly	O
if	O
the	O
document	O
has	O
a	O
term	O
	O
.	O
but	O
the	O
terms	O
not	O
in	O
the	O
query	O
there	O
will	O
be	O
zero	O
in	O
the	O
query	O
vector	O
.	O
so	O
those	O
don	O
t	O
count	O
.	O
so	O
as	O
a	O
result	O
this	O
scoring	O
function	O
basically	O
meshes	O
how	O
many	O
unique	O
query	B
terms	I
are	O
matched	O
in	O
a	O
document	O
.	O
this	O
is	O
how	O
we	O
interpret	O
this	O
score	O
.	O
now	O
we	O
can	O
also	O
take	O
a	O
look	O
at	O
the	O
d	O
	O
.	O
in	O
this	O
case	O
you	O
can	O
see	O
the	O
result	O
is	O
three	O
.	O
because	O
d	O
matched	O
the	O
three	O
distinctive	O
query	B
words	I
news	O
presidential	O
campaign	O
.	O
whereas	O
d	O
only	O
matched	O
two	O
.	O
now	O
in	O
this	O
case	O
it	O
seems	O
reasonable	O
to	O
rank	O
d	O
on	O
top	O
of	O
d	O
	O
.	O
and	O
this	O
simplest	O
vector	B
space	I
model	I
indeed	O
does	O
that	O
.	O
so	O
that	O
looks	O
pretty	O
good	O
.	O
however	O
if	O
we	O
examine	O
this	O
model	O
in	O
detail	O
we	O
likely	O
will	O
find	O
some	O
problems	O
.	O
so	O
here	O
i	O
m	O
going	O
to	O
show	O
all	O
the	O
scores	O
for	O
these	O
five	O
documents	O
.	O
and	O
you	O
can	O
even	O
verify	O
they	O
are	O
correct	O
.	O
because	O
we	O
re	O
basically	O
counting	O
the	O
number	O
of	O
unique	O
query	B
terms	I
matched	O
in	O
each	O
document	O
.	O
now	O
note	O
that	O
this	O
method	O
actually	O
makes	O
sense	O
.	O
right	O
it	O
basically	O
means	O
if	O
a	O
document	O
matches	O
more	O
unique	O
query	B
terms	I
then	O
the	O
document	O
will	O
be	O
assuming	O
to	O
be	O
more	O
relevant	O
.	O
and	O
that	O
seems	O
to	O
make	O
sense	O
.	O
the	O
only	O
problem	O
is	O
here	O
we	O
can	O
note	O
set	O
there	O
are	O
three	O
documents	O
d	O
d	O
and	O
d	O
	O
.	O
and	O
they	O
tied	O
with	O
a	O
three	O
as	O
a	O
score	O
.	O
so	O
that	O
s	O
a	O
problem	O
because	O
if	O
you	O
look	O
at	O
them	O
carefully	O
it	O
seems	O
that	O
d	O
should	O
be	O
right	O
above	O
d	O
	O
.	O
because	O
d	O
only	O
mentioned	O
the	O
presidential	O
once	O
.	O
but	O
d	O
mentioned	O
it	O
much	O
more	O
times	O
.	O
in	O
case	O
of	O
d	O
presidential	O
could	O
be	O
extended	O
mentioned	O
.	O
but	O
d	O
is	O
clearly	O
above	O
presidential	O
campaign	O
.	O
another	O
problem	O
is	O
that	O
d	O
and	O
d	O
also	O
have	O
the	O
same	O
soul	O
.	O
but	O
if	O
you	O
look	O
at	O
the	O
the	O
three	O
words	O
that	O
are	O
matched	O
.	O
in	O
the	O
case	O
of	O
d	O
it	O
matched	O
the	O
news	O
about	O
and	O
the	O
campaign	O
.	O
but	O
in	O
the	O
case	O
of	O
d	O
it	O
match	O
the	O
news	O
presidential	O
and	O
campaign	O
.	O
so	O
intuitively	O
d	O
is	O
better	O
.	O
because	O
matching	O
presidential	O
is	O
more	O
important	O
though	O
than	O
matching	O
about	O
.	O
even	O
though	O
about	O
and	O
the	O
presidential	O
are	O
both	O
in	O
the	O
query	O
.	O
so	O
intuitively	O
we	O
would	O
like	O
d	O
to	O
be	O
ranked	O
above	O
d	O
	O
.	O
but	O
this	O
model	O
doesn	O
t	O
do	O
that	O
.	O
so	O
that	O
means	O
this	O
is	O
still	O
not	O
good	O
enough	O
we	O
have	O
to	O
solve	O
these	O
problems	O
.	O
to	O
summarize	O
in	O
this	O
lecture	O
we	O
talked	O
about	O
how	O
to	O
instantiate	O
a	O
vector	B
space	I
model	I
.	O
we	O
may	O
need	O
to	O
do	O
three	O
things	O
.	O
one	O
is	O
to	O
define	O
the	O
dimension	O
.	O
the	O
second	O
is	O
to	O
decide	O
how	O
to	O
place	O
documents	O
as	O
vectors	O
in	O
the	O
vector	B
space	I
.	O
and	O
to	O
also	O
place	O
a	O
query	O
in	O
the	O
vector	B
space	I
as	O
a	O
vector	O
.	O
and	O
third	O
is	O
to	O
define	O
the	O
similarity	O
between	O
two	O
vectors	O
particularly	O
the	O
query	O
vector	O
and	O
the	O
document	O
vector	O
.	O
we	O
also	O
talked	O
about	O
a	O
very	O
simple	O
way	O
to	O
instantiate	O
the	O
vector	B
space	I
model	I
.	O
indeed	O
that	O
s	O
probably	O
the	O
simplest	O
vector	B
space	I
model	I
that	O
we	O
can	O
derive	O
.	O
in	O
this	O
case	O
we	O
use	O
each	O
word	O
to	O
define	O
a	O
dimension	O
.	O
we	O
use	O
a	O
zero	O
one	O
bit	B
vector	I
to	O
represent	O
a	O
document	O
or	O
a	O
query	O
.	O
in	O
this	O
case	O
we	O
basically	O
only	O
care	O
about	O
word	O
presence	O
or	O
absence	O
.	O
we	O
ignore	O
the	O
frequency	O
.	O
and	O
we	O
use	O
the	O
dot	B
product	I
as	O
the	O
similarity	O
function	O
.	O
and	O
with	O
such	O
a	O
a	O
in	O
situation	O
.	O
and	O
we	O
showed	O
that	O
the	O
scoring	O
function	O
is	O
basically	O
to	O
score	O
a	O
document	O
based	O
on	O
the	O
number	O
of	O
distinct	O
query	B
words	I
matched	O
in	O
the	O
document	O
.	O
we	O
also	O
show	O
that	O
such	O
a	O
single	O
vector	B
space	I
model	I
still	O
doesn	O
t	O
work	O
well	O
and	O
we	O
need	O
to	O
improve	O
it	O
.	O
and	O
this	O
is	O
the	O
topic	O
that	O
we	O
re	O
going	O
to	O
cover	O
in	O
the	O
next	O
lecture	O
.	O
	O
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
how	O
to	O
improve	O
the	O
instant	O
changing	O
of	O
the	O
vector	B
space	I
model	I
.	O
this	O
is	O
the	O
continued	O
discussion	O
of	O
the	O
vector	B
space	I
model	I
.	O
we	O
re	O
going	O
to	O
focus	O
on	O
how	O
to	O
improve	O
the	O
instant	O
changing	O
of	O
this	O
model	O
.	O
in	O
a	O
previous	O
lecture	O
you	O
have	O
seen	O
that	O
with	O
simple	O
situations	O
of	O
the	O
vector	B
space	I
model	I
we	O
can	O
come	O
up	O
with	O
a	O
simple	O
scoring	O
function	O
that	O
would	O
give	O
us	O
basically	O
a	O
count	O
of	O
how	O
many	O
unique	O
query	B
terms	I
are	O
matching	O
the	O
document	O
.	O
we	O
also	O
have	O
seen	O
that	O
this	O
function	O
has	O
a	O
problem	O
as	O
shown	O
on	O
this	O
slide	O
.	O
in	O
particular	O
if	O
you	O
look	O
at	O
these	O
three	O
documents	O
they	O
will	O
all	O
get	O
the	O
same	O
score	O
because	O
they	O
match	O
the	O
three	O
unique	O
query	B
words	I
.	O
but	O
intuitively	O
we	O
would	O
like	O
d	O
to	O
be	O
ranked	O
above	O
d	O
	O
.	O
and	O
d	O
is	O
really	O
non	O
relevant	O
.	O
so	O
the	O
problem	O
here	O
is	O
that	O
this	O
function	O
couldn	O
t	O
capture	O
the	O
following	O
characteristics	O
.	O
first	O
we	O
would	O
like	O
to	O
give	O
more	O
gratitude	O
to	O
d	O
because	O
it	O
matches	O
the	O
presidential	O
more	O
times	O
than	O
d	O
	O
.	O
second	O
intuitively	O
matching	O
presidential	O
should	O
be	O
more	O
important	O
than	O
matching	O
about	O
because	O
about	O
is	O
a	O
very	O
common	O
word	O
that	O
occurs	O
everywhere	O
.	O
it	O
doesn	O
t	O
really	O
carry	O
that	O
much	O
content	O
.	O
so	O
in	O
this	O
lecture	O
let	O
s	O
see	O
how	O
we	O
can	O
improve	O
the	O
model	O
to	O
solve	O
these	O
two	O
problems	O
.	O
it	O
s	O
worth	O
thinking	O
at	O
this	O
point	O
about	O
why	O
do	O
we	O
have	O
these	O
four	O
problems	O
.	O
if	O
we	O
look	O
back	O
at	O
the	O
assumptions	O
we	O
have	O
made	O
while	O
substantiating	O
the	O
vector	B
space	I
model	I
we	O
will	O
realize	O
that	O
the	O
problem	O
is	O
really	O
coming	O
from	O
some	O
of	O
the	O
assumptions	O
.	O
in	O
particular	O
it	O
has	O
to	O
do	O
with	O
how	O
we	O
place	O
the	O
vectors	O
in	O
the	O
vector	B
space	I
.	O
so	O
then	O
naturally	O
in	O
order	O
to	O
fix	O
these	O
problems	O
we	O
have	O
to	O
revisit	O
those	O
assumptions	O
.	O
perhaps	O
you	O
will	O
have	O
to	O
use	O
different	O
ways	O
to	O
instantiate	O
the	O
vector	B
space	I
model	I
.	O
in	O
particular	O
we	O
have	O
to	O
place	O
the	O
vectors	O
in	O
a	O
different	O
way	O
.	O
so	O
let	O
s	O
see	O
how	O
can	O
we	O
prove	O
this	O
well	O
our	O
natural	O
thought	O
is	O
in	O
order	O
to	O
consider	O
multiple	O
times	O
of	O
a	O
term	O
in	O
a	O
document	O
.	O
we	O
should	O
consider	O
the	O
term	B
frequency	I
instead	O
of	O
just	O
the	O
absence	O
or	O
presence	O
.	O
in	O
order	O
to	O
consider	O
the	O
difference	O
between	O
a	O
document	O
where	O
a	O
query	B
term	I
occurred	O
multiple	O
times	O
and	O
the	O
one	O
where	O
the	O
query	B
term	I
occurred	O
just	O
once	O
.	O
we	O
have	O
to	O
concede	O
a	O
term	B
frequency	I
the	O
count	O
of	O
a	O
term	O
being	O
in	O
the	O
document	O
.	O
in	O
the	O
simplest	O
model	O
we	O
only	O
model	O
the	O
presence	O
and	O
absence	O
of	O
a	O
term	O
.	O
we	O
ignore	O
the	O
actual	O
number	O
of	O
times	O
that	O
a	O
term	O
occurs	O
in	O
a	O
document	O
.	O
so	O
let	O
s	O
add	O
this	O
back	O
.	O
so	O
we	O
re	O
going	O
to	O
do	O
then	O
represent	O
a	O
document	O
by	O
a	O
vector	O
with	O
term	B
frequency	I
as	O
element	O
.	O
so	O
that	O
is	O
to	O
say	O
now	O
the	O
elements	O
of	O
both	O
the	O
query	O
vector	O
and	O
the	O
document	O
vector	O
will	O
not	O
be	O
zero	O
once	O
but	O
instead	O
there	O
will	O
be	O
the	O
counts	O
of	O
a	O
word	O
in	O
the	O
query	O
or	O
the	O
document	O
.	O
so	O
this	O
would	O
bring	O
additional	O
information	O
about	O
the	O
document	O
.	O
so	O
this	O
can	O
be	O
seen	O
as	O
a	O
more	O
accurate	O
representation	O
of	O
our	O
documents	O
.	O
so	O
now	O
let	O
s	O
see	O
what	O
the	O
formula	O
would	O
look	O
like	O
if	O
we	O
change	O
this	O
representation	O
.	O
so	O
as	O
you	O
see	O
on	O
this	O
slide	O
we	O
still	O
use	O
that	O
product	O
and	O
so	O
the	O
formula	O
looks	O
very	O
similar	O
in	O
the	O
form	O
.	O
in	O
fact	O
it	O
looks	O
identical	O
but	O
inside	O
of	O
the	O
sum	O
of	O
cos	O
xi	O
and	O
yi	O
are	O
now	O
different	O
.	O
they	O
re	O
now	O
the	O
counts	O
of	O
words	O
i	O
in	O
the	O
query	O
and	O
the	O
document	O
.	O
now	O
at	O
this	O
point	O
i	O
also	O
suggest	O
you	O
to	O
pause	O
the	O
lecture	O
for	O
moment	O
and	O
just	O
we	O
ll	O
think	O
about	O
how	O
we	O
have	O
interpret	O
the	O
score	O
of	O
this	O
new	O
function	O
.	O
it	O
s	O
doing	O
something	O
very	O
similar	O
to	O
what	O
the	O
simplest	O
vsm	B
is	O
doing	O
.	O
but	O
because	O
of	O
the	O
change	O
of	O
the	O
vector	O
now	O
the	O
new	O
score	O
has	O
a	O
different	O
interpretation	O
.	O
can	O
you	O
see	O
the	O
difference	O
and	O
it	O
has	O
to	O
do	O
with	O
the	O
consideration	O
of	O
multiple	O
occurrences	O
of	O
the	O
same	O
time	O
in	O
the	O
document	O
.	O
more	O
importantly	O
we	O
ll	O
try	O
to	O
know	O
whether	O
this	O
would	O
fix	O
the	O
problem	O
of	O
the	O
simplest	O
vector	B
space	I
model	I
.	O
so	O
let	O
s	O
look	O
at	O
the	O
this	O
example	O
again	O
.	O
so	O
suppose	O
we	O
change	O
the	O
vector	O
to	O
term	B
frequency	I
vectors	O
.	O
now	O
let	O
s	O
look	O
at	O
these	O
three	O
documents	O
again	O
.	O
the	O
query	O
vector	O
is	O
the	O
same	O
because	O
all	O
these	O
words	O
occurred	O
exactly	O
once	O
in	O
the	O
query	O
.	O
so	O
the	O
vector	O
is	O
still	O
vector	O
.	O
and	O
in	O
fact	O
d	O
is	O
also	O
essential	O
in	O
representing	O
the	O
same	O
way	O
because	O
none	O
of	O
these	O
words	O
has	O
been	O
repeated	O
many	O
times	O
.	O
as	O
a	O
result	O
the	O
score	O
is	O
also	O
the	O
same	O
still	O
three	O
.	O
the	O
same	O
issue	O
for	O
d	O
and	O
we	O
still	O
have	O
a	O
	O
.	O
but	O
d	O
would	O
be	O
different	O
because	O
now	O
presidential	O
occurred	O
twice	O
here	O
.	O
so	O
the	O
end	O
in	O
the	O
four	O
presidential	O
in	O
the	O
would	O
be	O
instead	O
of	O
	O
.	O
as	O
a	O
result	O
now	O
the	O
score	O
for	O
d	O
is	O
higher	O
.	O
it	O
s	O
a	O
four	O
now	O
.	O
so	O
this	O
means	O
by	O
using	O
term	B
frequency	I
we	O
can	O
now	O
rank	O
d	O
above	O
d	O
and	O
d	O
as	O
we	O
hope	O
to	O
.	O
so	O
this	O
solve	O
the	O
problem	O
with	O
default	O
.	O
but	O
we	O
can	O
also	O
see	O
that	O
d	O
and	O
d	O
are	O
still	O
featured	O
in	O
the	O
same	O
way	O
.	O
they	O
still	O
have	O
identical	O
scores	O
so	O
it	O
did	O
not	O
fix	O
the	O
problem	O
here	O
.	O
so	O
how	O
can	O
we	O
fix	O
this	O
problem	O
we	O
would	O
like	O
to	O
give	O
more	O
credit	O
for	O
matching	O
presidential	O
than	O
matching	O
about	O
.	O
but	O
how	O
can	O
we	O
solve	O
the	O
problem	O
in	O
a	O
general	O
way	O
is	O
there	O
any	O
way	O
to	O
determine	O
which	O
word	O
should	O
be	O
treated	O
more	O
importantly	O
and	O
which	O
word	O
can	O
be	O
basically	O
ignored	O
.	O
about	O
is	O
such	O
a	O
word	O
.	O
and	O
which	O
it	O
does	O
not	O
really	O
carry	O
that	O
much	O
content	O
we	O
can	O
essentially	O
ignore	O
that	O
.	O
we	O
sometimes	O
call	O
such	O
a	O
word	O
a	O
stock	O
word	O
.	O
those	O
are	O
generally	O
very	O
frequent	O
and	O
they	O
occur	O
everywhere	O
matching	O
it	O
doesn	O
t	O
really	O
mean	O
anything	O
.	O
but	O
computation	O
how	O
can	O
we	O
capture	O
that	O
so	O
again	O
i	O
encourage	O
you	O
to	O
think	O
a	O
little	O
bit	O
about	O
this	O
.	O
can	O
you	O
come	O
up	O
with	O
any	O
statistical	O
approaches	O
to	O
somehow	O
distinguish	O
presidential	O
from	O
about	O
.	O
if	O
you	O
think	O
about	O
it	O
for	O
a	O
moment	O
you	O
realize	O
that	O
one	O
difference	O
is	O
that	O
a	O
word	O
like	O
above	O
occurs	O
everywhere	O
.	O
so	O
if	O
you	O
count	O
the	O
currents	O
of	O
the	O
water	O
in	O
the	O
whole	O
collection	O
that	O
we	O
would	O
see	O
that	O
about	O
as	O
much	O
higher	O
for	O
this	O
than	O
presidential	O
which	O
it	O
tends	O
to	O
occur	O
only	O
in	O
some	O
documents	O
.	O
so	O
this	O
idea	O
suggests	O
that	O
we	O
could	O
somehow	O
use	O
the	O
global	O
statistics	O
of	O
terms	O
or	O
some	O
other	O
formation	O
to	O
try	O
to	O
down	O
weight	O
the	O
element	O
for	O
about	O
in	O
the	O
vector	O
representation	O
of	O
d	O
	O
.	O
at	O
the	O
same	O
time	O
we	O
hope	O
to	O
somehow	O
increase	O
the	O
weight	O
of	O
presidential	O
in	O
the	O
vector	O
of	O
d	O
	O
.	O
if	O
we	O
can	O
do	O
that	O
then	O
we	O
can	O
expect	O
that	O
d	O
will	O
get	O
the	O
overall	O
score	O
to	O
be	O
less	O
than	O
three	O
while	O
d	O
will	O
get	O
the	O
score	O
about	O
three	O
.	O
then	O
we	O
ll	O
be	O
able	O
to	O
rank	O
d	O
on	O
top	O
of	O
d	O
	O
.	O
so	O
how	O
can	O
we	O
do	O
this	O
systematically	O
again	O
we	O
can	O
rely	O
on	O
some	O
steps	O
that	O
people	O
count	O
.	O
and	O
in	O
this	O
case	O
the	O
particular	O
idea	O
is	O
called	O
the	O
inverse	B
document	B
frequency	I
.	O
we	O
have	O
seen	O
document	B
frequency	I
.	O
as	O
one	O
signal	O
used	O
in	O
the	O
moding	O
retrieval	O
functions	O
.	O
we	O
discussed	O
this	O
in	O
a	O
previous	O
lecture	O
.	O
so	O
here	O
s	O
the	O
specific	O
way	O
of	O
using	O
it	O
.	O
document	B
frequency	I
is	O
the	O
count	O
of	O
documents	O
that	O
contain	O
a	O
particular	O
term	O
.	O
here	O
we	O
say	O
inverse	B
document	B
frequency	I
because	O
we	O
actually	O
want	O
to	O
reword	O
a	O
word	O
that	O
doesn	O
t	O
occur	O
in	O
many	O
documents	O
.	O
and	O
so	O
the	O
way	O
to	O
incorporate	O
this	O
into	O
our	O
vector	O
is	O
then	O
to	O
modify	O
the	O
frequency	O
count	O
by	O
multiplying	O
it	O
by	O
the	O
idea	O
of	O
the	O
corresponding	O
word	O
as	O
shown	O
here	O
.	O
if	O
we	O
didn	O
t	O
do	O
that	O
then	O
we	O
can	O
penalize	O
common	O
words	O
which	O
generally	O
have	O
a	O
low	O
idea	O
of	O
and	O
reward	O
real	O
words	O
which	O
we	O
re	O
have	O
a	O
higher	O
idf	B
.	O
so	O
most	O
specific	O
idf	B
can	O
be	O
defined	O
as	O
the	O
logarithm	O
of	O
m	O
plus	O
one	O
divided	O
by	O
k	O
where	O
m	O
is	O
the	O
total	O
number	O
of	O
documents	O
in	O
the	O
collection	O
k	O
is	O
df	B
or	O
document	B
frequency	I
.	O
the	O
total	O
number	O
of	O
documents	O
containing	O
the	O
word	O
w	O
.	O
now	O
if	O
you	O
plot	O
this	O
function	O
by	O
varying	O
k	O
then	O
you	O
will	O
see	O
the	O
curve	O
that	O
look	O
like	O
this	O
.	O
in	O
general	O
you	O
can	O
see	O
it	O
would	O
give	O
a	O
higher	O
value	O
for	O
a	O
low	O
df	B
word	O
a	O
rare	O
word	O
.	O
you	O
can	O
also	O
see	O
the	O
maximum	O
value	O
of	O
this	O
function	O
is	O
log	O
of	O
m	O
plus	O
	O
.	O
will	O
be	O
interesting	O
for	O
you	O
to	O
think	O
about	O
what	O
s	O
minimum	O
value	O
for	O
this	O
function	O
this	O
could	O
be	O
interesting	O
exercise	O
.	O
now	O
the	O
specific	O
function	O
may	O
not	O
be	O
as	O
important	O
as	O
the	O
heuristic	O
to	O
simply	O
penalize	O
popular	O
terms	O
.	O
but	O
it	O
turns	O
out	O
this	O
particular	O
function	O
form	O
has	O
also	O
worked	O
very	O
well	O
.	O
now	O
whether	O
there	O
is	O
a	O
better	O
form	O
of	O
function	O
here	O
is	O
the	O
open	O
research	O
question	O
.	O
but	O
it	O
s	O
also	O
clear	O
that	O
if	O
we	O
use	O
a	O
linear	O
kernalization	O
like	O
what	O
s	O
shown	O
here	O
with	O
this	O
line	O
then	O
it	O
may	O
not	O
be	O
as	O
reasonable	O
as	O
the	O
standard	O
idf	B
.	O
in	O
particular	O
you	O
can	O
see	O
the	O
difference	O
in	O
the	O
standard	O
idf	B
and	O
we	O
somehow	O
have	O
a	O
point	O
here	O
.	O
after	O
this	O
point	O
we	O
re	O
going	O
to	O
say	O
these	O
terms	O
are	O
essentially	O
not	O
very	O
useful	O
.	O
they	O
can	O
be	O
essentially	O
ignored	O
.	O
and	O
this	O
makes	O
sense	O
when	O
the	O
term	O
occurs	O
so	O
frequently	O
and	O
let	O
s	O
say	O
a	O
term	O
occurs	O
in	O
more	O
than	O
of	O
the	O
documents	O
.	O
then	O
the	O
term	O
is	O
unlikely	O
very	O
important	O
and	O
it	O
s	O
it	O
s	O
basically	O
a	O
common	O
term	O
.	O
it	O
s	O
not	O
very	O
important	O
to	O
match	O
this	O
word	O
so	O
with	O
the	O
standard	O
idf	B
you	O
can	O
see	O
it	O
s	O
basically	O
assumed	O
that	O
they	O
all	O
have	O
lower	O
weights	O
there	O
s	O
no	O
difference	O
.	O
but	O
if	O
you	O
look	O
at	O
the	O
linear	O
kernelization	O
at	O
this	O
point	O
there	O
is	O
there	O
s	O
some	O
difference	O
.	O
so	O
intuitively	O
we	O
want	O
to	O
focus	O
more	O
on	O
the	O
discrimination	O
of	O
low	O
df	B
words	O
rather	O
than	O
these	O
common	O
words	O
.	O
well	O
of	O
course	O
which	O
one	O
works	O
better	O
still	O
has	O
to	O
be	O
validated	O
by	O
using	O
the	O
empirically	O
related	O
data	O
set	O
.	O
and	O
we	O
have	O
to	O
use	O
users	O
to	O
judge	O
which	O
results	O
of	O
that	O
.	O
so	O
now	O
let	O
s	O
see	O
how	O
this	O
can	O
solve	O
problem	O
two	O
.	O
so	O
now	O
let	O
s	O
look	O
at	O
the	O
two	O
documents	O
again	O
.	O
now	O
without	O
idf	B
weighting	O
before	O
we	O
just	O
have	O
vectors	O
but	O
with	O
idf	B
weighting	O
we	O
now	O
can	O
adjust	O
the	O
df	B
weight	O
by	O
multiplying	O
the	O
with	O
the	O
idf	B
value	O
.	O
for	O
example	O
here	O
you	O
can	O
see	O
is	O
the	O
adjustment	O
in	O
particular	O
for	O
about	O
there	O
is	O
an	O
adjustment	O
by	O
using	O
the	O
idf	B
value	O
of	O
about	O
which	O
is	O
smaller	O
than	O
the	O
idf	B
value	O
of	O
presidential	O
.	O
so	O
if	O
you	O
look	O
at	O
these	O
the	O
idf	B
will	O
distinguish	O
these	O
two	O
words	O
.	O
as	O
a	O
result	O
adjustment	O
here	O
would	O
be	O
larger	O
would	O
make	O
this	O
weight	O
larger	O
.	O
so	O
if	O
we	O
score	O
with	O
these	O
new	O
vectors	O
and	O
what	O
would	O
happen	O
is	O
that	O
the	O
of	O
course	O
they	O
share	O
the	O
same	O
weights	O
for	O
news	O
and	O
the	O
campaign	O
but	O
the	O
margin	O
of	O
about	O
and	O
presidential	O
with	O
this	O
grouping	O
may	O
.	O
so	O
now	O
as	O
a	O
result	O
of	O
idf	B
weighting	O
we	O
will	O
have	O
d	O
to	O
be	O
ranked	O
above	O
d	O
	O
.	O
because	O
it	O
matched	O
rail	O
word	O
where	O
as	O
d	O
matched	O
common	O
word	O
.	O
so	O
this	O
shows	O
that	O
the	O
idea	O
of	O
weighting	O
can	O
solve	O
problem	O
two	O
.	O
so	O
how	O
effective	O
is	O
this	O
model	O
in	O
general	O
when	O
we	O
use	O
tf-idf	B
weighting	O
well	O
let	O
s	O
look	O
at	O
all	O
these	O
documents	O
that	O
we	O
have	O
seen	O
before	O
.	O
these	O
are	O
the	O
new	O
scores	O
of	O
the	O
new	O
documents	O
.	O
but	O
how	O
effective	O
is	O
this	O
new	O
weighting	O
method	O
and	O
new	O
scoring	O
function	O
all	O
right	O
so	O
now	O
let	O
s	O
see	O
overall	O
how	O
effective	O
is	O
this	O
new	O
ranking	B
function	I
with	O
tf-idf	B
weighting	O
here	O
we	O
show	O
all	O
the	O
five	O
documents	O
that	O
we	O
have	O
seen	O
before	O
and	O
these	O
are	O
their	O
scores	O
.	O
now	O
we	O
can	O
see	O
the	O
scores	O
for	O
the	O
first	O
four	O
documents	O
here	O
seem	O
to	O
be	O
quite	O
reasonable	O
.	O
they	O
are	O
as	O
we	O
expected	O
.	O
however	O
we	O
also	O
see	O
a	O
new	O
problem	O
.	O
because	O
now	O
d	O
here	O
which	O
did	O
not	O
have	O
a	O
very	O
high	O
score	O
with	O
our	O
simplest	O
vector	B
space	I
model	I
.	O
now	O
after	O
it	O
has	O
a	O
very	O
high	O
score	O
.	O
in	O
fact	O
it	O
has	O
the	O
highest	O
score	O
here	O
.	O
so	O
this	O
creates	O
a	O
new	O
problem	O
.	O
this	O
actually	O
a	O
common	O
phenomenon	O
in	O
designing	O
material	O
functions	O
.	O
basically	O
when	O
you	O
try	O
to	O
fix	O
one	O
problem	O
you	O
tend	O
to	O
introduce	O
other	O
problems	O
.	O
and	O
that	O
s	O
why	O
it	O
s	O
very	O
tricky	O
how	O
to	O
design	O
effective	O
ranking	B
function	I
.	O
and	O
what	O
s	O
what	O
s	O
the	O
best	O
ranking	B
function	I
is	O
the	O
open	O
research	O
question	O
.	O
researchers	O
are	O
still	O
working	O
on	O
that	O
.	O
but	O
in	O
the	O
next	O
few	O
lecture	O
we	O
re	O
going	O
to	O
also	O
talk	O
about	O
some	O
additional	O
ideas	O
to	O
further	O
improve	O
this	O
model	O
and	O
try	O
to	O
fix	O
this	O
problem	O
.	O
so	O
to	O
summarize	O
this	O
lecture	O
we	O
ve	O
talked	O
about	O
how	O
to	O
improve	O
this	O
vector	B
space	I
model	I
.	O
and	O
we	O
ve	O
got	O
to	O
improve	O
the	O
of	O
the	O
vector	B
space	I
model	I
based	O
on	O
tf-idf	B
weighting	O
.	O
so	O
the	O
improvement	O
most	O
of	O
it	O
is	O
on	O
the	O
placement	O
of	O
the	O
vector	O
.	O
where	O
we	O
give	O
higher	O
weight	O
to	O
a	O
term	O
that	O
occurred	O
many	O
times	O
in	O
the	O
document	O
but	O
infrequently	O
in	O
the	O
whole	O
collection	O
.	O
and	O
we	O
have	O
seen	O
that	O
this	O
improved	O
model	O
indeed	O
works	O
better	O
than	O
the	O
simplest	O
vector	B
space	I
model	I
but	O
it	O
also	O
still	O
has	O
some	O
problems	O
.	O
in	O
the	O
next	O
lecture	O
we	O
re	O
going	O
to	O
look	O
at	O
the	O
how	O
to	O
address	O
these	O
additional	O
problems	O
.	O
in	O
this	O
lecture	O
we	O
continue	O
the	O
discussion	O
of	O
vector	B
space	I
model	I
.	O
in	O
particular	O
we	O
are	O
going	O
to	O
talk	O
about	O
the	O
tf	B
transformation	O
.	O
in	O
the	O
previous	O
lecture	O
we	O
have	O
derived	O
a	O
tf-idf	B
weighting	O
formula	O
using	O
the	O
vector	B
space	I
model	I
.	O
and	O
we	O
have	O
shown	O
that	O
this	O
model	O
actually	O
works	O
pretty	O
well	O
for	O
these	O
examples	O
as	O
shown	O
on	O
this	O
slide	O
except	O
for	O
d	O
which	O
has	O
received	O
a	O
very	O
high	O
score	O
.	O
indeed	O
it	O
has	O
received	O
the	O
highest	O
score	O
among	O
all	O
these	O
documents	O
.	O
but	O
this	O
document	O
is	O
intuitively	O
non-relevant	O
so	O
this	O
is	O
not	O
desirable	O
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
how	O
would	O
you	O
use	O
tf	B
transformation	O
to	O
solve	O
this	O
problem	O
.	O
before	O
we	O
discuss	O
the	O
details	O
let	O
s	O
take	O
a	O
look	O
at	O
the	O
formula	O
for	O
this	O
symbol	O
here	O
for	O
idf	B
weighting	O
ranking	B
function	I
and	O
see	O
why	O
this	O
document	O
has	O
received	O
such	O
a	O
high	O
score	O
.	O
so	O
this	O
is	O
the	O
formula	O
and	O
if	O
you	O
look	O
at	O
the	O
formula	O
carefully	O
then	O
you	O
will	O
see	O
it	O
involves	O
a	O
sum	O
over	O
all	O
the	O
matched	O
query	B
terms	I
.	O
and	O
inside	O
the	O
sum	O
each	O
matched	O
query	O
sum	O
has	O
a	O
particular	O
weight	O
.	O
and	O
this	O
weight	O
is	O
tf-idf	B
weighting	O
.	O
so	O
it	O
has	O
an	O
idf	B
component	O
where	O
we	O
see	O
variables	O
.	O
one	O
is	O
the	O
total	O
number	O
of	O
documents	O
in	O
the	O
collection	O
and	O
that	O
is	O
m	O
.	O
the	O
other	O
is	O
the	O
documentive	O
frequency	O
.	O
this	O
is	O
the	O
number	O
of	O
documents	O
that	O
contain	O
this	O
word	O
w	O
.	O
the	O
other	O
variables	O
in	O
involving	O
the	O
formula	O
include	O
the	O
count	O
of	O
the	O
query	B
term	I
.	O
w	O
in	O
the	O
query	O
and	O
the	O
count	O
of	O
the	O
word	O
in	O
the	O
document	O
.	O
if	O
you	O
look	O
at	O
this	O
document	O
again	O
now	O
it	O
s	O
not	O
hard	O
to	O
realize	O
that	O
the	O
reason	O
why	O
it	O
has	O
received	O
a	O
high	O
score	O
is	O
because	O
it	O
has	O
a	O
very	O
high	O
count	O
of	O
campaign	O
.	O
so	O
the	O
count	O
of	O
campaign	O
in	O
this	O
document	O
is	O
a	O
four	O
which	O
is	O
much	O
higher	O
than	O
the	O
other	O
documents	O
and	O
has	O
contributed	O
to	O
the	O
high	O
score	O
of	O
this	O
document	O
.	O
so	O
intriguingly	O
in	O
order	O
to	O
lower	O
the	O
score	O
for	O
this	O
document	O
we	O
need	O
to	O
somehow	O
restrict	O
the	O
contribution	O
of	O
the	O
matching	O
of	O
this	O
term	O
in	O
the	O
document	O
.	O
and	O
if	O
you	O
think	O
about	O
the	O
matching	O
of	O
terms	O
in	O
the	O
document	O
carefully	O
you	O
actually	O
would	O
realize	O
we	O
probably	O
shouldn	O
t	O
reward	O
multiple	O
occurrences	O
so	O
generously	O
.	O
and	O
by	O
that	O
i	O
mean	O
the	O
first	O
occurrence	O
of	O
a	O
term	O
says	O
a	O
lot	O
about	O
the	O
the	O
matching	O
of	O
this	O
term	O
because	O
it	O
goes	O
from	O
zero	O
count	O
to	O
a	O
count	O
of	O
one	O
and	O
that	O
increase	O
means	O
a	O
lot	O
.	O
once	O
we	O
see	O
a	O
word	O
in	O
the	O
document	O
it	O
s	O
very	O
likely	O
that	O
the	O
document	O
is	O
talking	O
about	O
this	O
word	O
.	O
if	O
we	O
see	O
an	O
extra	O
occurrence	O
on	O
top	O
of	O
the	O
first	O
occurrence	O
that	O
is	O
to	O
go	O
from	O
one	O
to	O
two	O
then	O
we	O
also	O
can	O
say	O
that	O
well	O
the	O
second	O
occurrence	O
kind	O
of	O
confirmed	O
that	O
it	O
s	O
not	O
a	O
accidental	O
mention	O
of	O
the	O
word	O
.	O
now	O
we	O
are	O
more	O
sure	O
that	O
this	O
document	O
is	O
talking	O
about	O
this	O
word	O
.	O
but	O
imagine	O
we	O
have	O
seen	O
let	O
s	O
say	O
times	O
of	O
the	O
word	O
in	O
the	O
document	O
.	O
then	O
adding	O
one	O
extra	O
occurrence	O
is	O
not	O
going	O
to	O
test	O
more	O
about	O
evidence	O
because	O
we	O
are	O
already	O
sure	O
that	O
this	O
document	O
is	O
about	O
this	O
word	O
.	O
so	O
if	O
you	O
re	O
thinking	O
this	O
way	O
it	O
seems	O
that	O
we	O
should	O
restrict	O
the	O
contributing	O
of	O
a	O
high	O
account	O
of	O
term	O
.	O
and	O
that	O
is	O
the	O
idea	O
of	O
tf	B
transformation	O
.	O
so	O
this	O
transformation	O
function	O
is	O
going	O
to	O
turn	O
the	O
raw	O
count	O
of	O
word	O
into	O
a	O
term	B
frequency	I
weight	O
for	O
the	O
word	O
in	O
the	O
document	O
.	O
so	O
here	O
i	O
show	O
in	O
x-axis	O
that	O
raw	O
count	O
and	O
in	O
y-axis	O
i	O
show	O
the	O
term	B
frequency	I
weight	O
.	O
so	O
in	O
the	O
previous	O
ranking	B
functions	I
we	O
actually	O
have	O
increasingly	O
used	O
some	O
kind	O
of	O
transformation	O
.	O
so	O
for	O
example	O
in	O
the	O
zero-one	O
bit	B
vector	I
retentation	O
we	O
actually	O
use	O
the	O
suchier	O
transformation	O
function	O
as	O
shown	O
here	O
.	O
basically	O
if	O
the	O
count	O
is	O
zero	O
then	O
it	O
has	O
zero	O
weight	O
.	O
otherwise	O
it	O
would	O
have	O
a	O
weight	O
of	O
one	O
.	O
it	O
s	O
flat	O
.	O
now	O
what	O
about	O
using	O
term	O
count	O
as	O
a	O
tf	B
weight	O
.	O
well	O
that	O
s	O
a	O
linear	O
function	O
right	O
so	O
it	O
has	O
just	O
exactly	O
the	O
same	O
weight	O
as	O
the	O
count	O
.	O
now	O
we	O
have	O
just	O
seen	O
that	O
this	O
is	O
not	O
desirable	O
.	O
so	O
what	O
we	O
want	O
is	O
something	O
like	O
this	O
.	O
so	O
for	O
example	O
with	O
a	O
logarithm	O
function	O
we	O
can	O
have	O
a	O
sub-linear	O
transformation	O
that	O
looks	O
like	O
this	O
.	O
and	O
this	O
will	O
control	O
the	O
influence	O
of	O
really	O
high	O
weight	O
because	O
it	O
s	O
going	O
to	O
lower	O
its	O
inference	B
yet	O
it	O
will	O
retain	O
the	O
inference	B
of	O
small	O
count	O
.	O
or	O
we	O
might	O
want	O
to	O
even	O
bend	O
the	O
curve	O
more	O
by	O
applying	O
logarithm	O
twice	O
.	O
now	O
people	O
have	O
tried	O
all	O
these	O
methods	O
and	O
they	O
are	O
indeed	O
working	O
better	O
than	O
the	O
linear	O
form	O
of	O
the	O
transformation	O
but	O
so	O
far	O
what	O
works	O
the	O
best	O
seems	O
to	O
be	O
this	O
special	O
transformation	O
called	O
a	O
bm	O
transformation	O
.	O
bm	O
stands	O
for	O
best	O
matching	O
.	O
now	O
in	O
this	O
transformation	O
you	O
can	O
see	O
there	O
s	O
a	O
parameter	O
k	O
here	O
.	O
and	O
this	O
k	O
controls	O
the	O
upper	O
bound	O
of	O
this	O
function	O
.	O
it	O
s	O
easy	O
to	O
see	O
this	O
function	O
has	O
a	O
upper	O
bound	O
because	O
if	O
you	O
look	O
at	O
the	O
x	O
divided	O
by	O
x	O
plus	O
k	O
where	O
k	O
is	O
not	O
an	O
active	O
number	O
then	O
the	O
numerator	O
will	O
never	O
be	O
able	O
to	O
exceed	O
the	O
denominator	O
right	O
so	O
it	O
s	O
upper	O
bounded	O
by	O
k	O
plus	O
	O
.	O
now	O
this	O
is	O
also	O
difference	O
between	O
this	O
transformation	O
function	O
and	O
the	O
logarithm	O
transformation	O
.	O
which	O
it	O
doesn	O
t	O
have	O
upperbound	O
.	O
now	O
furthermore	O
one	O
interesting	O
property	O
of	O
this	O
function	O
is	O
that	O
as	O
we	O
vary	O
k	O
we	O
can	O
actually	O
simulate	O
different	O
transformation	O
functions	O
including	O
the	O
two	O
extremes	O
that	O
are	O
shown	O
here	O
.	O
that	O
is	O
a	O
zero	O
one	O
bit	O
transformation	O
and	O
the	O
unit	O
transformation	O
.	O
so	O
for	O
example	O
if	O
we	O
set	O
k	O
to	O
zero	O
now	O
you	O
can	O
see	O
the	O
function	O
value	O
would	O
be	O
one	O
.	O
so	O
we	O
precisely	O
recover	O
the	O
zero	O
one	O
bit	O
transformation	O
.	O
if	O
you	O
set	O
k	O
to	O
a	O
very	O
large	O
number	O
on	O
the	O
other	O
hand	O
other	O
hand	O
it	O
s	O
going	O
to	O
look	O
more	O
like	O
the	O
linear	O
transformation	O
function	O
.	O
so	O
in	O
this	O
sense	O
this	O
transformation	O
is	O
very	O
flexible	O
it	O
allows	O
us	O
to	O
control	O
the	O
shape	O
of	O
the	O
transformation	O
.	O
it	O
also	O
has	O
a	O
nice	O
property	O
of	O
the	O
upper	O
bound	O
.	O
and	O
this	O
upper	O
bound	O
is	O
useful	O
to	O
control	O
the	O
inference	B
of	O
a	O
particular	O
term	O
.	O
and	O
so	O
that	O
we	O
can	O
prevent	O
a	O
a	O
spammer	O
from	O
just	O
increasing	O
the	O
count	O
of	O
term	O
to	O
spam	O
all	O
queries	O
that	O
might	O
match	O
this	O
term	O
.	O
in	O
other	O
words	O
this	O
upper	O
bound	O
might	O
also	O
ensure	O
that	O
all	O
terms	O
will	O
be	O
counted	O
when	O
we	O
aggregate	O
the	O
the	O
weights	O
to	O
compute	O
a	O
score	O
.	O
as	O
i	O
said	O
this	O
transformation	O
function	O
has	O
worked	O
well	O
so	O
far	O
.	O
so	O
to	O
summarise	O
this	O
lecture	O
the	O
main	O
point	O
is	O
that	O
we	O
need	O
to	O
do	O
some	O
sub	O
linearity	O
of	O
tf	B
transformation	O
.	O
and	O
this	O
is	O
needed	O
to	O
capture	O
the	O
intuition	O
of	O
diminishing	O
return	O
from	O
high	O
term	O
counts	O
.	O
it	O
s	O
also	O
to	O
avoid	O
a	O
dominance	O
by	O
one	O
single	O
term	O
over	O
all	O
others	O
.	O
this	O
bm	O
transformation	O
transformation	O
that	O
we	O
talked	O
about	O
is	O
very	O
interesting	O
.	O
it	O
s	O
so	O
far	O
one	O
of	O
the	O
best	O
performing	O
tf	B
transforming	O
formation	O
formulas	O
.	O
it	O
has	O
upper	O
bound	O
and	O
it	O
s	O
also	O
robust	O
and	O
effective	O
.	O
now	O
if	O
we	O
re	O
plug	O
in	O
this	O
function	O
into	O
our	O
tf-idf	B
weighting	O
vector	B
space	I
model	I
then	O
we	O
would	O
end	O
up	O
having	O
the	O
following	O
ranking	B
function	I
which	O
has	O
a	O
bm	O
tf	B
component	O
.	O
now	O
this	O
is	O
already	O
very	O
close	O
to	O
a	O
state	O
of	O
the	O
art	O
ranking	B
function	I
called	O
a	O
bm	O
	O
.	O
and	O
we	O
will	O
discuss	O
how	O
we	O
can	O
further	O
improve	O
this	O
formula	O
in	O
the	O
next	O
lecture	O
.	O
this	O
lecture	O
is	O
about	O
document	O
length	B
normalization	I
in	O
the	O
vector	B
space	I
model	I
.	O
in	O
this	O
lecture	O
we	O
are	O
going	O
to	O
continue	O
the	O
discussion	O
of	O
the	O
vector	B
space	I
model	I
in	O
particular	O
we	O
are	O
going	O
to	O
discuss	O
.	O
the	O
issue	O
of	O
document	O
length	B
normalization	I
.	O
so	O
far	O
in	O
the	O
lectures	O
about	O
the	O
vector	B
space	I
model	I
we	O
have	O
used	O
the	O
various	O
signals	O
from	O
the	O
document	O
to	O
assess	O
the	O
matching	O
of	O
the	O
document	O
though	O
with	O
a	O
preorder	O
.	O
in	O
particular	O
we	O
have	O
considered	O
the	O
term	B
frequency	I
the	O
count	O
of	O
a	O
term	O
in	O
a	O
document	O
.	O
we	O
have	O
also	O
considered	O
a	O
it	O
s	O
global	O
statistics	O
such	O
as	O
idf	B
in	O
words	O
document	B
frequency	I
.	O
but	O
we	O
have	O
not	O
considered	O
a	O
document	O
length	O
.	O
so	O
here	O
i	O
show	O
two	O
example	O
documents	O
.	O
d	O
is	O
much	O
shorter	O
with	O
only	O
words	O
.	O
d	O
on	O
the	O
other	O
hand	O
has	O
words	O
.	O
if	O
you	O
look	O
at	O
the	O
matching	O
of	O
these	O
query	B
words	I
we	O
see	O
that	O
in	O
d	O
there	O
are	O
more	O
matchings	O
of	O
the	O
query	B
words	I
but	O
one	O
might	O
reason	O
that	O
d	O
may	O
have	O
matched	O
these	O
query	B
words	I
.	O
in	O
a	O
scattered	O
manner	O
.	O
so	O
maybe	O
the	O
topic	O
of	O
d	O
is	O
not	O
really	O
about	O
the	O
topic	O
of	O
the	O
query	O
.	O
so	O
the	O
discussion	O
of	O
a	O
campaign	O
at	O
the	O
beginning	O
of	O
the	O
document	O
may	O
have	O
nothing	O
to	O
do	O
with	O
the	O
mention	O
of	O
presidential	O
at	O
the	O
end	O
.	O
in	O
general	O
if	O
you	O
think	O
about	O
the	O
long	O
documents	O
they	O
would	O
have	O
a	O
higher	O
chance	O
to	O
match	O
any	O
query	O
.	O
in	O
fact	O
if	O
you	O
generate	O
a	O
a	O
long	O
document	O
that	O
randomly	O
sampling	O
sampling	O
words	O
from	O
the	O
distribution	O
of	O
words	O
then	O
eventually	O
you	O
probably	O
will	O
match	O
any	O
query	O
.	O
so	O
in	O
this	O
sense	O
we	O
should	O
penalize	O
no	O
documents	O
because	O
they	O
just	O
naturally	O
have	O
better	O
chances	O
to	O
match	O
any	O
query	O
.	O
and	O
this	O
is	O
our	O
idea	O
of	O
document	O
answer	O
.	O
we	O
also	O
need	O
to	O
be	O
careful	O
in	O
avoiding	O
to	O
overpenalize	O
small	O
documents	O
.	O
on	O
the	O
one	O
hand	O
we	O
want	O
to	O
penalize	O
a	O
long	O
document	O
.	O
but	O
on	O
the	O
other	O
hand	O
we	O
also	O
don	O
t	O
want	O
to	O
over-penalize	O
them	O
.	O
and	O
the	O
reason	O
is	O
because	O
a	O
document	O
that	O
may	O
be	O
long	O
because	O
of	O
different	O
reason	O
.	O
in	O
one	O
case	O
the	O
document	O
may	O
be	O
more	O
long	O
because	O
it	O
uses	O
more	O
words	O
.	O
so	O
for	O
example	O
think	O
about	O
the	O
article	O
of	O
a	O
research	O
paper	O
.	O
it	O
would	O
use	O
more	O
words	O
than	O
the	O
corresponding	O
abstract	O
.	O
so	O
this	O
is	O
the	O
case	O
where	O
we	O
probably	O
should	O
penalize	O
the	O
matching	O
of	O
a	O
long	O
document	O
such	O
as	O
full	O
paper	O
.	O
when	O
we	O
compare	O
the	O
matching	O
of	O
words	O
in	O
such	O
long	O
document	O
with	O
matching	O
of	O
the	O
words	O
in	O
the	O
short	O
abstract	O
.	O
then	O
long	O
papers	O
generally	O
have	O
a	O
higher	O
chance	O
of	O
matching	O
query	B
words	I
.	O
therefore	O
we	O
should	O
penalize	O
them	O
.	O
however	O
there	O
is	O
another	O
case	O
when	O
the	O
document	O
is	O
long	O
and	O
that	O
is	O
when	O
the	O
document	O
simply	O
has	O
more	O
content	O
.	O
now	O
consider	O
another	O
case	O
of	O
a	O
long	O
document	O
where	O
we	O
simply	O
concatenated	O
a	O
lot	O
of	O
abstracts	O
of	O
different	O
papers	O
.	O
in	O
such	O
a	O
case	O
obviously	O
we	O
don	O
t	O
want	O
to	O
penalize	O
such	O
a	O
long	O
document	O
.	O
indeed	O
we	O
probably	O
don	O
t	O
want	O
to	O
penalize	O
such	O
a	O
document	O
because	O
it	O
s	O
long	O
.	O
so	O
that	O
s	O
why	O
we	O
need	O
to	O
be	O
careful	O
.	O
about	O
using	O
the	O
right	O
degree	O
of	O
penalization	O
.	O
a	O
method	O
that	O
has	O
been	O
working	O
well	O
based	O
on	O
recent	O
research	O
is	O
called	O
pivot	O
length	B
normalization	I
.	O
and	O
in	O
this	O
case	O
the	O
idea	O
is	O
to	O
use	O
.	O
the	O
average	O
document	O
length	O
as	O
a	O
p	O
word	O
as	O
a	O
reference	O
point	O
.	O
that	O
means	O
we	O
will	O
assume	O
that	O
for	O
the	O
average	O
length	O
documents	O
the	O
score	O
is	O
about	O
right	O
.	O
so	O
the	O
normalizer	O
would	O
be	O
	O
.	O
but	O
if	O
a	O
document	O
is	O
longer	O
than	O
the	O
average	O
document	O
length	O
then	O
there	O
will	O
be	O
some	O
penalization	O
.	O
where	O
as	O
if	O
it	O
s	O
shorter	O
than	O
there	O
s	O
even	O
some	O
reward	O
.	O
so	O
this	O
is	O
an	O
illustrator	O
that	O
using	O
this	O
slide	O
.	O
on	O
the	O
axis	O
s	O
axis	O
you	O
can	O
see	O
the	O
length	O
of	O
document	O
.	O
on	O
the	O
y-axis	O
we	O
show	O
the	O
normalizer	O
in	O
the	O
case	O
pivoted	O
length	B
normalization	I
formula	O
for	O
the	O
normalizer	O
is	O
is	O
seem	O
to	O
be	O
interpolation	B
of	O
one	O
and	O
the	O
normalize	O
the	O
document	O
lengths	O
controlled	O
by	O
a	O
parameter	O
b	O
here	O
.	O
so	O
you	O
can	O
see	O
here	O
when	O
we	O
first	O
divide	O
the	O
lengths	O
of	O
the	O
document	O
by	O
the	O
average	O
document	O
length	O
.	O
this	O
not	O
only	O
gives	O
us	O
some	O
sense	O
about	O
the	O
how	O
this	O
document	O
is	O
compared	O
with	O
the	O
average	O
document	O
length	O
but	O
also	O
gives	O
us	O
a	O
benefit	O
of	O
not	O
worrying	O
about	O
the	O
unit	O
of	O
length	O
we	O
can	O
measure	O
the	O
length	O
by	O
words	O
or	O
by	O
characters	O
.	O
anyway	O
this	O
normalizer	O
has	O
an	O
interesting	O
property	O
.	O
first	O
we	O
see	O
that	O
if	O
we	O
set	O
the	O
parameter	O
b	O
to	O
then	O
the	O
value	O
would	O
be	O
so	O
there	O
s	O
no	O
pair	O
length	B
normalization	I
at	O
all	O
.	O
so	O
b	O
in	O
this	O
sense	O
controls	O
the	O
length	B
normalization	I
where	O
as	O
if	O
we	O
set	O
d	O
to	O
a	O
non-zero	O
value	O
then	O
the	O
normalizer	O
will	O
look	O
like	O
this	O
right	O
.	O
so	O
the	O
value	O
would	O
be	O
higher	O
for	O
documents	O
that	O
are	O
longer	O
than	O
the	O
average	O
document	O
length	O
.	O
where	O
as	O
the	O
value	O
of	O
the	O
normalizer	O
will	O
be	O
short-	O
will	O
be	O
smaller	O
for	O
shorter	O
documents	O
.	O
so	O
in	O
this	O
sense	O
we	O
see	O
there	O
s	O
a	O
penalization	O
for	O
long	O
documents	O
.	O
and	O
there	O
s	O
a	O
reward	O
for	O
short	O
documents	O
.	O
the	O
degree	O
of	O
penalization	O
is	O
conjured	O
by	O
b	O
.	O
because	O
if	O
we	O
set	O
b	O
to	O
a	O
larger	O
value	O
then	O
the	O
normalizer	O
.	O
what	O
looked	O
like	O
this	O
.	O
there	O
s	O
even	O
more	O
penalization	O
for	O
long	O
documents	O
and	O
more	O
reward	O
for	O
the	O
short	O
documents	O
.	O
by	O
adjusting	O
b	O
which	O
varies	O
from	O
zero	O
to	O
one	O
we	O
can	O
control	O
the	O
degree	O
of	O
length	B
normalization	I
.	O
so	O
if	O
we	O
re	O
plucking	O
this	O
length	B
normalization	I
factor	O
into	O
the	O
vector	B
space	I
model	I
ranking	B
functions	I
that	O
we	O
have	O
already	O
examined	O
.	O
then	O
we	O
will	O
end	O
up	O
heading	O
with	O
formulas	O
and	O
these	O
are	O
in	O
fact	O
the	O
state	O
of	O
the	O
are	O
vector	B
space	I
models	O
.	O
formulas	O
.	O
so	O
let	O
s	O
talk	O
an	O
that	O
let	O
s	O
take	O
a	O
look	O
at	O
the	O
each	O
of	O
them	O
.	O
the	O
first	O
one	O
s	O
called	O
a	O
pivoted	O
length	B
normalization	I
vector	B
space	I
model	I
.	O
and	O
a	O
reference	O
in	O
the	O
end	O
has	O
detail	O
about	O
the	O
derivation	O
of	O
this	O
model	O
.	O
and	O
here	O
we	O
see	O
that	O
it	O
s	O
basically	O
the	O
tfidf	B
weighting	O
model	O
that	O
we	O
have	O
discussed	O
.	O
the	O
idf	B
component	O
should	O
be	O
very	O
familiar	O
now	O
to	O
you	O
.	O
there	O
is	O
also	O
a	O
query	B
term	B
frequency	I
component	O
here	O
.	O
and	O
and	O
then	O
in	O
the	O
middle	O
there	O
is	O
.	O
and	O
normalize	O
the	O
tf	B
.	O
and	O
in	O
this	O
case	O
we	O
see	O
we	O
use	O
the	O
double	O
algorithm	O
as	O
we	O
discussed	O
before	O
and	O
this	O
is	O
to	O
achieve	O
a	O
sublinear	O
transformation	O
.	O
but	O
we	O
also	O
put	O
document	O
length	O
normalizer	O
in	O
the	O
bottom	O
all	O
right	O
so	O
this	O
would	O
cause	O
penalty	O
for	O
a	O
long	O
document	O
because	O
the	O
larger	O
the	O
denominator	O
is	O
the	O
denominator	O
is	O
then	O
the	O
smaller	O
the	O
shift	O
weight	O
is	O
.	O
and	O
this	O
is	O
of	O
course	O
controlled	O
by	O
the	O
parameter	O
b	O
here	O
.	O
and	O
you	O
can	O
see	O
again	O
b	O
is	O
set	O
to	O
and	O
there	O
there	O
is	O
no	O
length	B
normalization	I
.	O
okay	O
.	O
so	O
this	O
is	O
one	O
of	O
the	O
two	O
most	O
effective	O
.	O
not	O
this	O
base	O
model	O
of	O
formulas	O
.	O
the	O
next	O
one	O
called	O
a	O
bm	O
or	O
okapi	B
is	O
also	O
similar	O
.	O
in	O
that	O
it	O
also	O
has	O
a	O
i	O
df	B
component	O
here	O
and	O
a	O
query	O
df	B
component	O
here	O
.	O
but	O
in	O
the	O
middle	O
the	O
normalization	O
s	O
a	O
little	O
bit	O
different	O
.	O
as	O
we	O
expand	O
there	O
is	O
this	O
or	O
copied	O
here	O
for	O
transformation	O
here	O
.	O
and	O
that	O
does	O
sublinear	O
transformation	O
with	O
an	O
upper	O
bound	O
.	O
in	O
this	O
case	O
we	O
have	O
put	O
the	O
length	B
normalization	I
factor	O
here	O
.	O
we	O
are	O
adjusting	O
k	O
but	O
it	O
achieves	O
a	O
similar	O
factor	O
because	O
we	O
put	O
a	O
normalizer	O
in	O
the	O
denominator	O
.	O
therefore	O
again	O
if	O
a	O
document	O
is	O
longer	O
then	O
the	O
term	O
weight	O
will	O
be	O
smaller	O
.	O
so	O
you	O
can	O
see	O
after	O
we	O
have	O
gone	O
through	O
all	O
the	O
instances	O
that	O
we	O
talked	O
about	O
and	O
we	O
have	O
in	O
the	O
end	O
reached	O
the	O
basically	O
the	O
state	O
of	O
the	O
art	O
mutual	O
function	O
.	O
so	O
so	O
far	O
we	O
have	O
talked	O
about	O
mainly	O
how	O
to	O
place	O
the	O
document	O
matter	O
in	O
the	O
matter	O
space	O
.	O
and	O
this	O
has	O
played	O
an	O
important	O
role	O
in	O
uh	O
determining	O
the	O
factors	O
of	O
the	O
function	O
.	O
but	O
there	O
are	O
also	O
other	O
dimensions	O
where	O
we	O
did	O
not	O
really	O
examine	O
detail	O
.	O
for	O
example	O
can	O
we	O
further	O
improve	O
the	O
instantiation	O
of	O
the	O
dimension	O
of	O
the	O
vector	B
space	I
model	I
.	O
now	O
we	O
ve	O
just	O
assumed	O
that	O
the	O
back	O
of	O
words	O
.	O
so	O
each	O
dimension	O
is	O
a	O
word	O
.	O
but	O
obviously	O
we	O
can	O
see	O
there	O
are	O
many	O
other	O
choices	O
.	O
for	O
example	O
stemmed	O
words	O
those	O
are	O
the	O
words	O
that	O
have	O
been	O
transformed	O
into	O
the	O
same	O
rule	O
form	O
.	O
so	O
that	O
computation	O
and	O
computing	O
will	O
all	O
become	O
the	O
same	O
and	O
they	O
can	O
be	O
matched	O
.	O
we	O
need	O
to	O
stop	O
water	O
removal	O
.	O
this	O
is	O
removes	O
on	O
very	O
common	O
words	O
that	O
don	O
t	O
carry	O
any	O
content	O
.	O
like	O
the	O
or	O
of	O
we	O
use	O
the	O
phrases	O
to	O
define	O
that	O
	O
.	O
we	O
can	O
even	O
use	O
late	O
in	O
the	O
semantica	O
an	O
answer	O
sort	O
of	O
find	O
in	O
the	O
sum	O
cluster	O
.	O
so	O
words	O
that	O
represent	O
a	O
legend	O
of	O
concept	O
as	O
one	O
.	O
we	O
can	O
also	O
use	O
smaller	O
units	O
like	O
a	O
character	O
in	O
grams	O
.	O
those	O
are	O
sequences	O
of	O
n	O
characters	O
for	O
dimensions	O
.	O
however	O
in	O
practice	O
people	O
have	O
found	O
that	O
the	O
bag-of-words	O
representation	O
with	O
the	O
phrases	O
is	O
where	O
the	O
the	O
most	O
effective	O
one	O
.	O
and	O
it	O
s	O
also	O
efficient	O
so	O
this	O
is	O
still	O
so	O
far	O
the	O
most	O
popular	O
dimension	O
instantiation	O
method	O
and	O
it	O
s	O
used	O
in	O
all	O
the	O
major	B
search	B
engines	I
.	O
i	O
should	O
also	O
mention	O
that	O
sometimes	O
we	O
did	O
to	O
do	O
language	O
specific	O
and	O
domain	O
specific	O
organization	O
.	O
and	O
this	O
is	O
actually	O
very	O
important	O
as	O
we	O
might	O
have	O
variations	O
of	O
the	O
terms	O
.	O
that	O
might	O
prevent	O
us	O
from	O
matching	O
them	O
with	O
each	O
other	O
.	O
even	O
though	O
they	O
mean	O
the	O
same	O
thing	O
.	O
and	O
some	O
of	O
them	O
which	O
is	O
like	O
chinese	O
the	O
results	O
of	O
the	O
.	O
segmenting	O
text	O
to	O
obtain	O
word	O
boundaries	O
.	O
because	O
it	O
s	O
just	O
a	O
sequence	O
of	O
characters	O
.	O
a	O
word	O
might	O
might	O
correspond	O
to	O
one	O
character	O
or	O
two	O
characters	O
or	O
even	O
three	O
characters	O
.	O
so	O
it	O
s	O
easier	O
in	O
english	O
when	O
we	O
have	O
a	O
space	O
to	O
separate	O
the	O
words	O
.	O
but	O
in	O
some	O
other	O
languages	O
we	O
may	O
need	O
to	O
do	O
some	O
natural	B
language	I
processing	I
to	O
figure	O
out	O
the	O
where	O
are	O
the	O
boundaries	O
for	O
words	O
.	O
there	O
is	O
also	O
possibility	O
to	O
improve	O
this	O
in	O
narrative	O
function	O
.	O
and	O
so	O
far	O
we	O
have	O
used	O
the	O
about	O
product	O
but	O
one	O
can	O
imagine	O
there	O
are	O
other	O
matches	O
.	O
for	O
example	O
we	O
can	O
match	O
the	O
cosine	O
of	O
the	O
angle	O
between	O
two	O
vectors	O
or	O
we	O
can	O
use	O
euclidean	O
distance	O
measure	O
.	O
and	O
these	O
are	O
all	O
possible	O
.	O
the	O
dot	B
product	I
seems	O
still	O
the	O
best	O
and	O
one	O
of	O
the	O
reasons	O
is	O
because	O
it	O
s	O
very	O
general	O
.	O
in	O
fact	O
it	O
s	O
sufficiently	O
general	O
.	O
if	O
you	O
consider	O
the	O
possibilities	O
of	O
doing	O
weighting	O
in	O
different	O
ways	O
.	O
so	O
for	O
example	O
cosine	O
measure	O
can	O
be	O
regarded	O
as	O
the	O
dot	B
product	I
of	O
two	O
normalized	O
vectors	O
.	O
that	O
means	O
we	O
first	O
normalize	O
each	O
vector	O
and	O
then	O
we	O
take	O
the	O
dot	B
product	I
.	O
that	O
would	O
be	O
equivalent	O
to	O
the	O
cosine	O
measure	O
.	O
i	O
just	O
mentioned	O
that	O
the	O
bm	O
	O
.	O
seems	O
to	O
be	O
one	O
of	O
the	O
most	O
effective	O
formulas	O
.	O
but	O
there	O
has	O
been	O
also	O
further	O
development	O
in	O
improving	O
bm	O
although	O
none	O
of	O
these	O
works	O
have	O
changed	O
the	O
bm	O
fundamentally	O
.	O
so	O
in	O
one	O
line	O
of	O
work	O
people	O
have	O
derived	O
bm	O
f	O
.	O
here	O
f	O
stands	O
for	O
field	O
and	O
this	O
is	O
a	O
little	O
use	O
bm	O
for	O
documents	O
with	O
a	O
structures	O
.	O
for	O
example	O
you	O
might	O
consider	O
title	O
field	O
the	O
abstract	O
or	O
body	O
of	O
the	O
reasearch	O
article	O
or	O
even	O
anchor	B
text	I
on	O
the	O
web	O
pages	O
.	O
those	O
are	O
the	O
text	O
fields	O
that	O
describe	O
links	O
to	O
other	O
pages	O
.	O
and	O
these	O
can	O
all	O
be	O
combined	O
with	O
a	O
appropriate	O
weight	O
on	O
different	O
fields	O
to	O
help	O
improve	O
scoring	O
for	O
document	O
.	O
use	O
bm	O
for	O
such	O
a	O
document	O
.	O
and	O
the	O
obvious	O
choice	O
is	O
to	O
apply	O
bm	O
for	O
each	O
field	O
and	O
then	O
combine	O
the	O
scores	O
.	O
basically	O
the	O
ideal	O
of	O
bm	O
f	O
is	O
to	O
first	O
combine	O
the	O
frequency	O
counts	O
of	O
tons	O
in	O
all	O
the	O
fields	O
and	O
then	O
apply	O
bm	O
	O
.	O
now	O
this	O
has	O
advantage	O
of	O
avoiding	O
over	O
counting	O
the	O
first	O
occurrence	O
of	O
the	O
term	O
.	O
remember	O
in	O
the	O
sublinear	O
transformation	O
of	O
tf	B
the	O
first	O
recurrence	O
is	O
very	O
important	O
then	O
and	O
contributes	O
a	O
large	O
weight	O
.	O
and	O
if	O
we	O
do	O
that	O
for	O
all	O
the	O
fields	O
then	O
the	O
same	O
term	O
might	O
have	O
gained	O
a	O
a	O
lot	O
of	O
advantage	O
in	O
every	O
field	O
but	O
when	O
we	O
combine	O
these	O
word	B
frequencies	I
together	O
.	O
we	O
just	O
do	O
the	O
transformation	O
one	O
time	O
and	O
that	O
time	O
then	O
the	O
extra	O
occurrences	O
will	O
not	O
be	O
counted	O
as	O
fresh	O
first	O
occurrences	O
.	O
and	O
this	O
method	O
has	O
been	O
working	O
very	O
well	O
for	O
scoring	O
structured	O
documents	O
.	O
the	O
other	O
line	O
of	O
extension	O
is	O
called	O
a	O
bm	O
plus	O
and	O
this	O
line	O
arresters	O
have	O
addressed	O
the	O
problem	O
of	O
over	O
penalization	O
of	O
long	O
documents	O
by	O
bm	O
	O
.	O
so	O
to	O
address	O
this	O
problem	O
the	O
fix	O
is	O
actually	O
quite	O
simple	O
.	O
we	O
can	O
simply	O
add	O
a	O
small	O
constant	O
to	O
the	O
tf	B
normalization	O
formula	O
.	O
but	O
what	O
s	O
interesting	O
is	O
that	O
we	O
can	O
analytically	O
prove	O
that	O
by	O
doing	O
such	O
a	O
small	O
modification	O
we	O
will	O
fix	O
the	O
problem	O
of	O
a	O
over	O
penalization	O
of	O
long	O
documents	O
by	O
the	O
original	O
bm	O
	O
.	O
so	O
the	O
new	O
formula	O
called	O
bm	O
-plus	O
is	O
empirically	O
and	O
analytically	O
shown	O
to	O
be	O
better	O
than	O
bm	O
	O
.	O
so	O
to	O
summarize	O
all	O
what	O
we	O
have	O
said	O
about	O
the	O
vector	B
space	I
model	I
.	O
here	O
are	O
the	O
major	O
takeaway	O
points	O
.	O
first	O
in	O
such	O
a	O
model	O
we	O
use	O
the	O
similarity	O
notion	O
of	O
relevance	O
assuming	O
that	O
the	O
relevance	O
of	O
a	O
document	O
with	O
respect	O
to	O
a	O
query	O
is	O
basically	O
proportional	O
to	O
the	O
similarity	O
between	O
the	O
query	O
and	O
the	O
document	O
.	O
so	O
naturally	O
that	O
implies	O
that	O
the	O
query	O
and	O
document	O
must	O
be	O
represented	O
in	O
the	O
same	O
way	O
and	O
in	O
this	O
case	O
we	O
represent	O
them	O
as	O
vectors	O
in	O
high	O
dimensional	B
vector	B
space	I
.	O
where	O
the	O
dimensions	O
are	O
defined	O
by	O
words	O
or	O
concepts	O
or	O
terms	O
in	O
general	O
.	O
and	O
we	O
generally	O
need	O
to	O
use	O
a	O
lot	O
of	O
heuristics	O
to	O
design	O
a	O
ranking	B
function	I
.	O
we	O
use	O
some	O
examples	O
which	O
show	O
the	O
need	O
for	O
several	O
heuristics	O
including	O
tf	B
waiting	O
and	O
transformation	O
.	O
and	O
idf	B
weighting	O
and	O
document	O
length	B
normalization	I
.	O
these	O
major	O
heuristics	O
are	O
the	O
most	O
important	O
heuristics	O
to	O
ensure	O
such	O
a	O
general	O
ranking	B
function	I
to	O
work	O
well	O
for	O
all	O
kinds	O
of	O
tasks	O
.	O
and	O
finally	O
bm	O
and	O
pivoted	O
normalization	O
seem	O
to	O
be	O
the	O
most	O
effective	O
formulas	O
out	O
of	O
that	O
space	O
model	O
.	O
now	O
i	O
have	O
to	O
say	O
that	O
i	O
ve	O
put	O
bm	O
in	O
the	O
category	O
of	O
vector	B
space	I
model	I
.	O
but	O
in	O
fact	O
the	O
bm	O
has	O
been	O
derived	O
using	O
model	O
.	O
so	O
the	O
reason	O
why	O
i	O
ve	O
put	O
it	O
in	O
the	O
vector	B
space	I
model	I
is	O
first	O
the	O
ranking	B
function	I
actually	O
has	O
a	O
nice	O
interpretation	O
in	O
the	O
vector	B
space	I
model	I
.	O
we	O
can	O
easily	O
see	O
it	O
looks	O
very	O
much	O
like	O
a	O
vector	B
space	I
model	I
with	O
a	O
special	O
weighting	O
function	O
.	O
the	O
second	O
reason	O
is	O
because	O
the	O
original	O
bm	O
has	O
a	O
somewhat	O
different	O
from	O
of	O
idf	B
.	O
and	O
that	O
form	O
of	O
idf	B
actually	O
doesn	O
t	O
really	O
work	O
so	O
well	O
as	O
the	O
standard	O
idf	B
that	O
you	O
have	O
seen	O
here	O
.	O
so	O
as	O
a	O
effective	O
original	O
function	O
bm	O
should	O
probably	O
use	O
a	O
heuristic	O
modification	O
of	O
the	O
idf	B
to	O
make	O
that	O
even	O
more	O
like	O
a	O
vector	B
space	I
model	I
.	O
there	O
are	O
some	O
additional	O
readings	O
.	O
the	O
first	O
is	O
a	O
paper	O
about	O
the	O
pivoted	O
length	B
normalization	I
.	O
it	O
s	O
an	O
excellent	O
example	O
of	O
using	O
empirical	O
data	O
enhances	O
to	O
suggest	O
a	O
need	O
for	O
length	B
normalization	I
and	O
then	O
further	O
derived	O
a	O
length	B
normalization	I
formula	O
.	O
the	O
second	O
is	O
the	O
original	O
paper	O
when	O
the	O
was	O
proposed	O
.	O
the	O
third	O
paper	O
has	O
a	O
thorough	O
discussion	O
of	O
and	O
its	O
extensions	O
particularly	O
bm-	O
f	O
.	O
and	O
finally	O
the	O
last	O
paper	O
has	O
a	O
discussion	O
of	O
improving	O
bm-	O
to	O
correct	O
the	O
overpenalization	O
of	O
long	O
documents	O
.	O
this	O
lecture	O
is	O
about	O
the	O
implementation	O
of	O
text	B
retrieval	B
systems	I
.	O
in	O
this	O
lecture	O
we	O
will	O
discuss	O
how	O
we	O
can	O
implement	O
a	O
text	B
retrieval	B
method	I
to	O
build	O
a	O
search	B
engine	I
.	O
the	O
main	O
challenge	O
is	O
to	O
manage	O
a	O
lot	O
of	O
text	O
data	O
and	O
to	O
enable	O
a	O
query	O
to	O
be	O
answered	O
very	O
quickly	O
and	O
to	O
respond	O
to	O
many	O
queries	O
.	O
this	O
is	O
a	O
typical	O
text	B
retrieval	I
system	O
architecture	O
.	O
we	O
can	O
see	O
the	O
documents	O
are	O
first	O
processed	O
by	O
a	O
tokenizer	O
to	O
get	O
tokenizer	O
units	O
for	O
example	O
words	O
.	O
and	O
then	O
these	O
words	O
or	O
tokens	O
would	O
be	O
processed	O
by	O
an	O
indexer	B
that	O
would	O
create	O
an	O
index	O
which	O
is	O
a	O
data	O
structure	O
for	O
the	O
search	B
engine	I
to	O
use	O
to	O
quickly	O
answer	O
a	O
query	O
.	O
and	O
the	O
query	O
will	O
be	O
going	O
through	O
a	O
similar	O
processing	O
step	O
.	O
so	O
the	O
tokenizer	O
will	O
be	O
apprised	O
to	O
query	O
as	O
well	O
so	O
that	O
the	O
text	O
can	O
be	O
processed	O
in	O
the	O
same	O
way	O
.	O
the	O
same	O
units	O
will	O
be	O
matched	O
with	O
each	O
other	O
.	O
and	O
the	O
query	O
s	O
representation	O
will	O
then	O
be	O
given	O
to	O
the	O
scorer	O
.	O
which	O
would	O
use	O
a	O
index	O
to	O
quickly	O
answer	O
a	O
user	O
s	O
query	O
by	O
scoring	O
the	O
documents	O
and	O
then	O
ranking	O
them	O
.	O
the	O
results	O
will	O
be	O
given	O
to	O
the	O
user	O
.	O
and	O
then	O
the	O
user	O
can	O
look	O
at	O
the	O
results	O
and	O
and	O
provide	O
some	O
feedback	O
that	O
can	O
be	O
expressed	O
judgements	O
about	O
which	O
documents	O
are	O
good	O
which	O
documents	O
are	O
bad	O
or	O
implicit	B
feedback	I
such	O
as	O
pixels	O
so	O
the	O
user	O
doesn	O
t	O
have	O
to	O
any	O
anything	O
extra	O
.	O
the	O
user	O
will	O
just	O
look	O
at	O
the	O
results	O
and	O
skip	O
some	O
and	O
click	O
on	O
some	O
results	O
to	O
view	O
.	O
so	O
these	O
interaction	O
signals	O
can	O
be	O
used	O
by	O
the	O
system	O
to	O
improve	O
the	O
ranking	O
accuracy	O
by	O
assuming	O
that	O
viewed	O
documents	O
are	O
better	O
than	O
the	O
skipped	O
ones	O
.	O
so	O
a	O
search	B
engine	I
system	O
then	O
can	O
be	O
divided	O
into	O
three	O
parts	O
.	O
the	O
first	O
part	O
is	O
the	O
indexer	B
and	O
the	O
second	O
part	O
is	O
the	O
scorer	O
that	O
responds	O
to	O
the	O
user	O
s	O
query	O
.	O
and	O
the	O
third	O
part	O
is	O
the	O
feedback	O
mechanism	O
.	O
now	O
typically	O
the	O
indexer	B
is	O
done	O
in	O
the	O
offline	O
manner	O
so	O
you	O
can	O
pre-process	O
the	O
correct	O
data	O
and	O
to	O
build	O
the	O
inverter	O
index	O
which	O
we	O
will	O
introduce	O
in	O
a	O
moment	O
.	O
and	O
this	O
data	O
structure	O
can	O
then	O
be	O
used	O
by	O
the	O
online	O
module	O
which	O
is	O
a	O
scorer	O
to	O
process	O
a	O
user	O
s	O
query	O
dynamically	O
and	O
quickly	O
generate	O
search	O
results	O
.	O
the	O
feedback	O
mechanism	O
can	O
be	O
done	O
online	O
or	O
offline	O
depending	O
on	O
the	O
method	O
.	O
the	O
implementation	O
of	O
the	O
index	O
and	O
the	O
the	O
scorer	O
is	O
fairly	O
standard	O
and	O
this	O
is	O
the	O
main	O
topic	O
of	O
this	O
lecture	O
and	O
the	O
next	O
few	O
lectures	O
.	O
the	O
feedback	O
mechanism	O
on	O
the	O
other	O
hand	O
has	O
variations	O
.	O
it	O
depends	O
on	O
what	O
method	O
is	O
used	O
.	O
so	O
that	O
is	O
usually	O
done	O
in	O
a	O
algorithm-specific	O
way	O
.	O
let	O
s	O
first	O
talk	O
about	O
the	O
tokenize	O
.	O
tokenization	O
is	O
a	O
normalize	O
lexical	O
units	O
into	O
the	O
same	O
form	O
so	O
that	O
semantically	O
similar	O
words	O
can	O
be	O
matched	O
with	O
each	O
other	O
.	O
now	O
in	O
the	O
language	O
of	O
english	O
stemming	O
is	O
often	O
used	O
and	O
this	O
what	O
map	B
all	O
the	O
inflectional	O
forms	O
of	O
words	O
into	O
the	O
same	O
root	O
form	O
.	O
so	O
for	O
example	O
computer	O
computation	O
and	O
computing	O
can	O
all	O
be	O
matched	O
to	O
the	O
root	O
form	O
compute	O
.	O
this	O
way	O
all	O
these	O
different	O
forms	O
of	O
computing	O
can	O
be	O
matched	O
with	O
each	O
other	O
.	O
normally	O
this	O
is	O
a	O
good	O
idea	O
to	O
increase	O
the	O
coverage	O
of	O
documents	O
that	O
are	O
matched	O
with	O
this	O
query	O
.	O
but	O
it	O
s	O
also	O
not	O
always	O
beneficial	O
because	O
sometimes	O
the	O
subtlest	O
difference	O
between	O
computer	O
and	O
computation	O
might	O
still	O
suggest	O
the	O
difference	O
in	O
the	O
coverage	O
of	O
the	O
content	O
.	O
but	O
in	O
most	O
cases	O
stemming	O
seems	O
to	O
be	O
beneficial	O
.	O
when	O
we	O
tokenize	O
the	O
text	O
in	O
some	O
other	O
languages	O
for	O
example	O
chinese	O
we	O
might	O
face	O
some	O
special	O
challenges	O
in	O
segmenting	O
the	O
text	O
to	O
find	O
the	O
word	O
boundaries	O
.	O
because	O
it	O
s	O
not	O
ob	O
obvious	O
where	O
the	O
boundary	O
is	O
as	O
there	O
s	O
no	O
space	O
separating	O
them	O
.	O
so	O
here	O
of	O
course	O
we	O
have	O
to	O
use	O
some	O
language-specific	O
natural	B
language	I
processing	I
techniques	I
.	O
once	O
we	O
do	O
tokenization	O
then	O
we	O
would	O
index	O
the	O
text	O
documents	O
and	O
that	O
it	O
will	O
convert	O
the	O
documents	O
into	O
some	O
data	O
structure	O
that	O
can	O
enable	O
fast	O
search	O
.	O
the	O
basic	O
idea	O
is	O
to	O
precompute	O
as	O
much	O
as	O
we	O
can	O
basically	O
.	O
so	O
the	O
most	O
commonly	O
used	O
index	O
is	O
called	O
a	O
inverted	B
index	I
.	O
and	O
this	O
has	O
been	O
used	O
to	O
in	O
many	O
search	B
engines	I
to	O
support	O
basic	O
search	O
algorithms	O
.	O
sometimes	O
other	O
indices	O
for	O
example	O
a	O
document	O
index	O
might	O
be	O
needed	O
in	O
order	O
to	O
support	O
a	O
a	O
feedback	O
.	O
like	O
i	O
said	O
this	O
this	O
kind	O
of	O
techniques	O
are	O
not	O
really	O
standard	O
in	O
that	O
they	O
vary	O
a	O
lot	O
according	O
to	O
the	O
feedback	O
methods	O
.	O
to	O
understand	O
why	O
we	O
are	O
using	O
inverted	B
index	I
.	O
it	O
will	O
be	O
useful	O
for	O
you	O
to	O
think	O
about	O
how	O
you	O
would	O
respond	O
to	O
a	O
single	O
term	O
query	O
quickly	O
.	O
so	O
if	O
you	O
want	O
to	O
use	O
more	O
time	O
to	O
think	O
about	O
that	O
pause	O
the	O
video	O
.	O
so	O
think	O
about	O
how	O
you	O
can	O
preprocess	O
the	O
text	O
data	O
so	O
that	O
you	O
can	O
quickly	O
respond	O
to	O
a	O
query	O
with	O
just	O
one	O
word	O
.	O
well	O
if	O
you	O
have	O
thought	O
about	O
question	O
you	O
might	O
realize	O
that	O
where	O
the	O
best	O
is	O
to	O
simply	O
create	O
a	O
list	O
of	O
documents	O
that	O
match	O
every	O
term	O
in	O
the	O
vocabulary	O
.	O
in	O
this	O
way	O
you	O
can	O
basically	O
pre-construct	O
the	O
answers	O
.	O
so	O
when	O
you	O
see	O
a	O
term	O
you	O
can	O
simply	O
just	O
fetch	O
the	O
ranked	O
list	O
of	O
documents	O
for	O
that	O
term	O
and	O
return	O
the	O
list	O
to	O
the	O
user	O
.	O
so	O
that	O
s	O
the	O
fastest	O
way	O
to	O
respond	O
to	O
single	O
term	O
query	O
.	O
now	O
the	O
idea	O
of	O
invert	O
index	O
is	O
actually	O
basically	O
like	O
that	O
.	O
we	O
can	O
do	O
pre-construct	O
such	O
a	O
index	O
.	O
that	O
would	O
allow	O
us	O
to	O
quickly	O
find	O
the	O
all	O
the	O
documents	O
that	O
match	O
a	O
particular	O
term	O
.	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
this	O
example	O
.	O
we	O
have	O
three	O
documents	O
here	O
and	O
these	O
are	O
the	O
documents	O
that	O
you	O
have	O
seen	O
in	O
some	O
previous	O
lectures	O
.	O
suppose	O
we	O
want	O
to	O
create	O
invert	O
index	O
for	O
these	O
documents	O
then	O
we	O
will	O
need	O
to	O
maintain	O
a	O
dictionary	O
.	O
in	O
the	O
dictionary	O
we	O
ll	O
have	O
one	O
entry	O
for	O
each	O
term	O
.	O
and	O
we	O
re	O
going	O
to	O
store	O
some	O
basic	O
statistics	O
about	O
the	O
term	O
.	O
for	O
example	O
the	O
number	O
of	O
documents	O
that	O
match	O
the	O
term	O
or	O
the	O
total	O
number	O
of	O
fre	O
total	O
frequency	O
of	O
the	O
term	O
which	O
means	O
we	O
would	O
encounter	O
duplicated	O
occurrences	O
of	O
the	O
term	O
.	O
and	O
so	O
for	O
example	O
news	O
.	O
this	O
term	O
occurred	O
in	O
all	O
the	O
three	O
documents	O
.	O
so	O
the	O
count	O
of	O
documents	O
is	O
three	O
.	O
and	O
you	O
might	O
also	O
realize	O
we	O
needed	O
this	O
count	O
of	O
documents	O
or	O
document	B
frequency	I
for	O
computing	O
some	O
statistics	O
to	O
be	O
used	O
in	O
the	O
vector	B
space	I
model	I
.	O
can	O
you	O
think	O
of	O
that	O
so	O
what	O
waiting	O
heuristic	O
would	O
need	O
this	O
count	O
well	O
that	O
s	O
the	O
idf	B
right	O
inverse	B
document	B
frequency	I
.	O
so	O
idf	B
is	O
a	O
property	O
of	O
the	O
term	O
and	O
we	O
can	O
compute	O
it	O
right	O
here	O
.	O
so	O
with	O
the	O
document	O
account	O
here	O
it	O
s	O
easy	O
to	O
compute	O
the	O
idf	B
either	O
at	O
this	O
time	O
or	O
when	O
we	O
build	O
an	O
index	O
or	O
.	O
at	O
running	O
time	O
when	O
we	O
see	O
a	O
query	O
.	O
now	O
in	O
addition	O
to	O
these	O
basic	O
statistics	O
we	O
also	O
saw	O
all	O
the	O
documents	O
that	O
matched	O
news	O
.	O
and	O
these	O
entries	O
are	O
stored	O
in	O
a	O
file	O
called	O
a	O
postings	O
.	O
so	O
in	O
this	O
case	O
it	O
matched	O
documents	O
and	O
we	O
store	O
information	O
about	O
these	O
documents	O
here	O
.	O
this	O
is	O
the	O
document	O
id	O
document	O
and	O
the	O
frequency	O
is	O
	O
.	O
the	O
tf	B
is	O
for	O
news	O
.	O
in	O
the	O
second	O
document	O
it	O
s	O
also	O
etc	O
.	O
so	O
from	O
this	O
list	O
that	O
we	O
can	O
get	O
all	O
the	O
documents	O
that	O
match	O
the	O
term	O
news	O
.	O
and	O
we	O
can	O
also	O
know	O
the	O
frequency	O
of	O
news	O
in	O
these	O
documents	O
.	O
so	O
if	O
the	O
query	O
has	O
just	O
one	O
word	O
news	O
and	O
we	O
can	O
easily	O
look	O
up	O
in	O
this	O
table	O
to	O
find	O
the	O
entry	O
and	O
go	O
quickly	O
to	O
the	O
postings	O
to	O
fetch	O
all	O
the	O
documents	O
that	O
match	O
news	O
.	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
another	O
term	O
.	O
now	O
this	O
time	O
let	O
s	O
take	O
a	O
look	O
at	O
the	O
word	O
presidential	O
.	O
all	O
right	O
this	O
word	O
occurred	O
in	O
only	O
document	O
document	O
	O
.	O
so	O
the	O
document	B
frequency	I
is	O
but	O
it	O
occurred	O
twice	O
in	O
this	O
document	O
.	O
and	O
so	O
the	O
frequency	O
count	O
is	O
and	O
the	O
frequency	O
count	O
is	O
used	O
for	O
in	O
some	O
other	O
retrieval	B
method	I
where	O
we	O
might	O
use	O
the	O
frequency	O
to	O
assess	O
the	O
popularity	O
of	O
a	O
a	O
term	O
in	O
the	O
collection	O
.	O
and	O
similarly	O
we	O
ll	O
have	O
a	O
pointer	O
to	O
the	O
postings	O
right	O
here	O
.	O
and	O
in	O
this	O
case	O
there	O
is	O
only	O
one	O
entry	O
here	O
because	O
the	O
term	O
occurred	O
in	O
just	O
one	O
document	O
.	O
and	O
that	O
s	O
here	O
.	O
the	O
document	O
id	O
is	O
and	O
it	O
occurred	O
twice	O
.	O
so	O
this	O
is	O
the	O
basic	O
idea	O
of	O
inverted	B
index	I
.	O
it	O
s	O
actually	O
pretty	O
simple	O
right	O
with	O
this	O
structure	O
we	O
can	O
easily	O
fetch	O
all	O
the	O
documents	O
that	O
match	O
a	O
term	O
.	O
and	O
this	O
will	O
be	O
the	O
basis	O
for	O
storing	O
documents	O
for	O
our	O
query	O
.	O
now	O
sometimes	O
we	O
also	O
want	O
to	O
store	O
the	O
positions	O
of	O
these	O
terms	O
.	O
so	O
in	O
many	O
of	O
these	O
cases	O
the	O
term	O
occurred	O
just	O
once	O
in	O
the	O
document	O
so	O
there	O
s	O
only	O
one	O
position	O
for	O
example	O
in	O
this	O
case	O
.	O
but	O
in	O
this	O
case	O
the	O
term	O
occurred	O
twice	O
so	O
it	O
would	O
store	O
two	O
positions	O
.	O
now	O
the	O
position	O
information	O
is	O
very	O
useful	O
for	O
checking	O
whether	O
the	O
matching	O
of	O
query	B
terms	I
is	O
actually	O
within	O
a	O
small	O
window	O
of	O
let	O
s	O
say	O
five	O
words	O
or	O
ten	O
words	O
or	O
whether	O
the	O
matching	O
of	O
the	O
two	O
query	B
terms	I
is	O
in	O
fact	O
a	O
phrase	O
of	O
two	O
words	O
.	O
this	O
can	O
all	O
be	O
checked	O
quickly	O
by	O
using	O
the	O
position	O
information	O
.	O
so	O
why	O
is	O
inverted	B
index	I
good	O
for	O
faster	O
search	O
well	O
we	O
just	O
talked	O
about	O
the	O
possibility	O
of	O
using	O
the	O
two	O
ends	O
of	O
a	O
single-word	O
query	O
.	O
and	O
that	O
s	O
very	O
easy	O
.	O
what	O
about	O
a	O
multiple-term	O
queries	O
well	O
let	O
s	O
look	O
at	O
the	O
some	O
special	O
cases	O
of	O
the	O
boolean	O
query	O
.	O
a	O
boolean	O
query	O
is	O
basically	O
a	O
boolean	O
expression	O
like	O
this	O
.	O
so	O
i	O
want	O
the	O
relevant	O
document	O
to	O
match	O
both	O
term	O
a	O
and	O
term	O
b	O
.	O
all	O
right	O
so	O
that	O
s	O
one	O
conjunctive	O
query	O
.	O
or	O
i	O
want	O
the	O
relevant	B
documents	I
to	O
match	O
term	O
a	O
or	O
term	O
b	O
.	O
that	O
s	O
a	O
disjunctive	O
query	O
.	O
now	O
how	O
can	O
we	O
answer	O
such	O
a	O
query	O
by	O
using	O
inverted	B
index	I
well	O
if	O
you	O
think	O
a	O
a	O
bit	O
about	O
it	O
it	O
would	O
be	O
obvious	O
.	O
because	O
we	O
have	O
simply	O
to	O
fetch	O
all	O
the	O
documents	O
that	O
match	O
term	O
a	O
and	O
also	O
fetch	O
all	O
the	O
documents	O
that	O
match	O
term	O
b	O
.	O
and	O
then	O
just	O
take	O
the	O
intersection	O
to	O
answer	O
a	O
query	O
like	O
a	O
and	O
b	O
.	O
or	O
to	O
take	O
the	O
union	O
to	O
answer	O
the	O
query	O
a	O
or	O
b	O
.	O
so	O
this	O
is	O
all	O
very	O
easy	O
to	O
answer	O
.	O
it	O
s	O
going	O
to	O
be	O
very	O
quick	O
.	O
now	O
what	O
about	O
the	O
multi-term	O
keyword	O
query	O
we	O
talked	O
about	O
the	O
vector	B
space	I
model	I
for	O
example	O
.	O
and	O
we	O
would	O
match	O
such	O
a	O
query	O
with	O
a	O
document	O
and	O
generate	O
a	O
score	O
.	O
and	O
the	O
score	O
is	O
based	O
on	O
aggregated	O
term	O
weights	O
.	O
so	O
in	O
this	O
case	O
it	O
s	O
not	O
a	O
boolean	O
query	O
but	O
the	O
scoring	O
can	O
be	O
actually	O
done	O
in	O
a	O
similar	O
way	O
.	O
basically	O
it	O
s	O
similar	O
to	O
disjunctive	O
boolean	O
query	O
.	O
basically	O
it	O
s	O
like	O
a	O
or	O
b	O
.	O
we	O
take	O
the	O
union	O
of	O
all	O
the	O
documents	O
that	O
matched	O
at	O
least	O
one	O
query	B
term	I
and	O
then	O
we	O
would	O
aggregate	O
the	O
term	O
weights	O
.	O
so	O
this	O
is	O
a	O
a	O
a	O
basic	O
idea	O
of	O
using	O
inverted	B
index	I
for	O
scoring	O
documents	O
in	O
general	O
.	O
and	O
we	O
re	O
going	O
to	O
talk	O
about	O
this	O
in	O
more	O
detail	O
later	O
.	O
but	O
for	O
now	O
let	O
s	O
just	O
look	O
at	O
the	O
question	O
why	O
is	O
inverted	B
index	I
a	O
good	O
idea	O
basically	O
why	O
is	O
it	O
more	O
efficient	O
than	O
sequentially	O
just	O
scanning	O
documents	O
right	O
this	O
is	O
the	O
obvious	O
approach	O
.	O
you	O
can	O
just	O
compute	O
the	O
score	O
for	O
each	O
document	O
and	O
then	O
you	O
can	O
score	O
them	O
sorry	O
you	O
can	O
then	O
sort	O
them	O
.	O
this	O
is	O
a	O
a	O
straightforward	O
method	O
.	O
but	O
this	O
is	O
going	O
to	O
be	O
very	O
slow	O
.	O
imagine	O
the	O
web	O
.	O
it	O
has	O
a	O
lot	O
of	O
documents	O
.	O
if	O
you	O
do	O
this	O
then	O
it	O
will	O
take	O
a	O
long	O
time	O
to	O
answer	O
your	O
query	O
.	O
so	O
the	O
question	O
now	O
is	O
why	O
would	O
the	O
in	O
the	O
inverted	B
index	I
be	O
much	O
faster	O
well	O
it	O
has	O
to	O
do	O
with	O
the	O
word	O
distribution	O
in	O
text	O
.	O
so	O
here	O
s	O
some	O
common	O
phenomenon	O
of	O
word	O
distribution	O
in	O
text	O
.	O
there	O
are	O
some	O
language-in	O
independent	O
patterns	O
that	O
seem	O
to	O
be	O
stable	O
.	O
and	O
these	O
patterns	O
are	O
basically	O
characterized	O
by	O
the	O
following	O
pattern	O
.	O
a	O
few	O
words	O
like	O
the	O
common	O
words	O
like	O
the	O
a	O
or	O
we	O
occur	O
very	O
very	O
frequently	O
in	O
text	O
.	O
so	O
they	O
account	O
for	O
a	O
large	O
percent	O
of	O
occurrences	O
of	O
words	O
.	O
but	O
most	O
word	O
would	O
occur	O
just	O
rarely	O
.	O
there	O
are	O
many	O
words	O
that	O
occur	O
just	O
once	O
let	O
s	O
say	O
in	O
a	O
document	O
or	O
once	O
in	O
the	O
collection	O
.	O
and	O
there	O
are	O
many	O
such	O
single	O
terms	O
.	O
it	O
s	O
also	O
true	O
that	O
the	O
most	O
frequent	O
words	O
in	O
one	O
corpus	O
may	O
actually	O
be	O
rare	O
in	O
another	O
.	O
that	O
means	O
although	O
the	O
general	O
phenomenon	O
is	O
applicable	O
or	O
is	O
observed	O
in	O
many	O
cases	O
the	O
exact	O
words	O
that	O
are	O
common	O
may	O
vary	O
from	O
context	O
to	O
context	O
.	O
so	O
this	O
phenomena	O
is	O
characterized	O
by	O
what	O
s	O
called	O
a	O
zipf	O
s	O
law	O
.	O
this	O
law	O
says	O
that	O
the	O
rank	O
of	O
a	O
word	O
multiplied	O
by	O
the	O
frequency	O
of	O
the	O
word	O
is	O
roughly	O
constant	O
.	O
so	O
formally	O
if	O
we	O
use	O
f	O
of	O
w	O
to	O
denote	O
the	O
frequency	O
r	O
of	O
w	O
to	O
denote	O
the	O
rank	O
of	O
a	O
word	O
then	O
this	O
is	O
the	O
formula	O
.	O
it	O
basically	O
says	O
the	O
same	O
thing	O
just	O
mathematical	O
term	O
where	O
c	O
is	O
basically	O
a	O
constant	O
right	O
so	O
as	O
so	O
.	O
and	O
there	O
is	O
also	O
parameter	O
alpha	O
that	O
might	O
be	O
adjusted	O
to	O
better	O
fit	O
any	O
empirical	O
observations	O
.	O
so	O
if	O
i	O
plot	O
the	O
word	B
frequencies	I
in	O
sorted	O
order	O
then	O
you	O
can	O
see	O
this	O
more	O
easily	O
.	O
the	O
x-axis	O
is	O
basically	O
the	O
word	O
rank	O
.	O
and	O
this	O
is	O
r	O
of	O
w	O
.	O
and	O
the	O
y-axis	O
is	O
the	O
word	B
frequency	I
or	O
f	O
of	O
w	O
.	O
now	O
this	O
curve	O
basically	O
shows	O
that	O
the	O
product	O
of	O
the	O
two	O
is	O
roughly	O
the	O
constant	O
.	O
now	O
if	O
you	O
look	O
these	O
words	O
we	O
can	O
see	O
.	O
they	O
can	O
be	O
separated	O
into	O
three	O
group	O
s	O
.	O
in	O
the	O
middle	O
it	O
s	O
the	O
immediate	O
frequency	O
words	O
.	O
these	O
words	O
tend	O
to	O
occur	O
in	O
quite	O
a	O
few	O
documents	O
right	O
but	O
they	O
re	O
not	O
like	O
those	O
most	O
frequent	O
words	O
.	O
and	O
they	O
are	O
also	O
not	O
very	O
rare	O
.	O
so	O
they	O
tend	O
to	O
be	O
often	O
used	O
in	O
in	O
in	O
queries	O
.	O
and	O
they	O
also	O
tend	O
to	O
have	O
high	O
tfi	O
diff	O
weights	O
in	O
these	O
intermediate	O
frequency	O
words	O
.	O
but	O
if	O
you	O
look	O
at	O
the	O
left	O
part	O
of	O
the	O
curve	O
.	O
these	O
are	O
the	O
highest	O
frequency	O
words	O
.	O
they	O
occur	O
very	O
frequently	O
.	O
they	O
are	O
usually	O
stopper	O
words	O
the	O
we	O
of	O
et	O
cetera	O
.	O
those	O
words	O
are	O
very	O
very	O
frequently	O
.	O
they	O
are	O
in	O
fact	O
a	O
too	O
frequently	O
to	O
be	O
discriminated	O
.	O
and	O
they	O
generally	O
are	O
not	O
very	O
useful	O
for	O
for	O
retrieval	O
.	O
so	O
they	O
are	O
often	O
removed	O
and	O
this	O
is	O
called	O
a	O
stop	O
words	O
removal	O
.	O
so	O
you	O
can	O
use	O
pretty	O
much	O
just	O
the	O
count	O
of	O
words	O
in	O
the	O
collection	O
to	O
kind	O
of	O
infer	O
what	O
words	O
might	O
be	O
stop	O
words	O
.	O
those	O
are	O
basically	O
the	O
highest	O
frequency	O
words	O
.	O
and	O
they	O
also	O
occupy	O
a	O
lot	O
of	O
space	O
in	O
the	O
invert	O
index	O
.	O
you	O
can	O
imagine	O
the	O
posting	O
entries	O
for	O
such	O
a	O
word	O
would	O
be	O
very	O
long	O
.	O
and	O
then	O
therefore	O
if	O
you	O
can	O
remove	O
such	O
words	O
you	O
can	O
save	O
a	O
lot	O
of	O
space	O
in	O
the	O
invert	O
index	O
.	O
we	O
also	O
show	O
the	O
tail	O
part	O
which	O
is	O
has	O
a	O
lot	O
of	O
rare	O
words	O
.	O
those	O
words	O
don	O
t	O
occur	O
very	O
frequently	O
and	O
there	O
are	O
many	O
such	O
words	O
.	O
those	O
words	O
are	O
actually	O
very	O
useful	O
for	O
search	O
also	O
if	O
a	O
user	O
happens	O
to	O
be	O
interested	O
in	O
such	O
a	O
topic	O
.	O
but	O
because	O
they	O
re	O
rare	O
it	O
s	O
often	O
true	O
that	O
users	O
are	O
aren	O
t	O
the	O
necessary	O
interest	O
in	O
those	O
words	O
.	O
but	O
retain	O
them	O
would	O
allow	O
us	O
to	O
match	O
such	O
a	O
document	O
accurately	O
and	O
they	O
generally	O
have	O
very	O
high	O
idfs	O
.	O
so	O
what	O
kind	O
of	O
data	O
structures	O
should	O
we	O
use	O
to	O
to	O
store	O
inverted	B
index	I
well	O
it	O
has	O
two	O
parts	O
right	O
if	O
you	O
recall	B
we	O
have	O
a	O
dictionary	O
and	O
we	O
also	O
have	O
postings	O
.	O
the	O
dictionary	O
has	O
modest	O
size	O
although	O
for	O
the	O
web	O
it	O
still	O
wouldn	O
t	O
be	O
very	O
large	O
.	O
but	O
compared	O
with	O
postings	O
it	O
s	O
modest	O
.	O
and	O
we	O
also	O
need	O
to	O
have	O
fast	O
random	O
access	O
to	O
the	O
entries	O
because	O
we	O
want	O
to	O
look	O
up	O
the	O
query	B
term	I
very	O
quickly	O
.	O
so	O
therefore	O
we	O
prefer	O
to	O
keep	O
such	O
a	O
dictionary	O
in	O
memory	O
if	O
it	O
s	O
possible	O
.	O
or	O
or	O
or	O
if	O
the	O
connection	O
is	O
not	O
very	O
large	O
and	O
this	O
is	O
visible	O
.	O
but	O
if	O
the	O
connection	O
is	O
very	O
large	O
then	O
it	O
s	O
in	O
general	O
not	O
possible	O
.	O
if	O
the	O
vocabulary	O
size	O
is	O
very	O
large	O
obviously	O
we	O
can	O
t	O
do	O
that	O
.	O
so	O
but	O
in	O
general	O
that	O
s	O
our	O
goal	O
.	O
so	O
the	O
data	O
structures	O
that	O
we	O
often	O
use	O
for	O
storing	O
dictionary	O
would	O
be	O
direct	O
access	O
data	O
structures	O
like	O
a	O
hash	O
table	O
or	O
b-tree	O
if	O
we	O
can	O
t	O
store	O
everything	O
in	O
memory	O
of	O
the	O
newest	O
disk	O
.	O
and	O
but	O
to	O
try	O
to	O
build	O
a	O
structure	O
that	O
would	O
allow	O
it	O
to	O
quickly	O
look	O
up	O
our	O
entries	O
.	O
right	O
.	O
for	O
postings	O
they	O
re	O
huge	O
you	O
can	O
see	O
.	O
and	O
in	O
general	O
we	O
don	O
t	O
have	O
to	O
have	O
direct	O
access	O
to	O
a	O
specific	O
engine	O
.	O
we	O
generally	O
would	O
just	O
look	O
up	O
a	O
a	O
sequence	O
of	O
document	O
ids	O
and	O
frequencies	O
for	O
all	O
of	O
the	O
documents	O
that	O
match	O
a	O
query	B
term	I
.	O
so	O
we	O
would	O
read	O
those	O
entries	O
sequentially	O
.	O
and	O
therefore	O
because	O
it	O
s	O
large	O
and	O
we	O
generate	O
have	O
store	O
postings	O
on	O
disk	O
so	O
they	O
have	O
to	O
stay	O
on	O
disk	O
.	O
and	O
they	O
would	O
contain	O
information	O
such	O
as	O
document	O
ids	O
term	O
frequencies	O
or	O
term	O
positions	O
et	O
cetera	O
.	O
now	O
because	O
they	O
re	O
very	O
large	O
compression	O
is	O
often	O
desirable	O
.	O
now	O
this	O
is	O
not	O
only	O
to	O
save	O
disk	O
space	O
and	O
this	O
is	O
of	O
course	O
one	O
benefit	O
of	O
compression	O
.	O
it	O
s	O
not	O
going	O
to	O
occupy	O
that	O
much	O
space	O
.	O
but	O
it	O
s	O
also	O
to	O
help	O
improving	O
speed	O
.	O
can	O
you	O
see	O
why	O
well	O
we	O
know	O
that	O
input	O
and	O
output	O
will	O
cost	O
a	O
lot	O
of	O
time	O
in	O
comparison	O
with	O
the	O
time	O
taken	O
by	O
cpu	O
.	O
so	O
cpu	O
is	O
much	O
faster	O
.	O
but	O
io	O
takes	O
time	O
.	O
and	O
so	O
by	O
compressing	B
the	O
inverted	B
index	I
the	O
posting	O
files	O
will	O
become	O
smaller	O
.	O
and	O
the	O
entries	O
that	O
we	O
have	O
to	O
read	O
into	O
memory	O
to	O
process	O
a	O
query	O
done	O
would	O
would	O
be	O
smaller	O
.	O
and	O
then	O
so	O
we	O
we	O
can	O
reduce	O
the	O
amount	O
of	O
traffic	O
and	O
io	O
.	O
and	O
that	O
can	O
save	O
a	O
lot	O
of	O
time	O
.	O
of	O
course	O
we	O
have	O
to	O
then	O
do	O
more	O
processing	O
of	O
the	O
data	O
when	O
we	O
uncompress	O
the	O
the	O
data	O
in	O
the	O
memory	O
.	O
but	O
as	O
i	O
said	O
cpu	O
is	O
fast	O
so	O
overall	O
we	O
can	O
still	O
save	O
time	O
.	O
so	O
compression	O
here	O
is	O
both	O
to	O
save	O
disk	O
space	O
and	O
to	O
speed	O
up	O
the	O
loading	O
of	O
the	O
inverted	B
index	I
.	O
	O
.	O
this	O
lecture	O
is	O
about	O
the	O
inverted	B
index	B
construction	I
.	O
in	O
this	O
lecture	O
we	O
will	O
continue	O
the	O
discussion	O
of	O
system	O
implementation	O
.	O
in	O
particular	O
we	O
re	O
going	O
to	O
discuss	O
how	O
to	O
construct	O
the	O
inverted	B
index	I
.	O
the	O
construction	O
of	O
the	O
inverted	B
index	I
is	O
actually	O
very	O
easy	O
if	O
the	O
data	O
set	O
is	O
very	O
small	O
.	O
it	O
s	O
very	O
easy	O
to	O
construct	O
a	O
dictionary	O
and	O
then	O
store	O
the	O
postings	O
in	O
a	O
file	O
.	O
the	O
problem	O
s	O
that	O
when	O
our	O
data	O
is	O
not	O
able	O
to	O
fit	O
to	O
the	O
memory	O
then	O
we	O
have	O
to	O
use	O
some	O
special	O
method	O
to	O
deal	O
with	O
it	O
.	O
and	O
unfortunately	O
in	O
most	O
retrieval	O
a	O
petitions	O
the	O
data	O
set	O
would	O
be	O
large	O
and	O
they	O
generally	O
cannot	O
be	O
loaded	O
into	O
the	O
memory	O
at	O
once	O
.	O
and	O
there	O
are	O
many	O
approaches	O
to	O
solving	O
that	O
problem	O
and	O
sorting-based	O
method	O
is	O
quite	O
common	O
and	O
works	O
in	O
four	O
steps	O
as	O
shown	O
here	O
.	O
first	O
we	O
collect	O
the	O
the	O
local	O
termid	O
document	O
id	O
and	O
frequency	O
tuples	O
.	O
basically	O
you	O
overlook	O
kinds	O
of	O
terms	O
in	O
a	O
small	O
set	O
of	O
documents	O
and	O
and	O
then	O
once	O
you	O
collect	O
those	O
counts	O
you	O
can	O
sort	O
those	O
counts	O
based	O
on	O
terms	O
so	O
that	O
you	O
build	O
a	O
local	O
a	O
partial	O
inverted	B
index	I
.	O
and	O
these	O
are	O
called	O
runs	O
.	O
and	O
then	O
you	O
write	O
them	O
into	O
a	O
temporary	O
file	O
on	O
the	O
disk	O
.	O
and	O
then	O
you	O
merge	O
in	O
step	O
three	O
with	O
do	O
pair-wise	O
merging	O
of	O
these	O
runs	O
and	O
here	O
you	O
eventually	O
merge	O
all	O
the	O
runs	O
we	O
generate	O
a	O
single	O
inverted	B
index	I
.	O
so	O
this	O
is	O
an	O
illustration	O
of	O
this	O
method	O
.	O
on	O
the	O
left	O
you	O
see	O
some	O
documents	O
.	O
and	O
on	O
the	O
right	O
we	O
have	O
show	O
a	O
term	O
lexicon	O
and	O
a	O
document	O
id	O
lexicon	O
.	O
and	O
these	O
lexicon	O
s	O
are	O
to	O
map	B
a	O
stream	O
based	O
representations	O
of	O
document	O
ids	O
or	O
terms	O
into	O
integer	O
representations	O
.	O
or	O
and	O
map	B
back	O
from	O
integers	O
to	O
the	O
screen	O
representation	O
.	O
and	O
the	O
reason	O
why	O
we	O
want	O
are	O
interested	O
in	O
using	O
integers	O
represent	O
these	O
ids	O
is	O
because	O
integers	O
are	O
often	O
easier	O
to	O
handle	O
.	O
for	O
example	O
integers	O
can	O
be	O
used	O
as	O
index	O
for	O
array	O
and	O
they	O
are	O
also	O
easy	O
to	O
compress	O
.	O
so	O
this	O
is	O
a	O
one	O
reason	O
why	O
we	O
tend	O
to	O
map	B
these	O
streams	O
into	O
integers	O
so	O
that	O
so	O
that	O
we	O
don	O
t	O
have	O
to	O
carry	O
these	O
streams	O
around	O
.	O
so	O
how	O
does	O
this	O
approach	O
work	O
well	O
it	O
s	O
very	O
simple	O
.	O
we	O
re	O
going	O
to	O
scan	O
these	O
documents	O
sequentially	O
and	O
then	O
pause	O
the	O
documents	O
and	O
a	O
count	O
the	O
frequencies	O
of	O
terms	O
.	O
and	O
in	O
this	O
stage	O
we	O
generally	O
sort	O
the	O
frequencies	O
by	O
document	O
ids	O
because	O
we	O
process	O
each	O
document	O
that	O
sequentially	O
.	O
so	O
we	O
first	O
encounter	O
all	O
the	O
terms	O
in	O
the	O
first	O
document	O
.	O
therefore	O
the	O
document	O
ids	O
are	O
all	O
once	O
in	O
this	O
stage	O
.	O
and	O
so	O
and	O
this	O
would	O
be	O
followed	O
by	O
document	O
ids	O
	O
.	O
and	O
and	O
they	O
re	O
naturally	O
sort	O
in	O
this	O
order	O
just	O
because	O
we	O
process	O
the	O
data	O
in	O
this	O
order	O
.	O
at	O
some	O
point	O
the	O
we	O
will	O
run	O
out	O
of	O
memory	O
and	O
that	O
would	O
have	O
to	O
to	O
write	O
them	O
into	O
the	O
disk	O
.	O
but	O
before	O
we	O
do	O
that	O
we	O
re	O
going	O
to	O
a	O
sort	O
them	O
just	O
use	O
whatever	O
memory	O
we	O
have	O
we	O
can	O
sort	O
them	O
and	O
then	O
this	O
time	O
we	O
re	O
going	O
to	O
sort	O
based	O
on	O
term	O
ids	O
.	O
note	O
that	O
here	O
we	O
re	O
using	O
this	O
the	O
term	O
ids	O
as	O
a	O
key	O
to	O
sort	O
.	O
so	O
all	O
the	O
entries	O
that	O
share	O
the	O
same	O
term	O
would	O
be	O
grouped	O
together	O
.	O
in	O
this	O
case	O
we	O
can	O
see	O
all	O
the	O
all	O
the	O
ids	O
of	O
documents	O
that	O
match	O
term	O
one	O
would	O
be	O
grouped	O
together	O
.	O
and	O
we	O
re	O
going	O
to	O
write	O
this	O
into	O
the	O
disk	O
as	O
a	O
temporary	O
file	O
.	O
and	O
that	O
would	O
allow	O
us	O
to	O
use	O
the	O
memory	O
to	O
process	O
the	O
next	O
batch	O
of	O
documents	O
and	O
we	O
re	O
going	O
to	O
do	O
that	O
for	O
all	O
the	O
documents	O
.	O
so	O
we	O
re	O
going	O
to	O
write	O
a	O
lot	O
of	O
temporary	O
files	O
into	O
the	O
disk	O
.	O
and	O
then	O
the	O
next	O
stage	O
is	O
to	O
do	O
merge	O
sort	O
.	O
basically	O
we	O
re	O
going	O
to	O
merge	O
them	O
and	O
the	O
sort	O
them	O
.	O
eventually	O
we	O
will	O
get	O
a	O
single	O
inverted	B
index	I
where	O
the	O
their	O
entries	O
are	O
sorted	O
based	O
on	O
term	O
ids	O
.	O
and	O
on	O
the	O
top	O
we	O
can	O
see	O
these	O
are	O
the	O
order	O
entries	O
for	O
the	O
documents	O
that	O
match	O
term	O
id	O
	O
.	O
so	O
this	O
is	O
basically	O
how	O
we	O
can	O
do	O
the	O
construction	O
of	O
inverted	B
index	I
even	O
though	O
that	O
they	O
re	O
or	O
cannot	O
be	O
or	O
loaded	O
into	O
the	O
memory	O
.	O
now	O
we	O
mentioned	O
earlier	O
that	O
because	O
the	O
po	O
postings	O
are	O
very	O
large	O
it	O
s	O
desirable	O
to	O
compress	O
them	O
.	O
so	O
let	O
s	O
now	O
talk	O
a	O
little	O
bit	O
about	O
how	O
we	O
compress	O
inverted	B
index	I
.	O
well	O
the	O
idea	O
of	O
compression	O
in	O
general	O
is	O
you	O
leverage	O
skewed	O
distributions	O
of	O
values	O
.	O
and	O
we	O
generally	O
have	O
to	O
use	O
variable	O
lengths	O
in	O
coding	O
instead	O
of	O
the	O
fixed	O
lengths	O
in	O
coding	O
as	O
we	O
using	O
by	O
defaulting	O
a	O
program	O
language	O
like	O
c	O
	O
.	O
and	O
so	O
how	O
can	O
we	O
leverage	O
the	O
skewed	O
distributions	O
of	O
values	O
to	O
compress	O
these	O
values	O
well	O
in	O
general	O
we	O
would	O
use	O
fewer	O
bits	O
to	O
encode	O
those	O
frequent	O
words	O
at	O
a	O
cost	O
of	O
using	O
longer	O
bits	O
from	O
the	O
code	O
than	O
those	O
rare	O
values	O
.	O
so	O
in	O
our	O
case	O
let	O
s	O
think	O
about	O
how	O
we	O
can	O
compress	O
the	O
tf	B
term	B
frequency	I
.	O
if	O
you	O
can	O
picture	O
what	O
the	O
inverted	B
index	I
would	O
look	O
like	O
and	O
you	O
ll	O
see	O
in	O
postings	O
there	O
are	O
a	O
lot	O
of	O
term	O
frequencies	O
.	O
those	O
are	O
the	O
frequencies	O
of	O
terms	O
in	O
all	O
those	O
documents	O
.	O
now	O
we	O
if	O
you	O
think	O
about	O
it	O
what	O
kind	O
of	O
values	O
are	O
most	O
frequent	O
there	O
you	O
probably	O
will	O
be	O
able	O
to	O
guess	O
that	O
the	O
small	O
numbers	O
tend	O
to	O
occur	O
far	O
more	O
frequently	O
than	O
large	O
numbers	O
.	O
why	O
well	O
think	O
of	O
about	O
the	O
distribution	O
of	O
words	O
and	O
this	O
is	O
due	O
to	O
zipf	O
s	O
law	O
and	O
many	O
words	O
occur	O
just	O
rarely	O
.	O
so	O
we	O
see	O
a	O
lot	O
of	O
small	O
numbers	O
therefore	O
we	O
can	O
use	O
fewer	O
bits	O
for	O
the	O
small	O
but	O
highly	O
frequent	O
integers	O
and	O
at	O
the	O
cost	O
of	O
using	O
more	O
bits	O
for	O
large	O
integers	O
.	O
this	O
is	O
a	O
trade-off	O
of	O
course	O
.	O
if	O
the	O
values	O
are	O
distributed	O
uniformly	O
and	O
this	O
won	O
t	O
save	O
us	O
any	O
spacing	O
.	O
but	O
because	O
we	O
tend	O
to	O
see	O
many	O
small	O
values	O
they	O
re	O
very	O
frequent	O
.	O
we	O
can	O
save	O
on	O
average	O
even	O
though	O
sometimes	O
when	O
we	O
see	O
a	O
large	O
number	O
we	O
have	O
to	O
use	O
a	O
lot	O
of	O
bits	O
.	O
what	O
about	O
the	O
document	O
ids	O
that	O
we	O
also	O
saw	O
in	O
postings	O
.	O
well	O
they	O
are	O
not	O
distributed	O
in	O
a	O
skewed	O
way	O
right	O
so	O
how	O
can	O
we	O
deal	O
with	O
that	O
well	O
it	O
turns	O
out	O
you	O
can	O
use	O
a	O
trick	O
called	O
the	O
d-gap	O
and	O
that	O
that	O
is	O
to	O
store	O
the	O
difference	O
of	O
these	O
term	O
ids	O
.	O
and	O
we	O
can	O
imagine	O
if	O
a	O
term	O
has	O
matched	O
many	O
documents	O
then	O
there	O
will	O
be	O
a	O
long	O
list	O
of	O
document	O
ids	O
.	O
so	O
when	O
we	O
take	O
the	O
gap	B
and	O
when	O
we	O
take	O
difference	O
between	O
adjacent	O
document	O
ids	O
those	O
gaps	O
will	O
be	O
small	O
.	O
so	O
we	O
ll	O
again	O
see	O
a	O
lot	O
of	O
small	O
numbers	O
whereas	O
if	O
a	O
term	O
occurred	O
in	O
only	O
a	O
few	O
documents	O
then	O
the	O
gap	B
would	O
be	O
large	O
.	O
the	O
larger	O
numbers	O
will	O
not	O
be	O
frequent	O
so	O
this	O
creates	O
some	O
skewed	O
distribution	O
that	O
would	O
allow	O
us	O
to	O
to	O
compress	O
these	O
values	O
.	O
this	O
is	O
also	O
possible	O
because	O
in	O
order	O
to	O
uncover	O
or	O
uncompress	O
these	O
document	O
ids	O
we	O
have	O
to	O
sequentially	O
process	O
the	O
data	O
because	O
we	O
stored	O
the	O
difference	O
.	O
and	O
in	O
order	O
to	O
recover	O
the	O
the	O
exact	O
document	O
id	O
we	O
have	O
to	O
first	O
recover	O
the	O
previous	O
document	O
id	O
and	O
then	O
we	O
can	O
add	O
the	O
difference	O
to	O
the	O
previous	O
document	O
id	O
to	O
restore	O
the	O
the	O
current	O
document	O
id	O
.	O
now	O
this	O
was	O
possible	O
because	O
we	O
only	O
needed	O
to	O
have	O
sequential	O
access	O
to	O
those	O
document	O
ids	O
.	O
once	O
we	O
look	O
up	O
a	O
term	O
we	O
fetch	O
all	O
the	O
document	O
ids	O
that	O
match	O
the	O
term	O
then	O
we	O
sequentially	O
process	O
them	O
.	O
so	O
it	O
s	O
very	O
natural	O
that	O
s	O
why	O
this	O
trick	O
actually	O
works	O
.	O
and	O
there	O
are	O
many	O
different	O
methods	O
for	O
encoding	O
.	O
so	O
binary	O
code	O
is	O
a	O
common	O
used	O
code	O
in	O
in	O
just	O
any	O
program	O
.	O
language	O
that	O
we	O
use	O
basically	O
a	O
fixed	O
length	O
in	O
coding	O
.	O
unary	O
code	O
and	O
gamma	B
code	O
and	O
delta	O
code	O
are	O
all	O
possible	O
in	O
this	O
and	O
there	O
are	O
many	O
other	O
possible	O
in	O
this	O
.	O
so	O
let	O
s	O
look	O
at	O
some	O
of	O
them	O
in	O
more	O
detail	O
.	O
binary	O
code	O
is	O
really	O
equal-length	O
in	O
coding	O
.	O
and	O
that	O
s	O
a	O
property	O
for	O
the	O
randomly	O
distributed	O
values	O
.	O
the	O
unary	O
coding	O
is	O
is	O
a	O
variable	O
and	O
it	O
s	O
important	O
	O
.	O
in	O
this	O
case	O
integer	O
that	O
is	O
i	O
ve	O
missed	O
one	O
or	O
we	O
encode	O
that	O
as	O
x	O
minus	O
bit	O
followed	O
by	O
	O
.	O
so	O
for	O
example	O
would	O
be	O
encoded	O
as	O
two	O
s	O
followed	O
by	O
a	O
whereas	O
would	O
be	O
encoded	O
as	O
four	O
s	O
followed	O
by	O
et	O
cetera	O
.	O
so	O
now	O
now	O
you	O
can	O
imagine	O
how	O
many	O
bits	O
do	O
we	O
have	O
to	O
use	O
for	O
a	O
large	O
number	O
like	O
	O
.	O
so	O
how	O
many	O
bits	O
do	O
i	O
have	O
to	O
use	O
for	O
exactly	O
for	O
a	O
number	O
like	O
well	O
exactly	O
we	O
have	O
to	O
use	O
bits	O
but	O
so	O
it	O
s	O
the	O
same	O
number	O
of	O
bits	O
as	O
the	O
value	O
of	O
this	O
number	O
.	O
so	O
this	O
is	O
very	O
inefficient	O
.	O
if	O
you	O
were	O
likely	O
to	O
see	O
some	O
large	O
numbers	O
imagine	O
if	O
you	O
occasionally	O
see	O
a	O
number	O
like	O
you	O
have	O
to	O
use	O
bits	O
.	O
so	O
this	O
only	O
works	O
where	O
if	O
you	O
are	O
absolutely	O
sure	O
that	O
there	O
would	O
be	O
no	O
large	O
numbers	O
.	O
mostly	O
very	O
frequent	O
they	O
re	O
often	O
using	O
very	O
small	O
numbers	O
.	O
now	O
how	O
do	O
you	O
decode	O
this	O
code	O
since	O
these	O
are	O
variables	O
lengths	O
in	O
coding	O
methods	O
and	O
you	O
can	O
t	O
just	O
count	O
how	O
many	O
bits	O
and	O
then	O
just	O
stop	O
.	O
right	O
you	O
can	O
say	O
eight	O
bits	O
or	O
bits	O
then	O
you	O
you	O
will	O
start	O
another	O
code	O
.	O
there	O
are	O
variable	O
lengths	O
so	O
you	O
have	O
to	O
rely	O
on	O
some	O
mechanism	O
.	O
in	O
this	O
case	O
for	O
unary	O
you	O
can	O
see	O
it	O
s	O
very	O
easy	O
to	O
see	O
the	O
boundary	O
.	O
now	O
you	O
can	O
easily	O
see	O
would	O
signal	O
the	O
end	O
of	O
encoding	O
.	O
so	O
you	O
just	O
count	O
how	O
many	O
s	O
you	O
have	O
seen	O
and	O
then	O
you	O
hit	O
the	O
	O
.	O
you	O
know	O
you	O
have	O
finished	O
one	O
number	O
you	O
start	O
another	O
number	O
.	O
now	O
which	O
is	O
to	O
start	O
at	O
unary	O
code	O
is	O
to	O
aggressive	O
in	O
rewarding	O
small	O
numbers	O
.	O
and	O
if	O
you	O
occasionally	O
can	O
see	O
a	O
very	O
big	O
number	O
it	O
will	O
be	O
a	O
disaster	O
.	O
so	O
what	O
about	O
some	O
other	O
less	O
aggressive	O
method	O
well	O
gamma	B
coding	O
is	O
one	O
of	O
them	O
.	O
and	O
in	O
this	O
method	O
we	O
can	O
do	O
use	O
unary	O
coding	O
for	O
a	O
transformed	O
form	O
of	O
the	O
value	O
.	O
so	O
it	O
s	O
plus	O
the	O
flow	O
of	O
log	O
of	O
x	O
.	O
so	O
the	O
magnitude	O
of	O
this	O
value	O
is	O
much	O
lower	O
than	O
the	O
original	O
x	O
.	O
so	O
that	O
s	O
why	O
we	O
have	O
four	O
using	O
urinary	O
code	O
for	O
that	O
so	O
and	O
so	O
we	O
first	O
we	O
have	O
the	O
urinary	O
code	O
for	O
coding	O
this	O
log	O
of	O
s	O
.	O
and	O
this	O
will	O
be	O
followed	O
by	O
a	O
uniform	O
code	O
or	O
binary	O
code	O
and	O
this	O
is	O
basically	O
the	O
same	O
uniform	O
code	O
and	O
binary	O
code	O
are	O
the	O
same	O
.	O
and	O
we	O
re	O
going	O
to	O
use	O
this	O
code	O
to	O
code	O
the	O
remaining	O
part	O
of	O
the	O
value	O
of	O
x	O
.	O
and	O
this	O
is	O
basically	O
precisely	O
x	O
minus	O
to	O
the	O
flow	O
of	O
log	O
of	O
x	O
.	O
so	O
the	O
unary	O
code	O
or	O
basically	O
code	O
with	O
a	O
flow	O
of	O
log	O
of	O
x	O
well	O
i	O
added	O
one	O
there	O
and	O
here	O
.	O
but	O
the	O
remaining	O
part	O
will	O
we	O
using	O
uniform	O
code	O
to	O
actually	O
code	O
the	O
difference	O
between	O
the	O
x	O
and	O
and	O
this	O
to	O
the	O
log	O
of	O
x	O
.	O
and	O
and	O
it	O
s	O
easy	O
to	O
to	O
show	O
that	O
for	O
this	O
this	O
value	O
there	O
s	O
difference	O
.	O
we	O
only	O
need	O
to	O
use	O
up	O
to	O
this	O
many	O
bits	O
and	O
in	O
flow	O
of	O
log	O
of	O
x	O
bits	O
.	O
and	O
this	O
is	O
easy	O
to	O
understand	O
if	O
the	O
difference	O
is	O
too	O
large	O
then	O
we	O
would	O
have	O
a	O
higher	O
flow	O
of	O
log	O
of	O
x	O
.	O
so	O
here	O
are	O
some	O
examples	O
.	O
for	O
example	O
is	O
encoded	O
as	O
	O
.	O
the	O
first	O
two	O
digits	O
are	O
the	O
unary	O
code	O
.	O
right	O
.	O
so	O
this	O
is	O
for	O
the	O
value	O
	O
.	O
right	O
.	O
encodes	O
in	O
unary	O
coding	O
.	O
and	O
so	O
that	O
means	O
log	O
of	O
x	O
the	O
flow	O
of	O
log	O
of	O
x	O
is	O
because	O
we	O
will	O
actually	O
use	O
unary	O
code	O
to	O
encode	O
plus	O
the	O
flow	O
of	O
log	O
of	O
x	O
.	O
since	O
this	O
is	O
then	O
we	O
know	O
that	O
the	O
floor	O
of	O
log	O
of	O
x	O
is	O
actually	O
	O
.	O
so	O
but	O
is	O
still	O
larger	O
than	O
to	O
the	O
so	O
the	O
difference	O
is	O
and	O
that	O
is	O
encoded	O
here	O
at	O
the	O
end	O
.	O
so	O
that	O
s	O
why	O
we	O
have	O
for	O
	O
.	O
now	O
similarly	O
is	O
encoded	O
as	O
followed	O
by	O
	O
.	O
and	O
in	O
this	O
case	O
the	O
unary	O
code	O
encodes	O
	O
.	O
so	O
this	O
is	O
the	O
unary	O
code	O
for	O
and	O
so	O
the	O
floor	O
of	O
log	O
of	O
x	O
is	O
	O
.	O
and	O
that	O
means	O
we	O
will	O
compute	O
the	O
difference	O
between	O
and	O
the	O
to	O
the	O
and	O
that	O
s	O
and	O
so	O
we	O
now	O
have	O
again	O
at	O
the	O
end	O
.	O
but	O
this	O
time	O
we	O
re	O
going	O
to	O
use	O
two	O
bits	O
because	O
with	O
this	O
level	O
of	O
flow	O
of	O
log	O
of	O
x	O
we	O
could	O
have	O
more	O
numbers	O
	O
.	O
they	O
would	O
all	O
share	O
the	O
same	O
prefix	O
here	O
	O
.	O
so	O
in	O
order	O
to	O
differentiate	O
them	O
we	O
have	O
to	O
use	O
two	O
bits	O
in	O
the	O
end	O
to	O
differentiate	O
them	O
.	O
so	O
you	O
can	O
imagine	O
would	O
be	O
here	O
in	O
the	O
end	O
instead	O
of	O
after	O
	O
.	O
it	O
s	O
also	O
true	O
that	O
the	O
form	O
of	O
a	O
gamma	B
code	O
is	O
always	O
the	O
first	O
odd	O
number	O
of	O
bits	O
and	O
in	O
the	O
center	O
there	O
was	O
a	O
	O
.	O
that	O
s	O
the	O
end	O
of	O
the	O
unary	O
code	O
.	O
and	O
before	O
that	O
or	O
to	O
on	O
the	O
left	O
side	O
of	O
this	O
there	O
will	O
be	O
all	O
s	O
.	O
and	O
on	O
the	O
right	O
side	O
of	O
this	O
it	O
s	O
binary	B
coding	I
or	O
uniform	O
coding	O
.	O
so	O
how	O
can	O
you	O
decode	O
such	O
a	O
code	O
well	O
you	O
again	O
first	O
do	O
unary	O
coding	O
right	O
once	O
you	O
hit	O
you	O
know	O
you	O
have	O
got	O
the	O
unary	O
code	O
.	O
and	O
this	O
also	O
will	O
tell	O
you	O
how	O
many	O
bits	O
you	O
have	O
to	O
read	O
further	O
to	O
decode	O
the	O
uniform	O
code	O
.	O
so	O
this	O
is	O
how	O
you	O
can	O
decode	O
a	O
gamma	B
code	O
.	O
there	O
is	O
also	O
delta	O
code	O
but	O
that	O
s	O
basically	O
same	O
as	O
gamma	B
code	O
except	O
that	O
you	O
replace	O
the	O
unary	O
prefix	O
with	O
the	O
gamma	B
code	O
.	O
so	O
that	O
s	O
even	O
less	O
conservative	O
than	O
gamma	B
code	O
in	O
terms	O
of	O
avoiding	O
the	O
small	O
integers	O
.	O
so	O
that	O
means	O
it	O
s	O
okay	O
if	O
you	O
occasionally	O
see	O
a	O
large	O
number	O
.	O
it	O
s	O
it	O
s	O
you	O
know	O
it	O
s	O
okay	O
with	O
delta	O
code	O
.	O
it	O
s	O
also	O
fine	O
with	O
gamma	B
code	O
.	O
it	O
s	O
really	O
a	O
big	O
loss	O
for	O
unary	O
code	O
and	O
they	O
are	O
all	O
operating	O
of	O
course	O
at	O
different	O
degrees	O
of	O
favoring	O
short	O
favoring	O
small	O
integers	O
.	O
and	O
that	O
also	O
means	O
they	O
would	O
appropriate	O
for	O
sorting	O
distribution	O
.	O
but	O
none	O
of	O
them	O
is	O
perfect	O
for	O
all	O
distributions	O
.	O
and	O
which	O
method	O
works	O
the	O
best	O
would	O
have	O
to	O
depend	O
on	O
the	O
actual	O
distribution	O
in	O
your	O
data	O
set	O
.	O
for	O
inverted	B
index	I
compression	O
people	O
have	O
found	O
that	O
gamma	B
coding	O
seems	O
to	O
work	O
well	O
.	O
so	O
how	O
to	O
uncompress	O
inverted	B
index	I
we	O
just	O
talked	O
about	O
this	O
.	O
firstly	O
you	O
decode	O
those	O
encode	O
integers	O
.	O
and	O
we	O
just	O
i	O
think	O
discussed	O
how	O
we	O
decode	O
unary	O
coding	O
and	O
gamma	B
coding	O
.	O
so	O
i	O
won	O
t	O
repeat	O
.	O
what	O
about	O
the	O
document	O
ids	O
that	O
might	O
be	O
compressed	O
using	O
d-gap	O
well	O
we	O
re	O
going	O
to	O
do	O
sequential	O
decoding	O
.	O
so	O
suppose	O
the	O
encoded	O
idealist	O
is	O
x	O
x	O
x	O
et	O
cetera	O
.	O
we	O
first	O
decode	O
x	O
to	O
obtain	O
the	O
first	O
document	O
id	O
id	O
	O
.	O
then	O
we	O
will	O
decode	O
x	O
which	O
is	O
actually	O
the	O
difference	O
between	O
the	O
second	O
id	O
and	O
the	O
first	O
one	O
.	O
so	O
we	O
have	O
to	O
add	O
the	O
decoded	O
value	O
of	O
x	O
to	O
id	O
to	O
recover	O
the	O
value	O
of	O
the	O
the	O
id	O
at	O
this	O
secondary	O
position	O
right	O
.	O
so	O
this	O
is	O
where	O
you	O
can	O
see	O
the	O
advantage	O
of	O
converting	O
document	O
ids	O
into	O
integers	O
.	O
and	O
that	O
allows	O
us	O
to	O
do	O
this	O
kind	O
of	O
compression	O
and	O
we	O
just	O
repeat	O
until	O
we	O
decode	O
all	O
the	O
documents	O
.	O
every	O
time	O
we	O
use	O
the	O
document	O
id	O
in	O
the	O
previous	O
position	O
to	O
help	O
recover	O
the	O
document	O
id	O
in	O
the	O
next	O
position	O
.	O
	O
.	O
this	O
lecture	O
is	O
about	O
how	O
to	O
do	O
fast	O
research	O
by	O
using	O
inverted	B
index	I
.	O
in	O
this	O
lecture	O
we	O
are	O
going	O
to	O
continue	O
the	O
discussion	O
of	O
the	O
system	O
implementation	O
.	O
in	O
particular	O
we	O
re	O
going	O
to	O
talk	O
about	O
to	O
how	O
to	O
support	O
a	O
faster	O
search	O
by	O
using	O
inverted	B
index	I
.	O
so	O
let	O
s	O
think	O
about	O
what	O
a	O
general	O
scoring	O
function	O
might	O
look	O
like	O
.	O
now	O
of	O
curse	O
the	O
vector	B
space	I
model	I
is	O
a	O
special	O
case	O
of	O
this	O
.	O
but	O
we	O
can	O
imagine	O
many	O
other	O
retrieval	O
functions	O
of	O
the	O
same	O
form	O
.	O
so	O
the	O
form	O
of	O
this	O
function	O
is	O
as	O
follows	O
.	O
we	O
see	O
this	O
scoring	O
function	O
of	O
document	O
d	O
and	O
query	O
q	O
is	O
defined	O
as	O
first	O
a	O
function	O
of	O
f	O
a	O
that	O
s	O
adjustment	O
in	O
the	O
function	O
.	O
that	O
what	O
consider	O
two	O
factors	O
that	O
are	O
shown	O
here	O
at	O
the	O
end	O
f	O
sub	O
d	O
of	O
d	O
and	O
f	O
sub	O
q	O
of	O
q	O
.	O
these	O
are	O
adjustment	O
factors	O
of	O
a	O
document	O
and	O
query	O
so	O
they	O
re	O
at	O
the	O
level	O
of	O
document	O
and	O
query	O
.	O
so	O
and	O
then	O
inside	O
of	O
this	O
function	O
we	O
also	O
see	O
there	O
s	O
a	O
another	O
function	O
called	O
edge	O
.	O
so	O
this	O
is	O
the	O
main	O
part	O
of	O
the	O
scoring	O
function	O
and	O
these	O
as	O
i	O
just	O
said	O
of	O
the	O
scoring	O
factors	O
at	O
the	O
level	O
of	O
the	O
whole	O
document	O
and	O
the	O
query	O
.	O
for	O
example	O
document	O
and	O
this	O
aggregate	O
function	O
would	O
then	O
combine	O
all	O
these	O
.	O
now	O
inside	O
this	O
h	O
function	O
there	O
are	O
functions	O
that	O
would	O
compute	O
the	O
weights	O
of	O
the	O
contribution	O
of	O
a	O
matched	O
query	B
term	I
t	O
i	O
.	O
so	O
this	O
this	O
g	O
the	O
function	O
g	O
gives	O
us	O
the	O
weight	O
of	O
a	O
matched	O
query	B
term	I
t	O
i	O
in	O
document	O
d	O
.	O
and	O
this	O
h	O
function	O
with	O
that	O
aggregate	O
all	O
these	O
weights	O
so	O
it	O
were	O
for	O
example	O
take	O
a	O
sum	O
but	O
it	O
of	O
all	O
the	O
matched	O
query	O
in	O
that	O
terms	O
.	O
but	O
it	O
can	O
also	O
be	O
a	O
product	O
or	O
could	O
be	O
another	O
way	O
of	O
aggregate	O
them	O
.	O
and	O
then	O
finally	O
this	O
adjustment	O
function	O
would	O
then	O
consider	O
the	O
document	O
level	O
or	O
query	O
level	O
factors	O
through	O
further	O
adjuster	O
score	O
for	O
example	O
document	O
lens	O
	O
.	O
so	O
this	O
general	O
form	O
would	O
cover	O
many	O
state	O
of	O
original	O
functions	O
.	O
let	O
s	O
look	O
at	O
how	O
we	O
can	O
score	O
such	O
score	O
documents	O
with	O
such	O
a	O
function	O
using	O
inverted	B
index	I
.	O
so	O
here	O
s	O
the	O
general	O
algorithm	O
that	O
works	O
as	O
follows	O
.	O
first	O
these	O
these	O
query	O
level	O
and	O
document	O
level	O
factors	O
can	O
be	O
pre-computed	O
in	O
the	O
indexing	O
term	O
.	O
of	O
course	O
for	O
the	O
query	O
we	O
have	O
to	O
compute	O
it	O
as	O
a	O
query	B
term	I
.	O
but	O
for	O
document	O
for	O
example	O
document	O
can	O
be	O
pre-computed	O
.	O
and	O
then	O
we	O
maintain	O
a	O
score	O
accumulator	O
for	O
each	O
document	O
d	O
to	O
compute	O
the	O
h	O
.	O
and	O
h	O
is	O
aggregation	O
function	O
of	O
all	O
the	O
matching	O
query	B
terms	I
.	O
so	O
how	O
do	O
we	O
do	O
that	O
well	O
for	O
each	O
query	B
term	I
we	O
going	O
to	O
do	O
fetch	O
inverted	O
list	O
from	O
the	O
inverted	B
index	I
.	O
this	O
will	O
give	O
us	O
all	O
the	O
documents	O
that	O
match	O
this	O
query	B
term	I
and	O
that	O
includes	O
d	O
f	O
and	O
so	O
d	O
and	O
fn	O
.	O
so	O
each	O
pair	O
is	O
document	O
id	O
and	O
the	O
frequency	O
of	O
the	O
term	O
in	O
the	O
document	O
.	O
then	O
for	O
each	O
entry	O
d	O
sub	O
j	O
and	O
f	O
sub	O
j	O
a	O
particular	O
match	O
of	O
the	O
term	O
in	O
this	O
particular	O
document	O
d	O
sub	O
j	O
we	O
re	O
going	O
to	O
computer	O
the	O
function	O
g	O
.	O
that	O
would	O
give	O
us	O
something	O
like	O
a	O
t	O
of	O
i	O
ef	O
weights	O
of	O
this	O
term	O
.	O
so	O
we	O
re	O
computing	O
the	O
weight	O
contribution	O
of	O
matching	O
this	O
query	B
term	I
in	O
this	O
document	O
.	O
and	O
then	O
we	O
re	O
going	O
to	O
update	O
the	O
score	O
accumulator	O
for	O
this	O
document	O
.	O
and	O
this	O
would	O
allow	O
us	O
to	O
add	O
this	O
to	O
our	O
accumulator	O
that	O
would	O
incrementally	O
compute	O
function	O
h	O
.	O
so	O
this	O
is	O
basically	O
a	O
general	O
way	O
to	O
allow	O
sort	O
of	O
computer	O
all	O
functions	O
of	O
this	O
form	O
by	O
using	O
inverted	B
index	I
.	O
note	O
that	O
we	O
don	O
t	O
have	O
to	O
attach	O
any	O
document	O
that	O
that	O
didn	O
t	O
match	O
any	O
query	B
term	I
but	O
this	O
is	O
why	O
it	O
s	O
fast	O
.	O
we	O
only	O
need	O
to	O
process	O
the	O
documents	O
that	O
tap	O
that	O
match	O
at	O
least	O
one	O
query	B
term	I
.	O
in	O
the	O
end	O
then	O
we	O
re	O
going	O
to	O
adjust	O
the	O
score	O
to	O
compute	O
a	O
this	O
function	O
f	O
of	O
a	O
and	O
then	O
we	O
can	O
sort	O
.	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
the	O
specific	O
example	O
.	O
in	O
this	O
case	O
let	O
s	O
assume	O
the	O
scoring	O
function	O
s	O
a	O
very	O
simple	O
one	O
.	O
it	O
just	O
takes	O
us	O
sum	O
of	O
tf	B
the	O
rule	O
of	O
tf	B
the	O
count	O
of	O
of	O
term	O
in	O
the	O
document	O
.	O
now	O
this	O
simple	O
equation	O
with	O
the	O
help	O
showing	O
the	O
algorithm	O
clearly	O
.	O
it	O
s	O
very	O
easy	O
to	O
extend	O
the	O
the	O
computation	O
to	O
include	O
other	O
weights	O
like	O
the	O
transformation	O
of	O
tf	B
or	O
document	O
or	O
idf	B
weighting	O
.	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
specific	O
example	O
with	O
the	O
query	O
s	O
information	O
security	O
and	O
shows	O
some	O
entries	O
of	O
the	O
inverted	B
index	I
on	O
the	O
right	O
side	O
.	O
information	O
occurring	O
before	O
documents	O
and	O
the	O
frequencies	O
is	O
also	O
there	O
security	O
is	O
coding	O
three	O
documents	O
.	O
so	O
let	O
s	O
see	O
how	O
the	O
algorithm	O
works	O
all	O
right	O
so	O
first	O
we	O
iterate	O
all	O
the	O
query	B
terms	I
and	O
we	O
fetch	O
the	O
first	O
query	O
then	O
.	O
what	O
is	O
that	O
that	O
s	O
information	O
.	O
right	O
so	O
and	O
imagine	O
we	O
have	O
all	O
these	O
score	O
accumulators	O
to	O
score	O
score	O
the	O
score	O
the	O
scores	O
for	O
these	O
documents	O
.	O
we	O
can	O
imagine	O
there	O
will	O
be	O
allocated	O
but	O
then	O
they	O
will	O
only	O
be	O
allocated	O
as	O
needed	O
.	O
so	O
before	O
we	O
do	O
any	O
weighting	O
of	O
terms	O
we	O
don	O
t	O
even	O
need	O
a	O
score	O
accumulators	O
.	O
but	O
conceptual	O
we	O
have	O
these	O
score	O
accumulators	O
eventually	O
allocated	O
right	O
so	O
let	O
s	O
fetch	O
the	O
the	O
entries	O
from	O
the	O
inverted	O
list	O
for	O
information	O
first	O
that	O
s	O
the	O
first	O
one	O
.	O
so	O
these	O
score	O
accumulators	O
obviously	O
would	O
be	O
initialized	O
as	O
zeros	O
.	O
so	O
the	O
first	O
entry	O
is	O
d	O
and	O
is	O
occurrences	O
of	O
information	O
in	O
this	O
document	O
.	O
since	O
our	O
scoring	O
function	O
assume	O
that	O
the	O
score	O
is	O
just	O
a	O
sum	O
of	O
these	O
raw	O
counts	O
.	O
we	O
just	O
need	O
to	O
add	O
a	O
to	O
the	O
score	O
accumulator	O
to	O
account	O
for	O
the	O
increase	O
of	O
score	O
due	O
to	O
matching	O
this	O
term	O
information	O
a	O
document	O
d	O
	O
.	O
and	O
now	O
we	O
go	O
to	O
the	O
next	O
entry	O
.	O
that	O
s	O
d	O
and	O
and	O
then	O
we	O
ll	O
add	O
a	O
to	O
the	O
score	O
accumulator	O
of	O
d	O
	O
.	O
of	O
course	O
at	O
this	O
point	O
we	O
will	O
allocate	O
the	O
score	O
accumulator	O
as	O
needed	O
.	O
and	O
so	O
at	O
this	O
point	O
we	O
have	O
located	O
d	O
and	O
d	O
and	O
the	O
next	O
one	O
is	O
d	O
	O
.	O
and	O
we	O
add	O
or	O
we	O
locate	O
another	O
score	O
coming	O
in	O
the	O
spot	O
d	O
and	O
add	O
to	O
it	O
.	O
and	O
finally	O
the	O
d	O
gets	O
a	O
because	O
the	O
information	O
the	O
term	O
information	O
occurred	O
ti	O
in	O
five	O
times	O
in	O
this	O
document	O
.	O
okay	O
so	O
this	O
completes	O
the	O
processing	O
of	O
all	O
the	O
entries	O
in	O
the	O
inverted	B
index	I
for	O
information	O
.	O
it	O
s	O
processed	O
all	O
the	O
contributions	O
of	O
matching	O
information	O
in	O
this	O
four	O
documents	O
.	O
so	O
now	O
our	O
arrows	O
will	O
go	O
to	O
the	O
next	O
query	B
term	I
that	O
s	O
security	O
.	O
so	O
we	O
re	O
going	O
to	O
factor	O
all	O
the	O
inverted	B
index	I
entries	O
for	O
security	O
.	O
so	O
in	O
this	O
case	O
there	O
were	O
three	O
entries	O
.	O
and	O
we	O
re	O
going	O
to	O
go	O
through	O
each	O
of	O
them	O
.	O
the	O
first	O
is	O
d	O
and	O
	O
.	O
and	O
that	O
means	O
security	O
occurred	O
three	O
times	O
in	O
d	O
and	O
what	O
do	O
we	O
do	O
well	O
we	O
do	O
exactly	O
the	O
same	O
as	O
what	O
we	O
did	O
for	O
information	O
.	O
so	O
this	O
time	O
we	O
re	O
going	O
to	O
do	O
change	O
the	O
score	O
accumulating	O
d	O
sees	O
it	O
s	O
already	O
allocate	O
.	O
and	O
what	O
we	O
do	O
is	O
we	O
ll	O
add	O
to	O
the	O
existing	O
value	O
which	O
is	O
a	O
so	O
we	O
now	O
get	O
the	O
for	O
d	O
	O
.	O
d	O
sc	O
score	O
is	O
increased	O
because	O
of	O
the	O
match	O
both	O
information	O
and	O
the	O
security	O
.	O
go	O
to	O
the	O
next	O
step	O
entry	O
that	O
s	O
d	O
and	O
so	O
we	O
ve	O
updated	O
the	O
score	O
for	O
d	O
and	O
again	O
we	O
add	O
to	O
d	O
so	O
d	O
goes	O
from	O
to	O
	O
.	O
finally	O
we	O
process	O
d	O
and	O
	O
.	O
since	O
we	O
have	O
not	O
yet	O
equated	O
a	O
score	O
accumulator	O
d	O
to	O
d	O
at	O
this	O
point	O
we	O
allocate	O
one	O
and	O
we	O
re	O
going	O
to	O
add	O
to	O
it	O
.	O
so	O
those	O
scores	O
on	O
the	O
last	O
row	O
are	O
the	O
final	O
scores	O
for	O
these	O
documents	O
.	O
if	O
our	O
scoring	O
function	O
is	O
just	O
a	O
a	O
simple	O
sum	O
of	O
tf	B
values	O
.	O
now	O
what	O
if	O
we	O
actually	O
would	O
like	O
to	O
to	O
do	O
lands	O
normalization	O
.	O
well	O
we	O
can	O
do	O
the	O
normalization	O
at	O
this	O
point	O
for	O
each	O
document	O
.	O
so	O
to	O
summarize	O
this	O
all	O
right	O
so	O
you	O
can	O
see	O
we	O
first	O
processed	O
the	O
information	O
determine	O
query	B
term	I
information	O
and	O
we	O
process	O
all	O
the	O
entries	O
in	O
the	O
inverted	B
index	I
for	O
this	O
term	O
.	O
then	O
we	O
process	O
the	O
security	O
all	O
right	O
let	O
s	O
think	O
about	O
the	O
what	O
should	O
be	O
the	O
order	O
of	O
processing	O
here	O
when	O
we	O
consider	O
query	B
terms	I
it	O
might	O
make	O
difference	O
especially	O
if	O
we	O
don	O
t	O
want	O
to	O
keep	O
to	O
keep	O
all	O
the	O
score	O
accumulators	O
.	O
let	O
s	O
say	O
we	O
only	O
want	O
to	O
keep	O
the	O
most	O
promising	O
score	O
accumulators	O
.	O
what	O
do	O
you	O
think	O
it	O
would	O
be	O
a	O
good	O
order	O
to	O
go	O
through	O
would	O
you	O
go	O
would	O
you	O
process	O
a	O
common	O
term	O
first	O
or	O
would	O
you	O
process	O
a	O
rare	O
term	O
first	O
the	O
answer	O
is	O
we	O
should	O
go	O
through	O
we	O
should	O
process	O
the	O
rare	O
term	O
first	O
.	O
a	O
rare	O
term	O
will	O
match	O
fewer	O
documents	O
and	O
then	O
the	O
score	O
confusion	O
will	O
be	O
higher	O
because	O
the	O
idf	B
value	O
will	O
be	O
higher	O
and	O
and	O
then	O
it	O
allows	O
us	O
to	O
attach	O
the	O
most	O
diplomacy	O
documents	O
first	O
.	O
so	O
it	O
helps	O
pruning	O
some	O
non	O
promising	O
ones	O
if	O
we	O
don	O
t	O
need	O
so	O
many	O
documents	O
to	O
be	O
returned	O
to	O
the	O
user	O
.	O
and	O
so	O
those	O
are	O
heuristics	O
for	O
further	O
improving	O
the	O
accuracy	O
.	O
here	O
can	O
also	O
see	O
how	O
we	O
can	O
incorporate	O
the	O
idea	O
of	O
weighting	O
.	O
all	O
right	O
.	O
so	O
they	O
can	O
when	O
we	O
incorporated	O
a	O
one	O
way	O
process	O
each	O
query	B
term	I
.	O
when	O
we	O
fetch	O
in	O
word	O
index	O
we	O
can	O
fetch	O
the	O
document	B
frequency	I
and	O
then	O
we	O
can	O
compute	O
the	O
idf	B
.	O
or	O
maybe	O
perhapsidf	O
value	O
has	O
already	O
been	O
pre-computed	O
when	O
we	O
index	O
the	O
document	O
.	O
at	O
that	O
time	O
we	O
already	O
computed	O
the	O
idf	B
value	O
that	O
we	O
can	O
just	O
fetch	O
it	O
.	O
so	O
all	O
these	O
can	O
be	O
down	O
at	O
this	O
time	O
.	O
so	O
that	O
will	O
mean	O
one	O
will	O
process	O
all	O
the	O
entries	O
for	O
information	O
these	O
these	O
weights	O
would	O
be	O
adjusted	O
by	O
the	O
same	O
idf	B
which	O
is	O
idf	B
for	O
information	O
.	O
so	O
this	O
is	O
the	O
basic	O
idea	O
of	O
using	O
inverted	B
index	I
for	O
faster	O
search	O
and	O
works	O
well	O
for	O
all	O
kinds	O
of	O
formulas	O
that	O
are	O
of	O
the	O
general	O
form	O
and	O
this	O
generally	O
cov	O
the	O
general	O
form	O
covers	O
actually	O
most	O
state	O
of	O
the	O
art	O
retrieval	O
functions	O
.	O
so	O
there	O
are	O
some	O
tricks	O
to	O
further	O
improve	O
the	O
efficiency	O
some	O
general	O
mac	O
tech	O
techniques	O
include	O
caching	O
.	O
this	O
is	O
just	O
a	O
to	O
store	O
some	O
results	O
of	O
popular	O
query	O
s	O
so	O
that	O
next	O
time	O
when	O
you	O
see	O
the	O
same	O
query	O
you	O
simply	O
return	O
the	O
stored	O
results	O
.	O
similarly	O
you	O
can	O
also	O
score	O
the	O
missed	O
of	O
inverted	B
index	I
in	O
the	O
memory	O
for	O
popular	O
term	O
.	O
and	O
if	O
the	O
query	O
comes	O
popular	O
you	O
will	O
assume	O
it	O
will	O
fetch	O
the	O
inverted	B
index	I
for	O
the	O
same	O
term	O
again	O
.	O
so	O
keeping	O
that	O
in	O
the	O
memory	O
would	O
help	O
.	O
and	O
these	O
are	O
general	O
techniques	O
for	O
improving	O
efficiency	O
.	O
we	O
can	O
also	O
only	O
keep	O
the	O
most	O
promising	O
accumulators	O
because	O
a	O
user	O
generally	O
doesn	O
t	O
want	O
to	O
examine	O
so	O
many	O
documents	O
.	O
we	O
only	O
want	O
to	O
return	O
high	O
quality	O
subset	O
of	O
documents	O
that	O
likely	O
ranked	O
on	O
the	O
top	O
in	O
in	O
for	O
that	O
purpose	O
we	O
can	O
then	O
prune	O
the	O
accumulators	O
.	O
we	O
don	O
t	O
have	O
to	O
store	O
all	O
the	O
accumulators	O
.	O
at	O
some	O
point	O
we	O
just	O
keep	O
the	O
highest	O
value	O
accumulators	O
.	O
another	O
technique	O
is	O
to	O
do	O
parallel	O
processing	O
and	O
that	O
s	O
needed	O
for	O
really	O
processing	O
such	O
a	O
large	O
data	O
set	O
like	O
the	O
web	O
data	O
set	O
.	O
and	O
to	O
scale	O
up	O
to	O
the	O
web-scale	O
we	O
need	O
to	O
special	O
to	O
have	O
the	O
special	O
techniques	O
to	O
do	O
parallel	O
processing	O
and	O
to	O
distribute	O
the	O
storage	O
of	O
files	O
on	O
multiple	O
machines	O
.	O
so	O
here	O
as	O
a	O
here	O
is	O
a	O
list	O
of	O
some	O
text	B
retrieval	I
toolkits	O
.	O
it	O
s	O
it	O
s	O
not	O
a	O
complete	O
list	O
.	O
you	O
can	O
find	O
the	O
more	O
information	O
at	O
this	O
url	O
on	O
the	O
bottom	O
.	O
here	O
i	O
listed	O
four	O
here	O
lucene	O
is	O
one	O
of	O
the	O
most	O
popular	O
toolkit	O
that	O
can	O
support	O
a	O
lot	O
of	O
applications	O
.	O
and	O
it	O
has	O
very	O
nice	O
support	O
for	O
applications	O
.	O
you	O
can	O
use	O
it	O
to	O
build	O
a	O
search	B
engine	I
very	O
quickly	O
the	O
downside	O
is	O
that	O
it	O
s	O
not	O
that	O
easy	O
to	O
extend	O
it	O
and	O
the	O
algorithms	O
incremented	O
there	O
are	O
not	O
the	O
most	O
advanced	O
algorithms	O
.	O
lemur	O
or	O
indri	O
is	O
another	O
toolkit	O
that	O
that	O
does	O
not	O
have	O
such	O
a	O
nice	O
support	O
application	O
as	O
lucene	O
.	O
but	O
it	O
has	O
many	O
advanced	O
search	O
algorithms	O
.	O
and	O
it	O
s	O
also	O
easy	O
to	O
extend	O
.	O
terrier	O
is	O
yet	O
another	O
toolkit	O
that	O
also	O
has	O
good	O
support	O
for	O
quotation	O
capability	O
and	O
some	O
advanced	O
algorithms	O
.	O
so	O
that	O
s	O
maybe	O
in	O
between	O
lemur	O
or	O
lucene	O
or	O
maybe	O
rather	O
combining	O
the	O
strands	O
of	O
both	O
so	O
that	O
s	O
also	O
useful	O
toolkit	O
.	O
meta	O
is	O
the	O
toolkit	O
that	O
we	O
ll	O
use	O
for	O
the	O
programming	O
assignment	O
and	O
this	O
is	O
a	O
new	O
toolkit	O
that	O
has	O
a	O
combination	O
of	O
both	O
text	B
retrieval	I
algorithms	O
and	O
text	O
mining	O
algorithms	O
.	O
and	O
so	O
toolkit	O
models	O
are	O
implement	O
they	O
are	O
there	O
are	O
a	O
number	O
of	O
text	O
analysis	O
algorithms	O
implemented	O
in	O
the	O
toolkit	O
as	O
well	O
as	O
basic	O
research	O
algorithms	O
.	O
so	O
to	O
summarize	O
all	O
the	O
discussion	O
about	O
the	O
system	O
implementation	O
here	O
are	O
the	O
major	O
take	O
away	O
points	O
.	O
inverted	B
index	I
is	O
the	O
primary	O
data	O
structure	O
for	O
supporting	O
a	O
search	B
engine	I
.	O
that	O
s	O
the	O
key	O
to	O
enable	O
faster	O
response	O
to	O
a	O
user	O
s	O
query	O
.	O
and	O
the	O
basic	O
idea	O
is	O
process	O
that	O
pre-process	O
the	O
data	O
as	O
much	O
as	O
we	O
can	O
and	O
we	O
want	O
to	O
do	O
compression	O
when	O
appropriate	O
.	O
so	O
that	O
we	O
can	O
save	O
disk	O
space	O
and	O
can	O
speed	O
up	O
io	O
and	O
processing	O
of	O
the	O
inverted	B
index	I
in	O
general	O
.	O
we	O
ll	O
talk	O
about	O
how	O
we	O
will	O
construct	O
the	O
inverted	B
index	I
when	O
the	O
data	O
can	O
fit	O
into	O
the	O
memory	O
.	O
and	O
then	O
we	O
talk	O
about	O
faster	O
search	O
using	O
inverted	B
index	I
basically	O
to	O
exploit	O
the	O
inverted	B
index	I
to	O
accumulate	O
scores	O
for	O
documents	O
matching	O
a	O
query	B
term	I
.	O
and	O
we	O
exploit	O
zipf	O
s	O
law	O
avoid	O
touching	O
many	O
documents	O
that	O
don	O
t	O
match	O
any	O
query	B
term	I
.	O
and	O
this	O
algorithm	O
can	O
can	O
support	O
a	O
wide	O
range	O
of	O
ranking	B
algorithms	I
.	O
so	O
these	O
basic	O
techniques	O
have	O
mm	O
have	O
great	O
potential	O
for	O
further	O
scanning	O
output	O
using	O
distribution	O
to	O
withstand	O
parallel	O
processing	O
and	O
the	O
caching	O
.	O
here	O
are	O
two	O
additional	O
readings	O
that	O
you	O
can	O
take	O
a	O
look	O
at	O
if	O
you	O
have	O
time	O
and	O
are	O
interested	O
in	O
learning	O
more	O
about	O
this	O
.	O
the	O
first	O
one	O
is	O
a	O
classic	O
textbook	O
on	O
the	O
scare	O
the	O
efficiency	O
of	O
inverted	B
index	I
and	O
the	O
compression	O
techniques	O
and	O
how	O
to	O
in	O
general	O
build	O
a	O
efficient	O
search	B
engine	I
in	O
terms	O
of	O
the	O
space	O
overhead	O
and	O
speed	O
.	O
the	O
second	O
one	O
is	O
a	O
newer	O
textbook	O
that	O
has	O
a	O
nice	O
discussion	O
of	O
implementing	O
and	O
evaluating	O
search	B
engines	I
.	O
this	O
lecture	O
is	O
about	O
evaluation	B
of	I
text	B
retrieval	B
systems	I
.	O
in	O
the	O
previous	O
lectures	O
we	O
have	O
talked	O
about	O
a	O
number	O
of	O
text	B
retrieval	B
methods	I
.	O
different	O
kinds	O
of	O
ranking	B
functions	I
.	O
but	O
how	O
do	O
we	O
know	O
which	O
one	O
works	O
the	O
best	O
in	O
order	O
to	O
answer	O
this	O
question	O
we	O
have	O
to	O
compare	O
them	O
and	O
that	O
means	O
we	O
ll	O
have	O
to	O
evaluate	O
these	O
retrieval	B
methods	I
.	O
so	O
this	O
is	O
the	O
main	O
topic	O
of	O
this	O
lecture	O
.	O
first	O
let	O
s	O
think	O
about	O
why	O
do	O
we	O
have	O
to	O
do	O
evaluation	O
i	O
already	O
gave	O
one	O
reason	O
.	O
and	O
that	O
is	O
we	O
have	O
to	O
use	O
evaluation	O
to	O
figure	O
out	O
which	O
retrieval	B
method	I
works	O
better	O
.	O
now	O
this	O
is	O
very	O
important	O
for	O
advancing	O
our	O
knowledge	O
.	O
otherwise	O
we	O
wouldn	O
t	O
know	O
whether	O
a	O
new	O
idea	O
works	O
better	O
than	O
old	O
idea	O
.	O
in	O
the	O
beginning	O
of	O
this	O
course	O
we	O
talked	O
about	O
the	O
the	O
problem	O
of	O
text	B
retrieval	I
we	O
compare	O
it	O
with	O
database	O
retrieval	O
.	O
there	O
we	O
mentioned	O
that	O
text	B
retrieval	I
is	O
imperative	O
to	O
find	O
the	O
problem	O
.	O
so	O
evaluation	O
must	O
rely	O
on	O
users	O
which	O
system	O
works	O
better	O
that	O
would	O
have	O
to	O
be	O
judged	O
by	O
our	O
users	O
.	O
so	O
this	O
becomes	O
very	O
challenging	O
problem	O
.	O
because	O
how	O
can	O
we	O
get	O
users	O
involved	O
in	O
in	O
matters	O
and	O
how	O
can	O
we	O
draw	O
a	O
fair	O
comparison	O
of	O
different	O
methods	O
.	O
so	O
just	O
go	O
back	O
to	O
the	O
reasons	O
for	O
evaluation	O
.	O
i	O
listed	O
two	O
reasons	O
here	O
.	O
the	O
second	O
reason	O
is	O
basically	O
what	O
i	O
just	O
said	O
but	O
there	O
is	O
also	O
another	O
reason	O
which	O
is	O
to	O
assess	O
the	O
actual	O
utility	O
of	O
a	O
test	O
regional	O
system	O
.	O
now	O
imagine	O
you	O
re	O
building	O
your	O
own	O
applications	O
.	O
would	O
be	O
interested	O
in	O
knowing	O
how	O
well	O
your	O
search	B
engine	I
works	O
for	O
your	O
users	O
.	O
so	O
in	O
this	O
case	O
measures	O
must	O
reflect	O
the	O
utility	O
to	O
the	O
actual	O
users	O
in	O
the	O
the	O
real	O
application	O
.	O
and	O
typically	O
this	O
has	O
been	O
done	O
by	O
using	O
user	O
studies	O
and	O
using	O
the	O
real	O
search	B
engine	I
.	O
in	O
the	O
second	O
case	O
or	O
for	O
the	O
second	O
reason	O
the	O
measures	O
actually	O
all	O
need	O
to	O
be	O
correlated	O
with	O
the	O
utility	O
to	O
actual	O
users	O
.	O
thus	O
they	O
don	O
t	O
have	O
to	O
accurately	O
reflect	O
the	O
the	O
exact	O
utility	O
to	O
users	O
.	O
so	O
the	O
measure	O
only	O
needs	O
to	O
be	O
good	O
enough	O
to	O
tell	O
which	O
method	O
works	O
better	O
.	O
and	O
this	O
is	O
usually	O
done	O
through	O
test	B
collection	I
.	O
and	O
this	O
is	O
the	O
main	O
idea	O
that	O
we	O
ll	O
be	O
talking	O
about	O
in	O
this	O
course	O
.	O
this	O
has	O
been	O
very	O
important	O
for	O
comparing	O
different	O
algorithms	O
and	O
for	O
improving	O
search	B
engines	I
systems	O
in	O
general	O
.	O
so	O
next	O
we	O
will	O
talk	O
about	O
what	O
to	O
measure	O
.	O
there	O
are	O
many	O
aspects	O
of	O
a	O
search	B
engine	I
we	O
can	O
measure	O
we	O
can	O
evaluate	O
and	O
here	O
i	O
list	O
the	O
three	O
major	O
aspects	O
.	O
one	O
is	O
effectiveness	O
or	O
accuracy	O
how	O
accurate	O
are	O
the	O
search	O
results	O
in	O
this	O
case	O
we	O
re	O
measuring	O
a	O
system	O
s	O
capability	O
of	O
ranking	O
relevant	B
documents	I
on	O
top	O
of	O
non	O
relevant	O
ones	O
.	O
the	O
second	O
is	O
efficiency	O
.	O
how	O
quickly	O
can	O
a	O
user	O
get	O
the	O
results	O
how	O
much	O
computing	O
resources	O
are	O
needed	O
to	O
answer	O
a	O
query	O
so	O
in	O
this	O
case	O
we	O
need	O
to	O
measure	O
the	O
space	O
and	O
time	O
overhead	O
of	O
the	O
system	O
.	O
the	O
third	O
aspect	O
is	O
usability	O
.	O
basically	O
the	O
question	O
is	O
how	O
useful	O
is	O
the	O
system	O
for	O
real	O
user	O
tasks	O
here	O
obviously	O
interfaces	O
and	O
many	O
other	O
things	O
are	O
also	O
important	O
and	O
we	O
typically	O
would	O
have	O
to	O
do	O
user	O
studies	O
.	O
now	O
in	O
this	O
course	O
we	O
re	O
going	O
to	O
talk	O
more	O
mostly	O
about	O
the	O
effectiveness	O
and	O
accuracy	O
measures	O
because	O
the	O
efficiency	O
and	O
usability	O
dimensions	O
are	O
not	O
really	O
unique	O
to	O
search	B
engines	I
and	O
so	O
they	O
are	O
needed	O
for	O
evaluating	O
any	O
other	O
software	O
systems	O
.	O
and	O
there	O
is	O
also	O
good	O
coverage	O
of	O
such	O
materials	O
in	O
other	O
courses	O
.	O
but	O
how	O
to	O
evaluate	O
a	O
search	B
engine	I
is	O
quite	O
you	O
know	O
accuracy	O
is	O
something	O
you	O
need	O
to	O
text	B
retrieval	I
and	O
we	O
re	O
going	O
to	O
talk	O
a	O
lot	O
about	O
this	O
.	O
the	O
main	O
idea	O
that	O
people	O
have	O
proposed	O
before	O
using	O
a	O
attitude	O
evaluate	O
a	O
text	B
retrieval	I
algorithm	O
is	O
called	O
the	O
cranfield	B
evaluation	B
methodology	I
.	O
this	O
one	O
actually	O
was	O
developed	O
long	O
time	O
ago	O
developed	O
in	O
the	O
s	O
.	O
it	O
s	O
a	O
methodology	O
for	O
laboratory	O
test	O
of	O
system	O
components	O
it	O
s	O
actually	O
a	O
methodology	O
that	O
has	O
been	O
very	O
useful	O
not	O
just	O
for	O
search	B
engine	I
evaluation	O
.	O
but	O
also	O
for	O
evaluating	O
virtually	O
all	O
kinds	O
of	O
empirical	O
tasks	O
.	O
and	O
for	O
example	O
in	O
processing	O
or	O
in	O
other	O
fields	O
where	O
the	O
problem	O
is	O
empirically	O
defined	O
we	O
typically	O
would	O
need	O
to	O
use	O
to	O
use	O
such	O
a	O
methodology	O
.	O
and	O
today	O
was	O
the	O
big	O
data	O
challenge	O
with	O
the	O
use	O
of	O
machine	B
learning	I
every	O
where	O
.	O
we	O
general	O
this	O
methodology	O
has	O
been	O
very	O
popular	O
but	O
it	O
was	O
first	O
developed	O
for	O
search	B
engine	I
application	O
in	O
the	O
s	O
.	O
so	O
the	O
basic	O
idea	O
of	O
this	O
approach	O
is	O
it	O
ll	O
build	O
a	O
reusable	O
test	O
collections	O
and	O
define	O
measures	O
.	O
once	O
such	O
a	O
test	B
collection	I
is	O
build	O
it	O
can	O
be	O
used	O
again	O
and	O
again	O
to	O
test	O
the	O
different	O
algorithms	O
.	O
and	O
we	O
re	O
going	O
to	O
define	O
measures	O
that	O
would	O
allow	O
you	O
to	O
quantify	O
performance	O
of	O
a	O
system	O
or	O
an	O
an	O
algorithm	O
.	O
so	O
how	O
exactly	O
would	O
this	O
work	O
well	O
we	O
re	O
going	O
to	O
do	O
have	O
assembled	O
collection	O
of	O
documents	O
and	O
this	O
is	O
just	O
similar	O
to	O
real	O
document	O
collection	O
in	O
your	O
search	O
application	O
.	O
we	O
can	O
also	O
have	O
a	O
sample	O
set	O
of	O
queries	O
or	O
topics	O
.	O
this	O
is	O
to	O
simulate	O
the	O
user	O
s	O
queries	O
.	O
then	O
we	O
ll	O
have	O
to	O
have	O
relevance	B
judgments	I
.	O
these	O
are	O
judgments	O
of	O
which	O
documents	O
should	O
be	O
returned	O
for	O
which	O
queries	O
.	O
ideally	O
they	O
have	O
to	O
made	O
by	O
users	O
who	O
formulated	O
the	O
queries	O
because	O
those	O
are	O
the	O
people	O
that	O
know	O
exactly	O
what	O
documents	O
would	O
be	O
used	O
for	O
.	O
and	O
then	O
finally	O
we	O
have	O
to	O
have	O
measures	O
to	O
quantify	O
how	O
well	O
a	O
system	O
s	O
result	O
matches	O
the	O
ideal	O
ranked	O
list	O
.	O
that	O
would	O
be	O
constructed	O
and	O
based	O
on	O
users	O
relevant	O
judgements	O
.	O
so	O
this	O
methodology	O
is	O
very	O
useful	O
for	O
starting	O
retrieval	O
algorithms	O
because	O
the	O
test	O
can	O
actually	O
can	O
be	O
reused	O
many	O
times	O
.	O
and	O
it	O
will	O
also	O
provide	O
a	O
fair	O
comparison	O
for	O
all	O
the	O
methods	O
.	O
we	O
have	O
the	O
same	O
criteria	O
same	O
data	O
set	O
to	O
use	O
and	O
to	O
compare	O
different	O
algorithms	O
.	O
this	O
allows	O
us	O
to	O
compare	O
a	O
new	O
algorithm	O
with	O
an	O
old	O
algorithm	O
that	O
was	O
the	O
method	O
of	O
many	O
years	O
ago	O
.	O
by	O
using	O
the	O
same	O
standard	O
.	O
so	O
this	O
is	O
the	O
illustration	O
of	O
how	O
this	O
works	O
so	O
as	O
i	O
said	O
we	O
need	O
a	O
queries	O
that	O
are	O
shown	O
here	O
.	O
we	O
have	O
q	O
q	O
et	O
cetera	O
.	O
we	O
also	O
need	O
a	O
documents	O
and	O
that	O
s	O
called	O
the	O
document	O
collection	O
and	O
on	O
the	O
right	O
side	O
you	O
see	O
we	O
need	O
relevance	B
judgment	I
.	O
these	O
are	O
basically	O
the	O
binary	O
judgments	O
of	O
documents	O
with	O
respect	O
to	O
a	O
query	O
.	O
so	O
for	O
example	O
d	O
is	O
judged	O
as	O
being	O
relevant	O
to	O
q	O
d	O
is	O
judged	O
as	O
being	O
relevant	O
as	O
well	O
.	O
and	O
d	O
is	O
judged	O
as	O
non	O
relevant	O
in	O
the	O
two	O
q	O
et	O
cetera	O
.	O
these	O
would	O
be	O
created	O
by	O
users	O
.	O
once	O
we	O
have	O
these	O
and	O
we	O
basically	O
have	O
a	O
test	O
correction	O
and	O
then	O
if	O
you	O
have	O
two	O
systems	O
you	O
want	O
to	O
compare	O
them	O
.	O
then	O
you	O
can	O
just	O
run	O
each	O
system	O
on	O
these	O
queries	O
and	O
documents	O
and	O
each	O
system	O
will	O
then	O
return	O
results	O
.	O
let	O
s	O
say	O
if	O
the	O
query	O
is	O
q	O
and	O
then	O
we	O
would	O
have	O
the	O
results	O
here	O
here	O
i	O
show	O
r	O
sub	O
a	O
as	O
results	O
from	O
system	O
a	O
.	O
so	O
this	O
is	O
remember	O
we	O
talked	O
about	O
task	O
of	O
computing	O
approximation	O
of	O
the	O
relevant	O
document	O
setter	O
.	O
so	O
a	O
is	O
the	O
system	O
a	O
s	O
approximation	O
here	O
and	O
also	O
b	O
is	O
system	O
b	O
s	O
approximation	O
of	O
relevant	B
documents	I
.	O
now	O
let	O
s	O
take	O
a	O
look	O
at	O
these	O
results	O
.	O
so	O
which	O
is	O
better	O
now	O
imagine	O
for	O
a	O
user	O
which	O
one	O
would	O
you	O
like	O
all	O
right	O
lets	O
take	O
a	O
look	O
at	O
both	O
results	O
.	O
and	O
there	O
are	O
some	O
differences	O
and	O
there	O
are	O
some	O
documents	O
that	O
are	O
return	O
to	O
both	O
systems	O
.	O
but	O
if	O
you	O
look	O
at	O
the	O
results	O
you	O
will	O
feel	O
that	O
well	O
maybe	O
an	O
a	O
is	O
better	O
in	O
the	O
sense	O
that	O
we	O
don	O
t	O
have	O
many	O
number	O
in	O
documents	O
.	O
and	O
among	O
the	O
three	O
documents	O
returned	O
the	O
two	O
of	O
them	O
are	O
relevant	O
so	O
that	O
s	O
good	O
it	O
s	O
precise	O
.	O
on	O
the	O
other	O
hand	O
can	O
also	O
say	O
maybe	O
b	O
is	O
better	O
because	O
we	O
ve	O
got	O
more	O
relevant	B
documents	I
we	O
ve	O
got	O
three	O
instead	O
of	O
two	O
.	O
so	O
which	O
one	O
is	O
better	O
and	O
how	O
do	O
we	O
quantify	O
this	O
well	O
obviously	O
this	O
question	O
highly	O
depends	O
on	O
a	O
user	O
s	O
task	O
.	O
and	O
it	O
depends	O
on	O
users	O
as	O
well	O
.	O
you	O
might	O
be	O
able	O
to	O
imagine	O
for	O
some	O
users	O
may	O
be	O
system	O
made	O
is	O
better	O
.	O
if	O
the	O
user	O
is	O
not	O
interested	O
in	O
getting	O
all	O
the	O
relevant	B
documents	I
right	O
in	O
this	O
case	O
this	O
is	O
the	O
user	O
doesn	O
t	O
have	O
to	O
read	O
.	O
user	O
would	O
see	O
most	O
relevant	B
documents	I
.	O
on	O
the	O
other	O
hand	O
on	O
one	O
count	O
imagine	O
user	O
might	O
need	O
to	O
have	O
as	O
many	O
relevant	B
documents	I
as	O
possible	O
for	O
example	O
taking	O
a	O
literature	O
survey	O
.	O
you	O
might	O
be	O
in	O
the	O
second	O
category	O
and	O
then	O
you	O
might	O
find	O
that	O
system	O
b	O
s	O
better	O
.	O
so	O
in	O
either	O
case	O
we	O
ll	O
have	O
to	O
also	O
define	O
measures	O
that	O
would	O
quantify	O
them	O
.	O
and	O
we	O
might	O
need	O
to	O
define	O
multiple	O
measures	O
because	O
users	O
have	O
different	O
perspectives	O
of	O
looking	O
at	O
results	O
.	O
this	O
lecture	O
is	O
about	O
the	O
the	O
basic	O
measures	O
for	O
evaluation	O
of	O
text	O
original	O
systems	O
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
discuss	O
how	O
we	O
design	O
basic	O
measures	O
to	O
quantitatively	O
compare	O
two	O
original	O
systems	O
.	O
this	O
is	O
a	O
slide	O
that	O
you	O
have	O
seen	O
earlier	O
in	O
the	O
lecture	O
where	O
we	O
talk	O
about	O
the	O
grand	O
evaluation	B
methodology	I
.	O
we	O
can	O
have	O
a	O
test	B
collection	I
that	O
consists	O
of	O
queries	O
documents	O
and	O
relevance	O
judgements	O
.	O
we	O
can	O
then	O
run	O
two	O
systems	O
on	O
these	O
da	O
data	O
sets	O
to	O
quantitatively	O
evaluate	O
your	O
performance	O
.	O
and	O
we	O
raised	O
to	O
the	O
question	O
about	O
which	O
settles	O
results	O
is	O
better	O
is	O
system	O
a	O
better	O
or	O
system	O
b	O
better	O
so	O
let	O
s	O
now	O
talk	O
about	O
how	O
to	O
actually	O
quantify	O
their	O
performance	O
.	O
suppose	O
we	O
have	O
a	O
total	O
of	O
of	O
random	O
documents	O
in	O
the	O
current	O
folder	O
for	O
this	O
query	O
.	O
now	O
the	O
relevance	O
judgements	O
shown	O
on	O
the	O
right	O
did	O
not	O
include	O
all	O
the	O
ten	O
obviously	O
.	O
and	O
we	O
have	O
only	O
seen	O
three	O
rendered	O
documents	O
there	O
but	O
we	O
can	O
imagine	O
there	O
are	O
other	O
random	O
documents	O
in	O
judging	O
for	O
this	O
query	O
.	O
so	O
now	O
intuitively	O
we	O
thought	O
that	O
system	O
a	O
is	O
better	O
because	O
it	O
did	O
not	O
have	O
much	O
noise	O
.	O
and	O
in	O
particular	O
we	O
have	O
seen	O
amount	O
of	O
three	O
results	O
two	O
of	O
them	O
are	O
relevant	O
but	O
in	O
system	O
b	O
we	O
have	O
five	O
results	O
and	O
only	O
three	O
of	O
them	O
are	O
relevant	O
.	O
so	O
intuitively	O
it	O
looks	O
like	O
system	O
a	O
is	O
more	O
accurate	O
.	O
and	O
this	O
can	O
be	O
captured	O
by	O
a	O
matching	O
order	O
precision	B
.	O
where	O
we	O
simply	O
compute	O
to	O
what	O
extent	O
all	O
the	O
retrieval	O
results	O
are	O
relevant	O
.	O
if	O
you	O
have	O
precision	B
that	O
would	O
mean	O
all	O
the	O
retrieval	O
documents	O
are	O
relevant	O
.	O
so	O
in	O
this	O
case	O
the	O
system	O
a	O
has	O
a	O
precision	B
of	O
two	O
out	O
of	O
three	O
.	O
system	O
b	O
as	O
three	O
over	O
five	O
.	O
and	O
this	O
shows	O
that	O
system	O
a	O
is	O
better	O
by	O
precision	B
.	O
but	O
we	O
also	O
talked	O
about	O
system	O
b	O
might	O
be	O
preferred	O
by	O
some	O
other	O
users	O
hold	O
like	O
to	O
retrieve	O
as	O
many	O
relevant	B
documents	I
as	O
possible	O
.	O
so	O
in	O
that	O
case	O
we	O
have	O
to	O
compare	O
the	O
number	O
of	O
relevant	B
documents	I
that	O
retrieve	O
.	O
and	O
there	O
is	O
an	O
other	O
measure	O
called	O
a	O
recall	B
.	O
this	O
measures	O
the	O
completeness	O
of	O
coverage	O
of	O
relevant	B
documents	I
in	O
your	O
retriever	O
result	O
.	O
so	O
we	O
just	O
assume	O
that	O
there	O
are	O
ten	O
relevant	B
documents	I
in	O
the	O
collection	O
.	O
and	O
here	O
we	O
ve	O
got	O
two	O
of	O
them	O
in	O
system	O
a	O
so	O
the	O
recall	B
is	O
two	O
out	O
of	O
ten	O
.	O
where	O
as	O
system	O
b	O
has	O
got	O
a	O
three	O
so	O
it	O
s	O
a	O
three	O
out	O
of	O
ten	O
.	O
now	O
we	O
can	O
see	O
by	O
recall	B
system	O
b	O
is	O
better	O
and	O
these	O
two	O
measures	O
turned	O
out	O
to	O
be	O
the	O
very	O
basic	O
measures	O
for	O
evaluating	O
search	B
engine	I
.	O
and	O
they	O
are	O
very	O
important	O
because	O
they	O
are	O
also	O
widely	O
used	O
in	O
many	O
other	O
testing	O
variation	O
problems	O
.	O
for	O
example	O
if	O
you	O
look	O
at	O
the	O
applications	O
of	O
machine	B
learning	I
you	O
tend	O
to	O
see	O
precision	B
recall	B
numbers	O
being	O
reported	O
for	O
all	O
kinds	O
of	O
tasks	O
.	O
okay	O
so	O
now	O
let	O
s	O
define	O
these	O
two	O
measures	O
more	O
precisely	O
and	O
these	O
measures	O
are	O
to	O
evaluate	O
a	O
set	O
of	O
retrieval	O
documents	O
.	O
so	O
that	O
means	O
we	O
are	O
considering	O
that	O
approximation	O
of	O
a	O
set	O
of	O
relevant	B
documents	I
.	O
we	O
can	O
distinguish	O
it	O
four	O
cases	O
depending	O
on	O
the	O
situation	O
of	O
a	O
document	O
.	O
a	O
document	O
that	O
can	O
be	O
retrieved	O
or	O
not	O
retrieved	O
right	O
because	O
we	O
re	O
talking	O
about	O
the	O
set	O
of	O
result	O
.	O
the	O
document	O
can	O
be	O
also	O
relevant	O
or	O
not	O
relevant	O
depending	O
on	O
whether	O
the	O
user	O
thinks	O
this	O
is	O
a	O
useful	O
document	O
.	O
so	O
we	O
can	O
now	O
have	O
counts	O
of	O
documents	O
in	O
each	O
of	O
the	O
four	O
categories	O
.	O
we	O
can	O
have	O
a	O
to	O
represent	O
the	O
number	O
of	O
documents	O
that	O
are	O
retrieved	O
and	O
relevant	O
b	O
for	O
documents	O
that	O
are	O
not	O
retrieved	O
but	O
relevant	O
etc	O
.	O
now	O
with	O
this	O
table	O
then	O
we	O
have	O
defined	O
precision	B
.	O
as	O
the	O
ratio	O
of	O
the	O
relevant	O
retriever	O
documents	O
a	O
to	O
the	O
total	O
number	O
of	O
retriever	O
documents	O
.	O
so	O
this	O
is	O
just	O
you	O
know	O
a	O
divided	O
by	O
the	O
sum	O
of	O
a	O
and	O
c	O
.	O
the	O
sum	O
of	O
this	O
column	O
.	O
signal	O
recall	B
is	O
defined	O
by	O
dividing	O
a	O
by	O
the	O
sum	O
of	O
a	O
and	O
b	O
.	O
so	O
that	O
s	O
again	O
to	O
divide	O
a	O
by	O
the	O
sum	O
of	O
the	O
rule	O
instead	O
of	O
the	O
column	O
.	O
all	O
right	O
so	O
we	O
going	O
to	O
see	O
precision	B
and	I
recall	B
is	O
all	O
focused	O
on	O
looking	O
at	O
the	O
a	O
that	O
s	O
the	O
number	O
of	O
retrieval	O
relevant	B
documents	I
but	O
we	O
re	O
going	O
to	O
use	O
different	O
denominators	O
.	O
okay	O
so	O
what	O
would	O
be	O
an	O
ideal	O
result	O
well	O
you	O
can	O
able	O
to	O
see	O
in	O
ideal	O
case	O
we	O
have	O
precision	B
and	I
recall	B
all	O
to	O
be	O
	O
.	O
that	O
means	O
we	O
have	O
got	O
of	O
all	O
the	O
random	O
documents	O
in	O
our	O
results	O
.	O
and	O
all	O
the	O
results	O
that	O
we	O
return	O
are	O
relevant	O
.	O
there	O
s	O
no	O
single	O
not	O
relevant	O
document	O
returned	O
.	O
the	O
reality	O
however	O
high	O
recall	B
tends	O
to	O
be	O
associated	O
with	O
low	O
precision	B
and	O
you	O
can	O
imagine	O
why	O
that	O
is	O
the	O
case	O
.	O
as	O
you	O
go	O
down	O
the	O
distant	O
to	O
try	O
to	O
get	O
as	O
many	O
relevant	O
actions	O
as	O
possible	O
.	O
you	O
tend	O
to	O
in	O
time	O
a	O
lot	O
of	O
non	O
relevant	B
documents	I
so	O
the	O
precision	B
goes	O
down	O
.	O
look	O
at	O
this	O
set	O
can	O
also	O
be	O
defined	O
by	O
a	O
cutoff	O
in	O
a	O
ranked	O
list	O
.	O
that	O
s	O
why	O
although	O
these	O
two	O
measures	O
are	O
defined	O
for	O
a	O
set	O
of	O
retrieved	B
documents	I
they	O
are	O
actually	O
very	O
useful	O
for	O
evaluating	O
a	O
ranked	O
list	O
.	O
they	O
are	O
the	O
fundamental	O
measures	O
in	O
tension	O
retrieval	O
and	O
many	O
other	O
tasks	O
.	O
we	O
often	O
are	O
interested	O
in	O
to	O
the	O
precision	B
up	O
to	O
ten	O
documents	O
for	O
web	B
search	I
.	O
this	O
means	O
we	O
look	O
at	O
the	O
how	O
many	O
documents	O
among	O
the	O
top	O
results	O
are	O
actually	O
relevant	O
.	O
now	O
this	O
is	O
a	O
very	O
meaningful	O
measure	O
because	O
it	O
tells	O
us	O
how	O
many	O
relevant	B
documents	I
a	O
user	O
can	O
expect	O
to	O
see	O
.	O
on	O
the	O
first	O
page	O
of	O
search	O
results	O
where	O
they	O
typically	O
show	O
ten	O
results	O
.	O
so	O
precision	B
and	I
recall	B
are	O
the	O
basic	O
measures	O
and	O
we	O
need	O
to	O
use	O
them	O
to	O
further	O
evaluate	O
a	O
search	B
engine	I
but	O
they	O
are	O
the	O
building	O
blocks	O
really	O
.	O
we	O
just	O
to	O
say	O
that	O
there	O
tends	O
to	O
be	O
a	O
trade	O
off	O
between	O
precision	B
and	I
recall	B
.	O
so	O
naturally	O
it	O
would	O
be	O
interesting	O
to	O
combine	O
them	O
and	O
here	O
s	O
one	O
measure	O
that	O
s	O
often	O
used	O
called	O
f	B
measure	I
.	O
and	O
it	O
s	O
harmonic	O
mean	O
of	O
precision	B
and	I
recall	B
it	O
s	O
defined	O
on	O
this	O
slide	O
.	O
so	O
you	O
can	O
see	O
it	O
first	O
computed	O
inverse	O
of	O
r	O
and	O
p	O
here	O
and	O
then	O
it	O
would	O
be	O
interpreted	O
to	O
by	O
using	O
a	O
co	O
coefficients	O
.	O
depending	O
on	O
the	O
parameter	O
beta	B
and	O
after	O
some	O
transformation	O
we	O
can	O
easily	O
see	O
it	O
would	O
be	O
of	O
this	O
form	O
.	O
and	O
in	O
many	O
cases	O
it	O
s	O
just	O
a	O
combination	O
of	O
precision	B
and	I
recall	B
.	O
and	O
and	O
beta	B
is	O
a	O
parameter	O
that	O
s	O
often	O
set	O
to	O
one	O
.	O
it	O
can	O
control	O
the	O
emphasis	O
on	O
precision	B
or	O
recall	B
.	O
when	O
we	O
set	O
beta	B
to	O
one	O
we	O
end	O
up	O
by	O
having	O
a	O
special	O
case	O
of	O
f	B
measure	I
often	O
called	O
f	O
	O
.	O
this	O
is	O
a	O
popular	O
measure	O
that	O
is	O
often	O
used	O
as	O
a	O
combined	O
precision	B
and	I
recall	B
.	O
and	O
the	O
formula	O
looks	O
very	O
simple	O
it	O
s	O
just	O
this	O
here	O
.	O
now	O
it	O
s	O
easy	O
to	O
see	O
that	O
if	O
you	O
have	O
a	O
larger	O
precision	B
or	O
larger	O
recall	B
than	O
f	B
measure	I
would	O
be	O
high	O
.	O
but	O
what	O
s	O
interesting	O
is	O
that	O
the	O
trade	O
off	O
between	O
precision	B
and	I
recall	B
is	O
captured	O
in	O
an	O
interesting	O
way	O
in	O
f	O
	O
.	O
so	O
in	O
order	O
to	O
understand	O
that	O
we	O
can	O
first	O
look	O
at	O
the	O
natural	O
question	O
.	O
why	O
not	O
just	O
the	O
combining	O
them	O
using	O
a	O
simple	O
arithmetic	O
mean	O
as	O
a	O
here	O
.	O
that	O
would	O
be	O
likely	O
the	O
most	O
natural	O
way	O
of	O
combining	O
them	O
.	O
so	O
what	O
do	O
you	O
think	O
if	O
you	O
want	O
to	O
think	O
more	O
you	O
can	O
pause	O
the	O
media	O
.	O
so	O
why	O
is	O
this	O
not	O
as	O
good	O
as	O
f	O
or	O
what	O
s	O
the	O
problem	O
with	O
this	O
now	O
if	O
you	O
think	O
about	O
the	O
arithmetic	O
mean	O
you	O
can	O
see	O
that	O
this	O
is	O
the	O
sum	O
of	O
of	O
multiple	O
terms	O
.	O
in	O
this	O
case	O
this	O
is	O
the	O
sum	O
of	O
precision	B
and	I
recall	B
.	O
in	O
the	O
case	O
of	O
the	O
sum	O
the	O
total	O
value	O
tends	O
to	O
be	O
dominated	O
by	O
the	O
large	O
values	O
.	O
that	O
means	O
if	O
you	O
have	O
a	O
very	O
high	O
p	O
or	O
a	O
very	O
high	O
r	O
then	O
you	O
really	O
don	O
t	O
care	O
about	O
the	O
whether	O
the	O
other	O
varies	O
is	O
low	O
.	O
so	O
the	O
whole	O
sum	O
would	O
be	O
high	O
.	O
now	O
this	O
is	O
not	O
the	O
desirable	O
because	O
one	O
can	O
easily	O
have	O
a	O
perfect	O
recall	B
.	O
we	O
can	O
have	O
a	O
perfect	O
recall	B
is	O
it	O
can	O
you	O
imagine	O
how	O
it	O
s	O
probably	O
very	O
easy	O
to	O
imagine	O
that	O
we	O
simply	O
retrieve	O
all	O
the	O
document	O
in	O
the	O
collection	O
then	O
we	O
have	O
a	O
perfect	O
recall	B
and	O
this	O
will	O
give	O
us	O
	O
.	O
as	O
the	O
average	O
.	O
but	O
search	O
results	O
are	O
clearly	O
not	O
very	O
useful	O
for	O
users	O
even	O
though	O
the	O
the	O
average	O
using	O
this	O
formula	O
would	O
be	O
relatively	O
high	O
.	O
now	O
in	O
contrast	O
you	O
can	O
see	O
f	O
will	O
reward	O
a	O
case	O
where	O
precision	B
and	I
recall	B
are	O
roughly	O
but	O
similar	O
.	O
so	O
it	O
would	O
paralyze	O
a	O
case	O
where	O
you	O
have	O
extremely	O
high	O
matter	O
for	O
one	O
of	O
them	O
.	O
so	O
this	O
means	O
f	O
encodes	O
a	O
different	O
trade	O
off	O
between	O
that	O
.	O
now	O
this	O
example	O
shows	O
actually	O
a	O
very	O
important	O
methodology	O
here	O
.	O
when	O
we	O
try	O
to	O
solve	O
a	O
problem	O
you	O
might	O
naturally	O
think	O
of	O
one	O
solution	O
.	O
let	O
s	O
say	O
in	O
this	O
case	O
it	O
s	O
this	O
arithmetic	O
mean	O
.	O
but	O
it	O
s	O
important	O
that	O
not	O
to	O
settle	O
on	O
this	O
solution	O
.	O
it	O
s	O
important	O
to	O
think	O
whether	O
you	O
have	O
other	O
ways	O
to	O
combine	O
them	O
.	O
and	O
once	O
you	O
think	O
about	O
the	O
multiple	O
variance	O
.	O
it	O
s	O
important	O
to	O
analyze	O
their	O
difference	O
and	O
then	O
think	O
about	O
which	O
one	O
makes	O
more	O
sense	O
.	O
in	O
this	O
case	O
if	O
you	O
think	O
more	O
carefully	O
you	O
will	O
feel	O
that	O
if	O
one	O
problem	O
makes	O
more	O
sense	O
.	O
then	O
the	O
simple	O
arithmetic	O
mean	O
.	O
although	O
in	O
other	O
cases	O
there	O
may	O
be	O
different	O
results	O
.	O
but	O
in	O
this	O
case	O
the	O
arithmetic	O
mean	O
seems	O
not	O
reasonable	O
.	O
but	O
if	O
you	O
don	O
t	O
pay	O
attention	O
to	O
these	O
subtle	O
differences	O
you	O
might	O
just	O
take	O
an	O
easy	O
way	O
to	O
combine	O
them	O
and	O
then	O
go	O
ahead	O
with	O
it	O
.	O
and	O
here	O
later	O
you	O
ll	O
find	O
that	O
hm	O
the	O
measure	O
doesn	O
t	O
seem	O
to	O
work	O
well	O
.	O
right	O
so	O
at	O
this	O
methodology	O
is	O
actually	O
very	O
important	O
in	O
general	O
in	O
solving	O
problem	O
and	O
try	O
to	O
think	O
about	O
the	O
best	O
solution	O
.	O
try	O
to	O
understand	O
that	O
the	O
problem	O
very	O
well	O
and	O
then	O
know	O
why	O
you	O
needed	O
this	O
measure	O
and	O
why	O
you	O
need	O
to	O
combine	O
precision	B
and	I
recall	B
.	O
and	O
then	O
use	O
that	O
to	O
guide	O
you	O
in	O
finding	O
a	O
good	O
way	O
to	O
solve	O
the	O
problem	O
.	O
to	O
summarize	O
we	O
talk	O
about	O
precision	B
which	O
addresses	O
the	O
question	O
are	O
the	O
retrieval	O
results	O
all	O
relevant	O
we	O
ll	O
also	O
talk	O
about	O
the	O
recall	B
which	O
addresses	O
the	O
question	O
have	O
all	O
the	O
relevant	B
documents	I
been	O
retrieved	O
these	O
two	O
are	O
the	O
two	O
basic	O
measures	O
in	O
testing	O
retrieval	O
in	O
variation	O
.	O
they	O
are	O
are	O
used	O
for	O
for	O
many	O
other	O
tasks	O
as	O
well	O
.	O
we	O
ll	O
talk	O
about	O
f	B
measure	I
as	O
a	O
way	O
to	O
combine	O
precision	B
and	I
recall	B
.	O
we	O
also	O
talked	O
about	O
the	O
trade	O
off	O
between	O
precision	B
and	I
recall	B
.	O
and	O
this	O
turns	O
out	O
to	O
depend	O
on	O
the	O
users	O
search	B
tasks	I
and	O
we	O
ll	O
discuss	O
this	O
point	O
more	O
in	O
the	O
later	O
lecture	O
.	O
so	O
average	B
precision	B
is	O
computer	O
for	O
just	O
one	O
.	O
one	O
query	O
.	O
but	O
we	O
generally	O
experiment	O
with	O
many	O
different	O
queries	O
and	O
this	O
is	O
to	O
avoid	O
the	O
variance	O
across	O
queries	O
.	O
depending	O
on	O
the	O
queries	O
you	O
use	O
you	O
might	O
make	O
different	O
conclusions	O
.	O
right	O
so	O
it	O
s	O
better	O
then	O
using	O
more	O
queries	O
.	O
if	O
you	O
use	O
more	O
queries	O
then	O
you	O
will	O
also	O
have	O
to	O
take	O
the	O
average	O
of	O
the	O
average	B
precision	B
over	O
all	O
these	O
queries	O
.	O
so	O
how	O
can	O
we	O
do	O
that	O
well	O
you	O
can	O
naturally	O
.	O
think	O
of	O
just	O
doing	O
arithmetic	O
mean	O
as	O
we	O
always	O
tend	O
to	O
to	O
think	O
in	O
in	O
this	O
way	O
.	O
so	O
this	O
would	O
give	O
us	O
what	O
s	O
called	O
a	O
mean	O
average	O
position	O
or	O
map	B
.	O
in	O
this	O
case	O
we	O
take	O
arithmetic	O
mean	O
of	O
all	O
the	O
average	O
precisions	O
over	O
several	O
queries	O
or	O
topics	O
.	O
but	O
as	O
i	O
just	O
mentioned	O
in	O
another	O
lecture	O
is	O
this	O
good	O
we	O
call	O
that	O
.	O
we	O
talked	O
about	O
the	O
different	O
ways	O
of	O
combining	O
precision	B
and	I
recall	B
.	O
and	O
we	O
conclude	O
that	O
the	O
arithmetic	O
mean	O
is	O
not	O
as	O
good	O
as	O
the	O
map	B
measure	O
.	O
but	O
here	O
it	O
s	O
the	O
same	O
.	O
we	O
can	O
also	O
think	O
about	O
the	O
alternative	O
ways	O
of	O
aggregating	O
the	O
numbers	O
.	O
don	O
t	O
just	O
automatically	O
assume	O
that	O
though	O
.	O
let	O
s	O
just	O
also	O
take	O
the	O
arithmetic	O
mean	O
of	O
the	O
average	O
position	O
over	O
these	O
queries	O
.	O
let	O
s	O
think	O
about	O
what	O
s	O
the	O
best	O
way	O
of	O
aggregating	O
them	O
.	O
if	O
you	O
think	O
about	O
the	O
different	O
ways	O
naturally	O
you	O
will	O
probably	O
be	O
able	O
to	O
think	O
about	O
another	O
way	O
which	O
is	O
geometric	B
mean	I
.	O
and	O
we	O
call	O
this	O
kind	O
of	O
average	O
a	O
gmap	O
.	O
this	O
is	O
another	O
way	O
.	O
so	O
now	O
once	O
you	O
think	O
about	O
the	O
two	O
different	O
ways	O
.	O
of	O
doing	O
the	O
same	O
thing	O
.	O
the	O
natural	O
question	O
to	O
ask	O
is	O
which	O
one	O
is	O
better	O
so	O
.	O
so	O
do	O
you	O
use	O
map	B
or	O
gmap	O
again	O
that	O
s	O
important	O
question	O
.	O
imagine	O
you	O
are	O
again	O
testing	O
a	O
new	O
algorithm	O
in	O
by	O
comparing	O
the	O
ways	O
your	O
old	O
algorithms	O
made	O
the	O
search	B
engine	I
.	O
now	O
you	O
tested	O
multiple	O
topics	O
.	O
now	O
you	O
ve	O
got	O
the	O
average	B
precision	B
for	O
these	O
topics	O
.	O
now	O
you	O
are	O
thinking	O
of	O
looking	O
at	O
the	O
overall	O
performance	O
.	O
you	O
have	O
to	O
take	O
the	O
average	O
.	O
but	O
which	O
which	O
strategy	O
would	O
you	O
use	O
now	O
first	O
you	O
should	O
also	O
think	O
about	O
the	O
question	O
well	O
did	O
it	O
make	O
a	O
difference	O
can	O
you	O
think	O
of	O
scenarios	O
where	O
using	O
one	O
of	O
them	O
would	O
make	O
a	O
difference	O
that	O
is	O
they	O
would	O
give	O
different	O
rankings	O
of	O
those	O
methods	O
.	O
and	O
that	O
also	O
means	O
depending	O
on	O
the	O
way	O
you	O
average	O
or	O
detect	O
the	O
.	O
average	O
of	O
these	O
average	O
positions	O
.	O
you	O
will	O
get	O
different	O
conclusions	O
.	O
this	O
makes	O
the	O
question	O
becoming	O
even	O
more	O
important	O
.	O
right	O
so	O
which	O
one	O
would	O
you	O
use	O
well	O
again	O
if	O
you	O
look	O
at	O
the	O
difference	O
between	O
these	O
.	O
different	O
ways	O
of	O
aggregating	O
the	O
average	O
position	O
.	O
you	O
ll	O
realize	O
in	O
arithmetic	O
mean	O
the	O
sum	O
is	O
dominating	O
by	O
large	O
values	O
.	O
so	O
what	O
does	O
large	O
value	O
here	O
mean	O
it	O
means	O
the	O
query	O
is	O
relatively	O
easy	O
.	O
you	O
can	O
have	O
a	O
high	O
pres	O
average	O
position	O
.	O
whereas	O
gmap	O
tends	O
to	O
be	O
affected	O
more	O
by	O
low	O
values	O
.	O
and	O
those	O
are	O
the	O
queries	O
that	O
don	O
t	O
have	O
good	O
performance	O
.	O
the	O
average	B
precision	B
is	O
low	O
.	O
so	O
if	O
you	O
think	O
about	O
the	O
improving	O
the	O
search	B
engine	I
for	O
those	O
difficult	O
queries	O
then	O
gmap	O
would	O
be	O
preferred	O
right	O
on	O
the	O
other	O
hand	O
if	O
you	O
just	O
want	O
to	O
.	O
have	O
improved	O
a	O
lot	O
.	O
over	O
all	O
the	O
kinds	O
of	O
queries	O
or	O
particular	O
popular	O
queries	O
that	O
might	O
be	O
easy	O
and	O
you	O
want	O
to	O
make	O
the	O
perfect	O
and	O
maybe	O
map	B
would	O
be	O
then	O
preferred	O
.	O
so	O
again	O
the	O
answer	O
depends	O
on	O
your	O
users	O
your	O
users	O
tasks	O
and	O
their	O
pref	O
their	O
preferences	O
.	O
so	O
the	O
point	O
that	O
here	O
is	O
to	O
think	O
about	O
the	O
multiple	O
ways	O
to	O
solve	O
the	O
same	O
problem	O
and	O
then	O
compare	O
them	O
and	O
think	O
carefully	O
about	O
the	O
differences	O
.	O
and	O
which	O
one	O
makes	O
more	O
sense	O
.	O
often	O
when	O
one	O
of	O
them	O
might	O
make	O
sense	O
in	O
one	O
situation	O
and	O
another	O
might	O
make	O
more	O
sense	O
in	O
a	O
different	O
situation	O
.	O
so	O
it	O
s	O
important	O
to	O
pick	O
out	O
under	O
what	O
situations	O
one	O
is	O
preferred	O
.	O
as	O
a	O
special	O
case	O
of	O
the	O
mean	O
average	O
position	O
we	O
can	O
also	O
think	O
about	O
the	O
case	O
where	O
there	O
was	O
precisely	O
one	O
rank	O
in	O
the	O
document	O
.	O
and	O
this	O
happens	O
often	O
for	O
example	O
in	O
what	O
s	O
called	O
a	O
known	O
item	O
search	O
.	O
where	O
you	O
know	O
a	O
target	O
page	O
let	O
s	O
say	O
you	O
have	O
to	O
find	O
amazon	O
homepage	O
.	O
you	O
have	O
one	O
relevant	O
document	O
there	O
and	O
you	O
hope	O
to	O
find	O
it	O
.	O
that	O
s	O
call	O
a	O
known	O
item	O
search	O
	O
.	O
in	O
that	O
case	O
there	O
s	O
precisely	O
one	O
relevant	O
document	O
.	O
or	O
in	O
another	O
application	O
like	O
a	O
question	O
and	O
answering	O
maybe	O
there	O
s	O
only	O
one	O
answer	O
.	O
are	O
there	O
.	O
so	O
if	O
you	O
rank	O
the	O
answers	O
then	O
your	O
goal	O
is	O
to	O
rank	O
that	O
one	O
particular	O
answer	O
on	O
top	O
right	O
so	O
in	O
this	O
case	O
you	O
can	O
easily	O
verify	O
the	O
average	O
position	O
will	O
basically	O
boil	O
down	O
to	O
reciprocal	B
rank	I
.	O
that	O
is	O
over	O
r	O
where	O
r	O
is	O
the	O
rank	O
position	O
of	O
that	O
single	O
relevant	O
document	O
.	O
so	O
if	O
that	O
document	O
is	O
ranked	O
on	O
the	O
very	O
top	O
or	O
is	O
and	O
then	O
it	O
s	O
for	O
reciprocal	B
rank	I
.	O
if	O
it	O
s	O
ranked	O
at	O
the	O
the	O
second	O
then	O
it	O
s	O
over	O
	O
.	O
et	O
cetera	O
.	O
and	O
then	O
we	O
can	O
also	O
take	O
a	O
a	O
average	O
of	O
all	O
these	O
average	B
precision	B
or	O
reciprocal	B
rank	I
over	O
a	O
set	O
of	O
topics	O
and	O
that	O
would	O
give	O
us	O
something	O
called	O
a	O
mean	O
reciprocal	B
rank	I
.	O
it	O
s	O
a	O
very	O
popular	O
measure	O
.	O
for	O
no	O
item	O
search	O
or	O
you	O
know	O
an	O
problem	O
where	O
you	O
have	O
just	O
one	O
relevant	O
item	O
.	O
now	O
again	O
here	O
you	O
can	O
see	O
this	O
r	O
actually	O
is	O
meaningful	O
here	O
.	O
and	O
this	O
r	O
is	O
basically	O
indicating	O
how	O
much	O
effort	O
a	O
user	O
would	O
have	O
to	O
make	O
in	O
order	O
to	O
find	O
that	O
relevant	O
document	O
.	O
if	O
it	O
s	O
ranked	O
on	O
the	O
top	O
it	O
s	O
low	O
effort	O
that	O
you	O
have	O
to	O
make	O
or	O
little	O
effort	O
.	O
but	O
if	O
it	O
s	O
ranked	O
at	O
then	O
you	O
actually	O
have	O
to	O
read	O
presumably	O
documents	O
in	O
order	O
to	O
find	O
it	O
.	O
so	O
in	O
this	O
sense	O
r	O
is	O
also	O
a	O
meaningful	O
measure	O
and	O
the	O
reciprocal	B
rank	I
will	O
take	O
the	O
reciprocal	O
of	O
r	O
instead	O
of	O
using	O
r	O
directly	O
.	O
so	O
my	O
natural	O
question	O
here	O
is	O
why	O
not	O
simply	O
using	O
r	O
i	O
imagine	O
if	O
you	O
were	O
to	O
design	O
a	O
ratio	O
to	O
measure	O
the	O
performance	O
of	O
a	O
random	O
system	O
when	O
there	O
is	O
only	O
one	O
relevant	O
item	O
.	O
you	O
might	O
have	O
thought	O
about	O
using	O
r	O
directly	O
as	O
the	O
measure	O
.	O
after	O
all	O
that	O
measures	O
the	O
user	O
s	O
effort	O
right	O
but	O
think	O
about	O
if	O
you	O
take	O
a	O
average	O
of	O
this	O
over	O
a	O
large	O
number	O
of	O
topics	O
.	O
again	O
it	O
would	O
make	O
a	O
difference	O
.	O
right	O
for	O
one	O
single	O
topic	O
using	O
r	O
or	O
using	O
over	O
r	O
wouldn	O
t	O
make	O
any	O
difference	O
.	O
it	O
s	O
the	O
same	O
.	O
larger	O
r	O
with	O
corresponds	O
to	O
a	O
small	O
over	O
r	O
right	O
but	O
the	O
difference	O
would	O
only	O
show	O
when	O
show	O
up	O
when	O
you	O
have	O
many	O
topics	O
.	O
so	O
again	O
think	O
about	O
the	O
average	O
of	O
mean	O
reciprocal	B
rank	I
versus	O
average	O
of	O
just	O
r	O
.	O
what	O
s	O
the	O
difference	O
do	O
you	O
see	O
any	O
difference	O
and	O
would	O
would	O
this	O
difference	O
change	O
the	O
oath	O
of	O
systems	O
.	O
in	O
our	O
conclusion	O
.	O
and	O
this	O
it	O
turns	O
out	O
that	O
there	O
is	O
actually	O
a	O
big	O
difference	O
and	O
if	O
you	O
think	O
about	O
it	O
if	O
you	O
want	O
to	O
think	O
about	O
it	O
and	O
then	O
yourself	O
then	O
pause	O
the	O
video	O
.	O
basically	O
the	O
difference	O
is	O
if	O
you	O
take	O
some	O
of	O
our	O
directory	O
then	O
.	O
again	O
it	O
will	O
be	O
dominated	O
by	O
large	O
values	O
of	O
r	O
.	O
so	O
what	O
are	O
those	O
values	O
those	O
are	O
basically	O
large	O
values	O
that	O
indicate	O
that	O
lower	O
ranked	O
results	O
.	O
that	O
means	O
the	O
relevant	B
items	I
rank	O
very	O
low	O
down	O
on	O
the	O
list	O
.	O
and	O
the	O
sum	O
that	O
s	O
also	O
the	O
average	O
that	O
would	O
then	O
be	O
dominated	O
by	O
.	O
where	O
those	O
relevant	B
documents	I
are	O
ranked	O
in	O
in	O
in	O
in	O
the	O
lower	O
portion	O
of	O
the	O
ranked	O
.	O
but	O
from	O
a	O
users	O
perspective	O
we	O
care	O
more	O
about	O
the	O
highly	O
ranked	O
documents	O
.	O
so	O
by	O
taking	O
this	O
transformation	O
by	O
using	O
reciprocal	B
rank	I
.	O
here	O
we	O
emphasize	O
more	O
on	O
the	O
difference	O
on	O
the	O
top	O
.	O
you	O
know	O
think	O
about	O
the	O
difference	O
between	O
and	O
the	O
it	O
would	O
make	O
a	O
big	O
difference	O
in	O
over	O
r	O
but	O
think	O
about	O
the	O
and	O
and	O
where	O
and	O
when	O
won	O
t	O
make	O
much	O
difference	O
if	O
you	O
use	O
this	O
.	O
but	O
if	O
you	O
use	O
this	O
there	O
will	O
be	O
a	O
big	O
difference	O
in	O
and	O
let	O
s	O
say	O
right	O
.	O
so	O
this	O
is	O
not	O
the	O
desirable	O
.	O
on	O
the	O
other	O
hand	O
a	O
and	O
won	O
t	O
make	O
much	O
difference	O
.	O
so	O
this	O
is	O
yet	O
another	O
case	O
where	O
there	O
may	O
be	O
multiple	O
choices	O
of	O
doing	O
the	O
same	O
thing	O
and	O
then	O
you	O
need	O
to	O
figure	O
out	O
which	O
one	O
makes	O
more	O
sense	O
.	O
so	O
to	O
summarize	O
we	O
showed	O
that	O
the	O
precision-recall	B
curve	I
.	O
can	O
characterize	O
the	O
overall	O
accuracy	O
of	O
a	O
ranked	O
list	O
.	O
and	O
we	O
emphasized	O
that	O
the	O
actual	O
utility	O
of	O
a	O
ranked	O
list	O
depends	O
on	O
how	O
many	O
top	O
ranked	O
results	O
a	O
user	O
would	O
actually	O
examine	O
.	O
some	O
users	O
will	O
examine	O
more	O
.	O
than	O
others	O
.	O
an	O
average	O
person	O
uses	O
a	O
standard	O
measure	O
for	O
comparing	O
two	O
ranking	B
methods	I
.	O
it	O
combines	O
precision	B
and	I
recall	B
and	O
it	O
s	O
sensitive	O
to	O
the	O
rank	O
of	O
every	O
random	O
document	O
.	O
this	O
lecture	O
is	O
about	O
how	O
to	O
evaluate	O
the	O
text	B
retrieval	I
system	O
when	O
we	O
have	O
multiple	O
levels	O
of	O
judgments	O
.	O
in	O
this	O
lecture	O
we	O
will	O
continue	O
the	O
discussion	O
of	O
evaluation	O
.	O
we	O
re	O
going	O
to	O
look	O
at	O
how	O
to	O
evaluate	O
the	O
text	B
retrieval	I
system	O
.	O
and	O
we	O
have	O
multiple	O
level	O
of	O
judgements	O
.	O
so	O
so	O
far	O
we	O
have	O
talked	O
about	O
binding	O
judgements	O
that	O
means	O
a	O
documents	O
is	O
judged	O
as	O
being	O
relevant	O
or	O
not-relevant	O
.	O
but	O
earlier	O
we	O
will	O
also	O
talk	O
about	O
relevance	O
as	O
a	O
matter	O
of	O
degree	O
.	O
so	O
we	O
often	O
can	O
distinguish	O
it	O
very	O
higher	O
relevant	O
options	O
those	O
are	O
very	O
useful	O
options	O
from	O
you	O
know	O
lower	O
rated	O
relevant	O
options	O
.	O
they	O
are	O
okay	O
they	O
are	O
useful	O
perhaps	O
.	O
and	O
further	O
from	O
non-relevant	O
documents	O
.	O
those	O
are	O
not	O
useful	O
.	O
right	O
so	O
imagine	O
you	O
can	O
have	O
ratings	O
for	O
these	O
pages	O
.	O
then	O
you	O
would	O
have	O
much	O
more	O
levels	O
of	O
ratings	O
.	O
for	O
example	O
here	O
i	O
show	O
an	O
example	O
of	O
three	O
levels	O
three	O
were	O
relevant	O
.	O
sorry	O
three	O
were	O
very	O
relevant	O
.	O
two	O
for	O
marginally	O
relevant	O
and	O
one	O
for	O
non-relevant	O
.	O
now	O
how	O
do	O
we	O
evaluate	O
such	O
a	O
new	O
system	O
using	O
these	O
judgements	O
of	O
use	O
of	O
the	O
map	B
doesn	O
t	O
work	O
average	O
of	O
precision	B
doesn	O
t	O
work	O
precision	B
and	O
record	O
doesn	O
t	O
work	O
because	O
they	O
rely	O
on	O
vinyl	O
judgement	O
.	O
so	O
let	O
s	O
look	O
at	O
the	O
sum	O
top	O
regular	O
results	O
when	O
using	O
these	O
judgments	O
.	O
right	O
imagine	O
the	O
user	O
would	O
be	O
mostly	O
care	O
about	O
the	O
top	O
ten	O
results	O
here	O
.	O
right	O
.	O
and	O
we	O
mark	O
the	O
the	O
rating	O
levels	O
or	O
relevance	O
levels	O
for	O
these	O
documents	O
as	O
shown	O
here	O
.	O
three	O
two	O
one	O
one	O
three	O
et	O
cetera	O
.	O
and	O
we	O
call	O
these	O
gain	B
.	O
and	O
the	O
reason	O
why	O
we	O
call	O
it	O
a	O
gain	B
is	O
because	O
the	O
measure	O
that	O
we	O
are	O
infusing	O
is	O
called	O
ntcg	O
normalizer	O
discount	O
of	O
accumulative	O
gain	B
.	O
so	O
this	O
gain	B
basically	O
can	O
mesh	O
your	O
how	O
much	O
gain	B
of	O
random	O
information	O
a	O
user	O
can	O
obtain	O
by	O
looking	O
at	O
each	O
document	O
alright	O
.	O
so	O
looking	O
after	O
the	O
first	O
document	O
the	O
user	O
can	O
gain	B
three	O
points	O
.	O
looking	O
at	O
the	O
non-relevant	O
document	O
the	O
user	O
would	O
only	O
gain	B
one	O
point	O
.	O
right	O
.	O
looking	O
at	O
the	O
multi-level	B
relevant	O
or	O
marginally	O
relevant	O
document	O
the	O
user	O
would	O
get	O
two	O
points	O
et	O
cetera	O
.	O
so	O
this	O
gain	B
usually	O
matches	O
the	O
utility	O
of	O
a	O
document	O
from	O
a	O
user	O
s	O
perspective	O
.	O
of	O
course	O
if	O
we	O
assume	O
the	O
user	O
stops	O
at	O
the	O
ten	O
documents	O
and	O
we	O
re	O
looking	O
at	O
the	O
cutoff	O
at	O
ten	O
we	O
can	O
look	O
after	O
the	O
total	O
gain	B
of	O
the	O
user	O
.	O
and	O
what	O
s	O
that	O
well	O
that	O
s	O
simply	O
the	O
sum	O
of	O
these	O
and	O
we	O
call	O
it	O
the	O
cumulative	B
gain	B
.	O
so	O
if	O
we	O
use	O
a	O
stops	O
at	O
the	O
positua	O
that	O
s	O
just	O
a	O
three	O
.	O
if	O
the	O
user	O
looks	O
at	O
another	O
document	O
that	O
s	O
a	O
plus	O
	O
.	O
if	O
the	O
user	O
looks	O
at	O
the	O
more	O
documents	O
.	O
then	O
the	O
cumulative	B
gain	B
is	O
more	O
.	O
of	O
course	O
this	O
is	O
at	O
the	O
cost	O
of	O
spending	O
more	O
time	O
to	O
examine	O
the	O
list	O
.	O
so	O
cumulative	B
gain	B
gives	O
us	O
some	O
idea	O
about	O
how	O
much	O
total	O
gain	B
the	O
user	O
would	O
have	O
if	O
the	O
user	O
examines	O
all	O
these	O
documents	O
.	O
now	O
in	O
ndcg	B
we	O
also	O
have	O
another	O
letter	O
here	O
d	O
discounted	B
cumulative	B
gain	B
.	O
so	O
why	O
do	O
we	O
want	O
to	O
do	O
discounting	O
well	O
if	O
you	O
look	O
at	O
this	O
cumulative	B
gain	B
there	O
is	O
one	O
deficiency	O
which	O
is	O
it	O
did	O
not	O
consider	O
the	O
rank	O
position	O
of	O
these	O
these	O
documents	O
.	O
so	O
for	O
example	O
looking	O
at	O
the	O
this	O
sum	O
here	O
and	O
we	O
only	O
know	O
there	O
is	O
only	O
one	O
highly	O
relevant	O
document	O
one	O
marginally	O
relevant	O
document	O
two	O
non-relevant	O
documents	O
.	O
we	O
don	O
t	O
really	O
care	O
where	O
they	O
are	O
ranked	O
.	O
ideally	O
we	O
want	O
these	O
two	O
to	O
be	O
ranked	O
on	O
the	O
top	O
.	O
which	O
is	O
the	O
case	O
here	O
.	O
but	O
how	O
can	O
we	O
capture	O
that	O
intuition	O
well	O
we	O
have	O
to	O
say	O
well	O
this	O
here	O
is	O
not	O
as	O
good	O
as	O
this	O
on	O
the	O
top	O
.	O
and	O
that	O
means	O
the	O
contribution	O
of	O
the	O
game	O
from	O
different	O
positions	O
has	O
to	O
be	O
weight	O
by	O
their	O
position	O
.	O
and	O
this	O
is	O
the	O
idea	O
of	O
discounting	O
basically	O
.	O
so	O
we	O
re	O
going	O
to	O
say	O
well	O
the	O
first	O
one	O
doesn	O
t	O
it	O
need	O
to	O
be	O
discounted	O
because	O
the	O
user	O
can	O
be	O
assume	O
that	O
you	O
always	O
see	O
this	O
document	O
but	O
the	O
second	O
one	O
this	O
one	O
will	O
be	O
discounted	O
a	O
little	O
bit	O
because	O
there	O
s	O
a	O
small	O
possibility	O
that	O
the	O
user	O
wouldn	O
t	O
notice	O
it	O
.	O
so	O
we	O
divide	O
this	O
gain	B
by	O
the	O
weight	O
based	O
on	O
the	O
position	O
.	O
so	O
log	O
of	O
two	O
two	O
is	O
the	O
rank	O
position	O
of	O
this	O
document	O
and	O
when	O
we	O
go	O
to	O
the	O
third	O
position	O
we	O
discount	O
even	O
more	O
because	O
the	O
numbers	O
is	O
log	O
of	O
three	O
and	O
so	O
on	O
and	O
so	O
forth	O
.	O
so	O
when	O
we	O
take	O
a	O
such	O
a	O
sum	O
then	O
a	O
lowly	O
ranked	O
document	O
would	O
not	O
contribute	O
contribute	O
that	O
much	O
as	O
a	O
highly	O
ranked	O
document	O
.	O
so	O
that	O
means	O
if	O
you	O
for	O
example	O
switch	O
the	O
position	O
of	O
this	O
and	O
let	O
s	O
say	O
this	O
position	O
and	O
this	O
one	O
and	O
then	O
you	O
would	O
get	O
more	O
discount	O
if	O
you	O
put	O
for	O
example	O
very	O
relevant	O
document	O
here	O
as	O
opposed	O
to	O
two	O
here	O
.	O
imagine	O
if	O
you	O
put	O
the	O
three	O
here	O
then	O
it	O
would	O
have	O
to	O
be	O
discounted	O
.	O
so	O
it	O
s	O
not	O
as	O
good	O
as	O
if	O
you	O
would	O
put	O
the	O
three	O
here	O
.	O
so	O
this	O
is	O
the	O
idea	O
of	O
discounting	O
.	O
okay	O
so	O
n	O
now	O
at	O
this	O
point	O
that	O
we	O
have	O
got	O
this	O
discounted	B
cumulative	B
gain	B
for	O
measuring	O
the	O
utility	O
of	O
this	O
ranked	O
list	O
with	O
multiple	O
levels	O
of	O
judgments	O
.	O
so	O
are	O
we	O
happy	O
with	O
this	O
well	O
we	O
can	O
use	O
this	O
rank	O
systems	O
.	O
now	O
we	O
still	O
need	O
to	O
do	O
a	O
little	O
bit	O
more	O
in	O
order	O
to	O
make	O
this	O
measure	O
comfortable	O
across	O
different	O
topics	O
.	O
and	O
this	O
is	O
the	O
last	O
step	O
.	O
and	O
by	O
the	O
way	O
here	O
we	O
just	O
show	O
that	O
dcg	B
at	O
the	O
ten	O
.	O
alright	O
.	O
so	O
this	O
is	O
the	O
total	O
sum	O
of	O
dcg	B
over	O
all	O
these	O
ten	O
documents	O
.	O
so	O
the	O
last	O
step	O
is	O
called	O
n	O
normalization	O
.	O
and	O
if	O
we	O
do	O
that	O
then	O
we	O
get	O
normalized	B
dcg	B
.	O
so	O
how	O
do	O
we	O
do	O
that	O
well	O
the	O
idea	O
here	O
is	O
within	O
the	O
normalized	B
dcg	B
by	O
the	O
ideal	O
dcg	B
at	O
the	O
same	O
cutoff	O
.	O
what	O
is	O
the	O
ideal	O
dcg	B
well	O
this	O
is	O
a	O
dcg	B
of	O
ideal	O
ranking	O
.	O
so	O
imagine	O
if	O
we	O
have	O
nine	O
documents	O
in	O
the	O
whole	O
collection	O
rated	O
a	O
three	O
here	O
and	O
that	O
means	O
in	O
total	O
we	O
have	O
nine	O
documents	O
rated	O
three	O
.	O
then	O
our	O
ideal	O
ranked	O
the	O
lister	O
would	O
have	O
put	O
all	O
these	O
nine	O
documents	O
on	O
the	O
very	O
top	O
.	O
so	O
all	O
these	O
would	O
have	O
to	O
be	O
three	O
and	O
then	O
this	O
would	O
be	O
followed	O
by	O
a	O
two	O
here	O
because	O
that	O
s	O
the	O
best	O
we	O
could	O
do	O
after	O
we	O
have	O
run	O
out	O
of	O
threes	O
.	O
but	O
all	O
these	O
positions	O
would	O
be	O
threes	O
.	O
right	O
so	O
this	O
would	O
be	O
our	O
ideal	O
ranked	O
list	O
.	O
and	O
then	O
we	O
can	O
compute	O
the	O
dcg	B
for	O
this	O
ideal	O
rank	O
list	O
.	O
so	O
this	O
would	O
be	O
given	O
by	O
this	O
formula	O
you	O
see	O
here	O
and	O
so	O
this	O
idea	O
dcg	B
would	O
be	O
used	O
as	O
the	O
normalizer	O
dcg	B
.	O
like	O
so	O
here	O
and	O
this	O
idealdcg	O
would	O
be	O
used	O
as	O
a	O
normalizer	O
.	O
so	O
you	O
can	O
imagine	O
now	O
normalization	O
essentially	O
is	O
to	O
compare	O
the	O
actual	O
dcg	B
with	O
the	O
best	O
decision	O
you	O
can	O
possibly	O
get	O
for	O
this	O
topic	O
.	O
now	O
why	O
do	O
we	O
want	O
to	O
do	O
this	O
well	O
by	O
doing	O
this	O
we	O
ll	O
map	B
the	O
dcg	B
values	O
in	O
to	O
a	O
range	O
of	O
zero	O
through	O
one	O
so	O
the	O
best	O
value	O
or	O
the	O
highest	O
value	O
for	O
every	O
query	O
would	O
be	O
one	O
.	O
that	O
s	O
when	O
you	O
re	O
relevance	O
is	O
in	O
fact	O
the	O
idealist	O
.	O
but	O
otherwise	O
in	O
general	O
you	O
will	O
be	O
lower	O
than	O
one	O
.	O
now	O
what	O
if	O
we	O
don	O
t	O
do	O
that	O
well	O
you	O
can	O
see	O
this	O
transformation	O
or	O
this	O
numberization	O
doesn	O
t	O
really	O
affect	O
the	O
relative	O
comparison	O
of	O
systems	O
for	O
just	O
one	O
topic	O
because	O
this	O
ideal	O
dcg	B
is	O
the	O
same	O
for	O
all	O
the	O
systems	O
.	O
so	O
the	O
ranking	O
of	O
systems	O
based	O
on	O
only	O
dcg	B
would	O
be	O
exactly	O
the	O
same	O
.	O
as	O
if	O
you	O
rank	O
them	O
based	O
on	O
the	O
normalized	O
decision	O
.	O
the	O
difference	O
however	O
is	O
when	O
we	O
have	O
multiple	O
topics	O
because	O
if	O
we	O
don	O
t	O
do	O
normalization	O
different	O
topics	O
will	O
have	O
different	O
scales	O
of	O
dcg	B
.	O
for	O
a	O
topic	O
like	O
this	O
one	O
we	O
have	O
nine	O
highly	O
relevant	B
documents	I
.	O
the	O
dcg	B
can	O
get	O
really	O
high	O
.	O
but	O
imagine	O
that	O
in	O
another	O
case	O
there	O
are	O
only	O
two	O
very	O
relevant	B
documents	I
.	O
in	O
total	O
in	O
the	O
whole	O
collection	O
.	O
then	O
the	O
highest	O
dcg	B
that	O
any	O
system	O
could	O
achieve	O
for	O
such	O
a	O
topic	O
would	O
not	O
be	O
very	O
high	O
.	O
so	O
again	O
we	O
face	O
the	O
problem	O
of	O
different	O
scales	O
of	O
dcg	B
values	O
and	O
when	O
we	O
take	O
an	O
average	O
we	O
don	O
t	O
want	O
the	O
average	O
to	O
be	O
dominated	O
by	O
those	O
high	O
values	O
.	O
those	O
are	O
again	O
easy	O
quires	O
.	O
so	O
by	O
doing	O
the	O
normalization	O
we	O
have	O
all	O
avoid	O
the	O
problem	O
.	O
making	O
all	O
the	O
purists	O
contribute	O
equal	O
to	O
the	O
average	O
.	O
so	O
this	O
is	O
the	O
idea	O
of	O
ndcg	B
.	O
it	O
s	O
used	O
for	O
measuring	O
relevance	O
based	O
on	O
much	O
more	O
level	O
relevance	B
judgments	I
.	O
so	O
more	O
in	O
the	O
more	O
general	O
way	O
this	O
is	O
basically	O
a	O
measure	O
that	O
can	O
be	O
applied	O
through	O
any	O
ranked	O
task	O
with	O
much	O
more	O
level	O
of	O
of	O
judgments	O
.	O
and	O
the	O
scale	O
of	O
the	O
judgments	O
can	O
be	O
multiple	O
can	O
be	O
more	O
than	O
binary	O
not	O
only	O
more	O
than	O
binary	O
they	O
can	O
be	O
multiple	O
levels	O
like	O
one	O
s	O
or	O
five	O
or	O
even	O
more	O
depending	O
on	O
your	O
application	O
.	O
and	O
the	O
main	O
idea	O
of	O
this	O
measure	O
just	O
to	O
summarize	O
is	O
to	O
measure	O
the	O
total	O
utility	O
of	O
the	O
top	O
k	O
documents	O
.	O
so	O
you	O
always	O
choose	O
a	O
cutoff	O
and	O
then	O
you	O
measure	O
the	O
total	O
utility	O
.	O
and	O
it	O
would	O
discount	O
the	O
contribution	O
from	O
a	O
lowly	O
ranked	O
document	O
and	O
finally	O
it	O
would	O
do	O
normalization	O
to	O
ensure	O
comparability	O
across	O
queries	O
	O
.	O
this	O
lecture	O
is	O
about	O
some	O
practical	O
issues	O
that	O
you	O
would	O
have	O
to	O
address	O
in	O
evaluation	B
of	I
text	B
retrieval	B
systems	I
.	O
in	O
this	O
lecture	O
we	O
will	O
continue	O
the	O
discussion	O
of	O
evaluation	O
.	O
we	O
will	O
cover	O
some	O
practical	O
issues	O
that	O
you	O
will	O
have	O
to	O
solve	O
in	O
actual	O
evaluation	B
of	I
text	B
retrieval	B
systems	I
.	O
so	O
in	O
order	O
to	O
create	O
a	O
test	B
collection	I
we	O
have	O
to	O
create	O
a	O
set	O
of	O
queries	O
a	O
set	O
of	O
documents	O
and	O
a	O
set	O
of	O
relevance	B
judgments	I
.	O
it	O
turns	O
out	O
that	O
each	O
is	O
actually	O
challenging	O
to	O
create	O
.	O
so	O
first	O
the	O
documents	O
and	O
queries	O
must	O
be	O
representative	O
.	O
they	O
must	O
rep	O
represent	O
the	O
real	O
queries	O
and	O
real	O
documents	O
that	O
the	O
users	O
handle	O
.	O
and	O
we	O
also	O
have	O
to	O
use	O
many	O
queries	O
and	O
many	O
documents	O
in	O
order	O
to	O
avoid	O
biased	O
conclusions	O
.	O
for	O
the	O
matching	O
of	O
relevant	B
documents	I
with	O
the	O
queries	O
we	O
also	O
need	O
to	O
ensure	O
that	O
there	O
exists	O
a	O
lot	O
of	O
relevant	B
documents	I
for	O
each	O
query	O
.	O
if	O
a	O
query	O
has	O
only	O
one	O
that	O
is	O
a	O
relevant	O
document	O
in	O
the	O
collection	O
then	O
you	O
know	O
it	O
s	O
not	O
very	O
informative	O
to	O
compare	O
different	O
methods	O
using	O
such	O
a	O
query	O
because	O
there	O
is	O
not	O
much	O
room	O
for	O
us	O
to	O
see	O
difference	O
.	O
so	O
ideally	O
there	O
should	O
be	O
more	O
relevant	B
documents	I
in	O
the	O
collection	O
.	O
but	O
yet	O
the	O
queries	O
also	O
should	O
represent	O
real	O
queries	O
that	O
we	O
care	O
about	O
.	O
in	O
terms	O
of	O
relevance	O
judgements	O
the	O
challenge	O
is	O
to	O
ensure	O
complete	O
judgements	O
of	O
all	O
the	O
documents	O
for	O
all	O
the	O
queries	O
yet	O
minimizing	O
human	O
and	O
fault	O
.	O
because	O
we	O
have	O
to	O
use	O
the	O
human	O
labor	O
to	O
label	O
these	O
documents	O
.	O
it	O
s	O
very	O
labor	O
intensive	O
.	O
and	O
as	O
a	O
result	O
it	O
s	O
impossible	O
to	O
actually	O
label	O
all	O
of	O
the	O
documents	O
for	O
all	O
the	O
queries	O
especially	O
considering	O
a	O
joint	O
data	O
set	O
like	O
the	O
web	O
.	O
so	O
this	O
is	O
actually	O
a	O
major	O
challenge	O
.	O
it	O
s	O
a	O
very	O
difficult	O
challenge	O
.	O
for	O
measures	O
it	O
s	O
also	O
challenging	O
because	O
what	O
we	O
want	O
with	O
measures	O
is	O
that	O
with	O
accuracy	O
reflected	O
the	O
perceived	O
utility	O
of	O
users	O
.	O
we	O
have	O
to	O
consider	O
carefully	O
what	O
the	O
users	O
care	O
about	O
and	O
then	O
design	O
measures	O
to	O
measure	O
that	O
.	O
if	O
we	O
your	O
measure	O
is	O
not	O
measuring	O
the	O
right	O
thing	O
then	O
your	O
conclusion	O
would	O
would	O
be	O
misled	O
.	O
so	O
it	O
s	O
very	O
important	O
.	O
so	O
we	O
re	O
going	O
to	O
talk	O
about	O
a	O
couple	O
issues	O
here	O
.	O
one	O
is	O
the	O
statistical	B
significance	B
test	I
and	O
this	O
also	O
is	O
the	O
reason	O
why	O
we	O
have	O
to	O
use	O
a	O
lot	O
of	O
queries	O
and	O
the	O
question	O
here	O
is	O
how	O
sure	O
can	O
you	O
be	O
that	O
i	O
observed	O
the	O
difference	O
it	O
doesn	O
t	O
simply	O
result	O
from	O
the	O
particular	O
queries	O
you	O
choose	O
.	O
so	O
here	O
are	O
some	O
sample	O
results	O
of	O
average	B
precision	B
for	O
system	O
a	O
and	O
system	O
b	O
in	O
two	O
different	O
experiments	O
.	O
and	O
you	O
can	O
see	O
in	O
the	O
bottom	O
we	O
have	O
mean	O
average	O
position	O
all	O
right	O
so	O
the	O
mean	O
if	O
you	O
look	O
at	O
the	O
mean	O
average	O
position	O
the	O
mean	O
average	O
positions	O
are	O
exactly	O
the	O
same	O
in	O
both	O
experiments	O
.	O
all	O
right	O
so	O
you	O
can	O
see	O
this	O
is	O
	O
.	O
this	O
is	O
	O
.	O
for	O
system	O
b	O
and	O
again	O
here	O
its	O
also	O
	O
.	O
and	O
	O
.	O
	O
.	O
so	O
they	O
are	O
identical	O
.	O
yet	O
if	O
you	O
look	O
at	O
the	O
these	O
exact	O
average	O
positions	O
for	O
different	O
queries	O
if	O
you	O
look	O
at	O
these	O
numbers	O
in	O
detail	O
you	O
will	O
realize	O
that	O
in	O
one	O
case	O
you	O
would	O
feel	O
that	O
you	O
can	O
trust	O
the	O
conclusion	O
here	O
given	O
by	O
the	O
average	O
.	O
in	O
another	O
case	O
in	O
the	O
other	O
case	O
you	O
will	O
feel	O
that	O
well	O
i	O
m	O
not	O
sure	O
.	O
so	O
why	O
don	O
t	O
you	O
take	O
a	O
look	O
at	O
all	O
these	O
numbers	O
for	O
a	O
moment	O
.	O
pause	O
the	O
video	O
.	O
so	O
if	O
you	O
at	O
the	O
average	O
the	O
main	O
average	O
position	O
we	O
can	O
easily	O
say	O
that	O
well	O
system	O
b	O
is	O
better	O
right	O
so	O
it	O
s	O
after	O
all	O
it	O
s	O
	O
.	O
and	O
then	O
this	O
is	O
twice	O
as	O
much	O
as	O
	O
.	O
	O
.	O
so	O
that	O
s	O
a	O
better	O
performance	O
.	O
but	O
if	O
you	O
look	O
at	O
these	O
two	O
experiments	O
and	O
look	O
at	O
the	O
detailed	O
results	O
you	O
will	O
see	O
that	O
we	O
ll	O
be	O
more	O
confident	O
to	O
say	O
that	O
in	O
the	O
case	O
one	O
.	O
in	O
experiment	O
one	O
.	O
in	O
this	O
case	O
because	O
these	O
numbers	O
seem	O
to	O
be	O
consistently	O
better	O
than	O
for	O
system	O
b	O
.	O
where	O
as	O
in	O
experiment	O
two	O
we	O
re	O
not	O
sure	O
.	O
because	O
looking	O
at	O
some	O
results	O
like	O
this	O
after	O
system	O
a	O
is	O
better	O
.	O
and	O
this	O
is	O
another	O
case	O
where	O
system	O
a	O
is	O
better	O
.	O
but	O
yet	O
if	O
we	O
look	O
at	O
on	O
the	O
average	O
system	O
b	O
is	O
better	O
.	O
so	O
what	O
do	O
you	O
think	O
you	O
know	O
how	O
reliable	O
is	O
our	O
conclusion	O
if	O
we	O
only	O
look	O
at	O
the	O
average	O
now	O
in	O
this	O
case	O
intuitively	O
we	O
feel	O
it	O
s	O
better	O
than	O
one	O
it	O
s	O
more	O
reliable	O
.	O
but	O
how	O
can	O
we	O
quantitatively	O
answer	O
this	O
question	O
and	O
this	O
is	O
why	O
we	O
need	O
to	O
do	O
statistical	B
significance	B
test	I
.	O
so	O
the	O
idea	O
of	O
a	O
statistical	B
significance	B
test	I
is	O
basically	O
to	O
assess	O
the	O
vary	O
variance	O
across	O
these	O
different	O
queries	O
.	O
if	O
there	O
s	O
a	O
a	O
big	O
variance	O
that	O
means	O
that	O
the	O
results	O
could	O
fluctuate	O
a	O
lot	O
according	O
to	O
different	O
queries	O
.	O
then	O
we	O
should	O
believe	O
that	O
unless	O
you	O
have	O
used	O
a	O
lot	O
of	O
queries	O
the	O
results	O
might	O
change	O
if	O
we	O
use	O
another	O
set	O
of	O
queries	O
.	O
right	O
so	O
this	O
is	O
then	O
not	O
so	O
if	O
you	O
have	O
seen	O
high	O
variance	O
then	O
it	O
s	O
not	O
very	O
reliable	O
.	O
so	O
let	O
s	O
look	O
at	O
these	O
results	O
again	O
in	O
the	O
second	O
case	O
.	O
so	O
here	O
we	O
show	O
two	O
different	O
ways	O
to	O
compare	O
them	O
.	O
one	O
is	O
a	O
sign	O
test	O
.	O
and	O
we	O
ll	O
we	O
ll	O
just	O
look	O
at	O
the	O
sign	O
.	O
if	O
system	O
b	O
is	O
better	O
than	O
system	O
a	O
then	O
we	O
have	O
a	O
plus	O
sign	O
.	O
when	O
system	O
a	O
is	O
better	O
we	O
have	O
a	O
minus	O
sign	O
etc	O
.	O
using	O
this	O
case	O
if	O
you	O
see	O
this	O
well	O
there	O
are	O
seven	O
cases	O
.	O
we	O
actually	O
have	O
four	O
cases	O
where	O
system	O
b	O
is	O
better	O
.	O
but	O
three	O
cases	O
system	O
a	O
is	O
better	O
.	O
you	O
know	O
intuitively	O
this	O
is	O
almost	O
like	O
random	O
results	O
.	O
right	O
so	O
if	O
you	O
just	O
take	O
a	O
random	O
sample	O
of	O
to	O
to	O
flip	O
seven	O
coins	O
and	O
if	O
you	O
use	O
plus	O
to	O
denote	O
the	O
head	O
and	O
then	O
minus	O
to	O
denote	O
the	O
tail	O
and	O
that	O
could	O
easily	O
be	O
the	O
results	O
of	O
just	O
randomly	O
flipping	O
these	O
seven	O
coins	O
.	O
so	O
the	O
fact	O
that	O
the	O
the	O
average	O
is	O
larger	O
doesn	O
t	O
tell	O
us	O
anything	O
.	O
you	O
know	O
we	O
can	O
t	O
reliably	O
concur	O
that	O
.	O
and	O
this	O
can	O
be	O
quantitative	O
in	O
the	O
measure	O
by	O
a	O
p	O
value	O
.	O
and	O
that	O
basically	O
means	O
the	O
probability	O
that	O
this	O
result	O
is	O
in	O
fact	O
from	O
random	O
fluctuation	O
.	O
in	O
this	O
case	O
probability	O
is	O
one	O
.	O
it	O
means	O
it	O
surely	O
is	O
a	O
random	O
fluctuation	O
.	O
now	O
in	O
wilcoxon	O
test	O
it	O
s	O
a	O
non	O
parametrical	O
test	O
and	O
we	O
would	O
be	O
not	O
only	O
looking	O
at	O
the	O
signs	O
we	O
ll	O
be	O
also	O
looking	O
at	O
the	O
magnitude	O
of	O
the	O
difference	O
.	O
but	O
we	O
we	O
we	O
can	O
draw	O
a	O
similar	O
conclusion	O
where	O
you	O
say	O
well	O
it	O
s	O
very	O
likely	O
to	O
be	O
from	O
random	O
.	O
so	O
to	O
illustrate	O
this	O
let	O
s	O
think	O
about	O
such	O
a	O
distribution	O
.	O
and	O
this	O
is	O
called	O
a	O
normal	O
distribution	O
.	O
we	O
assume	O
that	O
the	O
mean	O
is	O
zero	O
here	O
.	O
let	O
s	O
say	O
well	O
we	O
started	O
with	O
the	O
assumption	O
that	O
there	O
s	O
no	O
difference	O
between	O
the	O
two	O
systems	O
.	O
but	O
we	O
assume	O
that	O
because	O
of	O
random	O
fluctuations	O
depending	O
on	O
the	O
queries	O
we	O
might	O
observe	O
a	O
difference	O
so	O
the	O
actual	O
difference	O
might	O
be	O
on	O
the	O
left	O
side	O
here	O
or	O
on	O
the	O
right	O
side	O
here	O
right	O
and	O
and	O
this	O
curve	O
kind	O
of	O
shows	O
the	O
probability	O
that	O
we	O
would	O
actually	O
observe	O
values	O
that	O
are	O
deviating	O
from	O
zero	O
here	O
.	O
now	O
so	O
if	O
we	O
look	O
at	O
this	O
picture	O
then	O
we	O
see	O
that	O
if	O
a	O
difference	O
is	O
observed	O
here	O
then	O
the	O
chance	O
is	O
very	O
high	O
that	O
this	O
is	O
in	O
fact	O
a	O
random	O
observation	O
right	O
.	O
we	O
can	O
define	O
region	O
of	O
you	O
know	O
likely	O
observation	O
because	O
of	O
random	O
fluctuation	O
.	O
and	O
this	O
is	O
of	O
all	O
outcomes	O
.	O
and	O
in	O
this	O
interval	O
then	O
the	O
observed	O
values	O
may	O
still	O
be	O
from	O
random	O
fluctuation	O
.	O
but	O
if	O
you	O
observe	O
a	O
value	O
in	O
this	O
region	O
or	O
a	O
difference	O
on	O
this	O
side	O
then	O
the	O
difference	O
is	O
unlikely	O
from	O
random	O
fluctuation	O
.	O
right	O
so	O
there	O
is	O
a	O
very	O
small	O
probability	O
that	O
you	O
will	O
observe	O
such	O
a	O
difference	O
just	O
because	O
of	O
random	O
fluctuation	O
.	O
so	O
in	O
that	O
case	O
we	O
can	O
then	O
conclude	O
the	O
difference	O
must	O
be	O
real	O
.	O
so	O
system	O
b	O
is	O
indeed	O
better	O
.	O
so	O
this	O
is	O
the	O
idea	O
of	O
the	O
statistical	B
significance	B
test	I
.	O
the	O
takeaway	O
message	O
here	O
is	O
that	O
you	O
have	O
used	O
many	O
queries	O
to	O
avoid	O
jumping	O
into	O
a	O
conclusion	O
as	O
in	O
this	O
case	O
to	O
say	O
system	O
b	O
is	O
better	O
.	O
there	O
are	O
many	O
different	O
ways	O
of	O
doing	O
this	O
statistical	B
significance	B
test	I
.	O
so	O
now	O
let	O
s	O
talk	O
about	O
the	O
other	O
problem	O
of	O
making	O
judgements	O
and	O
as	O
we	O
said	O
earlier	O
it	O
s	O
very	O
hard	O
to	O
judge	O
all	O
the	O
documents	O
completely	O
unless	O
it	O
is	O
a	O
small	O
data	O
set	O
.	O
so	O
the	O
question	O
is	O
if	O
we	O
can	O
t	O
afford	O
judging	O
all	O
the	O
documents	O
in	O
the	O
collection	O
which	O
subset	O
should	O
we	O
judge	O
and	O
the	O
solution	O
here	O
is	O
pooling	O
.	O
and	O
this	O
is	O
a	O
strategy	O
that	O
has	O
been	O
used	O
in	O
many	O
cases	O
to	O
solve	O
this	O
problem	O
.	O
so	O
the	O
idea	O
of	O
pulling	O
is	O
the	O
following	O
.	O
we	O
would	O
first	O
choose	O
a	O
diverse	O
set	O
of	O
ranking	B
methods	I
these	O
are	O
types	O
of	O
retrieval	B
systems	I
.	O
and	O
we	O
hope	O
these	O
methods	O
can	O
help	O
us	O
nominate	O
likely	O
relevance	O
in	O
the	O
documents	O
.	O
so	O
the	O
goal	O
is	O
to	O
pick	O
out	O
the	O
relevant	O
documents.	O
.	O
it	O
means	O
we	O
are	O
to	O
make	O
judgements	O
on	O
relevant	B
documents	I
because	O
those	O
are	O
the	O
most	O
useful	O
documents	O
from	O
the	O
users	O
perspective	O
.	O
so	O
that	O
way	O
we	O
would	O
have	O
each	O
to	O
return	O
top-k	O
documents	O
.	O
and	O
the	O
k	O
can	O
vary	O
from	O
systems	O
right	O
.	O
but	O
the	O
point	O
is	O
to	O
ask	O
them	O
to	O
suggest	O
the	O
most	O
likely	O
relevant	B
documents	I
.	O
and	O
then	O
we	O
simply	O
combine	O
all	O
these	O
top-k	O
sets	O
to	O
form	O
a	O
pool	O
of	O
documents	O
for	O
human	O
assessors	O
to	O
judge	O
.	O
so	O
imagine	O
you	O
have	O
many	O
systems	O
.	O
each	O
will	O
return	O
k	O
documents	O
you	O
know	O
take	O
the	O
top-k	O
documents	O
and	O
we	O
form	O
the	O
unit	O
.	O
now	O
of	O
course	O
there	O
are	O
many	O
documents	O
that	O
are	O
duplicated	O
because	O
many	O
systems	O
might	O
have	O
retrieved	O
the	O
same	O
random	O
documents	O
.	O
so	O
there	O
will	O
be	O
some	O
duplicate	O
documents	O
.	O
and	O
there	O
are	O
there	O
are	O
also	O
unique	O
documents	O
that	O
are	O
only	O
returned	O
by	O
one	O
system	O
so	O
the	O
idea	O
of	O
having	O
diverse	O
set	O
of	O
result	O
ranking	B
methods	I
is	O
to	O
ensure	O
the	O
pool	O
is	O
broad	O
.	O
and	O
can	O
include	O
as	O
many	O
possible	O
random	O
documents	O
as	O
possible	O
.	O
and	O
then	O
the	O
users	O
with	O
the	O
human	O
assessors	O
would	O
make	O
complete	O
the	O
judgements	O
on	O
this	O
data	O
set	O
this	O
pool	O
.	O
and	O
the	O
other	O
unjudged	O
documents	O
are	O
usually	O
just	O
a	O
assumed	O
to	O
be	O
non-relevant	O
.	O
now	O
if	O
the	O
pool	O
is	O
large	O
enough	O
this	O
assumption	O
is	O
okay	O
.	O
but	O
the	O
if	O
the	O
pool	O
is	O
not	O
very	O
large	O
this	O
actually	O
has	O
to	O
be	O
reconsidered	O
and	O
we	O
might	O
use	O
other	O
strategies	O
to	O
deal	O
with	O
them	O
and	O
there	O
are	O
indeed	O
other	O
methods	O
to	O
handle	O
such	O
cases	O
.	O
and	O
such	O
a	O
strategy	O
is	O
generally	O
okay	O
for	O
comparing	O
systems	O
that	O
contribute	O
to	O
the	O
pool	O
.	O
that	O
means	O
if	O
you	O
participate	O
in	O
contributing	O
to	O
the	O
pool	O
then	O
it	O
s	O
unlikely	O
that	O
it	O
will	O
penalize	O
your	O
system	O
because	O
the	O
top	O
ranked	O
documents	O
have	O
all	O
been	O
judged	O
.	O
however	O
this	O
is	O
problematic	O
for	O
even	O
evaluating	O
a	O
new	O
system	O
that	O
may	O
not	O
have	O
contributed	O
to	O
the	O
pool	O
.	O
in	O
this	O
case	O
you	O
know	O
a	O
new	O
system	O
might	O
be	O
penalized	O
because	O
it	O
might	O
have	O
nominated	O
some	O
relevant	B
documents	I
that	O
have	O
not	O
been	O
judged	O
.	O
so	O
those	O
documents	O
might	O
be	O
assumed	O
to	O
be	O
non-relevant	O
.	O
and	O
that	O
that	O
s	O
unfair	O
.	O
so	O
to	O
summarize	O
the	O
whole	O
part	O
of	O
text	B
retrieval	I
evaluation	O
it	O
s	O
extremely	O
important	O
because	O
the	O
problem	O
is	O
an	O
empirically	O
defined	O
problem	O
.	O
if	O
we	O
don	O
t	O
rely	O
on	O
users	O
there	O
s	O
no	O
way	O
to	O
tell	O
whether	O
one	O
method	O
works	O
better	O
.	O
if	O
we	O
have	O
inappropriate	O
experiment	O
design	O
we	O
might	O
misguide	O
our	O
research	O
or	O
applications	O
.	O
and	O
we	O
might	O
just	O
draw	O
wrong	O
conclusions	O
.	O
and	O
we	O
have	O
seen	O
this	O
in	O
some	O
of	O
our	O
discussion	O
.	O
so	O
make	O
sure	O
to	O
get	O
it	O
right	O
for	O
your	O
research	O
or	O
application	O
.	O
the	O
main	O
methodology	O
is	O
cranfield	B
evaluation	B
methodology	I
and	O
this	O
is	O
near	O
the	O
main	O
paradigm	O
used	O
in	O
all	O
kinds	O
of	O
empirical	B
evaluation	I
tasks	O
not	O
just	O
a	O
search	B
engine	I
variation	O
.	O
map	B
and	O
ndcg	B
are	O
the	O
two	O
main	O
measures	O
that	O
should	O
definitely	O
know	O
about	O
and	O
they	O
are	O
appropriate	O
for	O
comparing	O
ranking	B
algorithms	I
.	O
you	O
will	O
see	O
them	O
often	O
in	O
research	O
papers	O
.	O
perceiving	O
up	O
to	O
ten	O
documents	O
is	O
easier	O
to	O
interpret	O
from	O
users	O
perspective	O
.	O
so	O
that	O
s	O
also	O
often	O
useful	O
.	O
what	O
s	O
not	O
covered	O
is	O
some	O
other	O
evaluation	O
strategy	O
like	O
a-b	O
test	O
where	O
the	O
system	O
would	O
mix	O
two	O
of	O
the	O
results	O
of	O
two	O
methods	O
randomly	O
.	O
and	O
then	O
will	O
show	O
the	O
mix	O
of	O
results	O
to	O
users	O
.	O
of	O
course	O
the	O
users	O
don	O
t	O
see	O
which	O
result	O
is	O
from	O
which	O
method	O
.	O
the	O
users	O
would	O
judge	O
those	O
results	O
or	O
click	O
on	O
those	O
those	O
documents	O
in	O
in	O
a	O
search	B
engine	I
application	O
.	O
in	O
this	O
case	O
then	O
the	O
search	B
engine	I
can	O
keep	O
track	O
of	O
the	O
clicked	O
documents	O
and	O
see	O
if	O
one	O
method	O
has	O
contributed	O
more	O
to	O
the	O
clicked	O
documents	O
.	O
if	O
the	O
user	O
tends	O
to	O
click	O
on	O
one	O
the	O
results	O
from	O
one	O
method	O
then	O
it	O
s	O
just	O
that	O
method	O
may	O
may	O
be	O
better	O
.	O
so	O
this	O
is	O
what	O
leverages	O
a	O
real	O
users	O
of	O
a	O
search	B
engine	I
to	O
do	O
evaluation	O
.	O
it	O
s	O
called	O
a-b	O
test	O
and	O
it	O
s	O
a	O
strategy	O
that	O
s	O
often	O
used	O
by	O
the	O
modern	O
search	B
engines	I
the	O
commercial	O
search	B
engines	I
.	O
another	O
way	O
to	O
evaluate	O
ir	O
or	O
text	B
retrieval	I
is	O
user	O
studies	O
and	O
we	O
haven	O
t	O
covered	O
that	O
.	O
i	O
ve	O
put	O
some	O
references	O
here	O
that	O
you	O
can	O
look	O
at	O
if	O
you	O
want	O
to	O
know	O
more	O
about	O
that	O
.	O
so	O
there	O
are	O
three	O
additional	O
readings	O
here	O
these	O
are	O
three	O
mini	O
books	O
about	O
evaluation	O
.	O
and	O
they	O
are	O
all	O
excellent	O
in	O
covering	O
a	O
broad	O
review	O
of	O
information	B
retrieval	I
and	O
evaluation	O
.	O
and	O
this	O
covered	O
some	O
of	O
the	O
things	O
that	O
we	O
discussed	O
.	O
but	O
they	O
also	O
have	O
a	O
lot	O
of	O
others	O
to	O
offer	O
.	O
this	O
lecture	O
is	O
about	O
the	O
probabilistic	B
retrieval	B
model	I
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
continue	O
the	O
discussion	O
of	O
text	B
retrieval	B
methods	I
.	O
we	O
re	O
going	O
to	O
look	O
at	O
another	O
kind	O
of	O
very	O
different	O
way	O
to	O
design	O
ranking	B
functions	I
then	O
the	O
vector	B
space	I
model	I
that	O
we	O
discussed	O
before	O
.	O
in	O
probabilistic	B
models	I
we	O
define	O
the	O
ranking	B
function	I
based	O
on	O
the	O
probability	O
that	O
this	O
document	O
is	O
random	O
to	O
this	O
query	O
.	O
in	O
other	O
words	O
we	O
are	O
we	O
introduced	O
a	O
binary	O
random	O
variable	O
here	O
.	O
this	O
is	O
the	O
variable	O
r	O
here	O
.	O
and	O
we	O
also	O
assume	O
that	O
the	O
query	O
and	O
the	O
documents	O
are	O
all	O
observations	O
from	O
random	O
variables	O
.	O
note	O
that	O
in	O
the	O
vector	B
space	I
model	I
we	O
assume	O
they	O
are	O
vectors	O
.	O
but	O
here	O
we	O
assumed	O
we	O
assumed	O
they	O
are	O
the	O
data	O
observed	O
from	O
random	O
variables	O
.	O
and	O
so	O
the	O
problem	O
model	O
retrieval	O
becomes	O
to	O
estimate	O
the	O
probability	O
of	O
relevance	O
.	O
in	O
this	O
category	O
of	O
models	O
there	O
are	O
different	O
variants	O
.	O
the	O
classic	O
probabilistic	B
model	I
has	O
led	O
to	O
the	O
bm	O
retrieval	O
function	O
which	O
we	O
discussed	O
in	O
the	O
vector	B
space	I
model	I
because	O
it	O
s	O
form	O
is	O
actually	O
similar	O
to	O
a	O
vector	B
space	I
model	I
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
discuss	O
another	O
subclass	O
in	O
this	O
big	O
class	O
called	O
a	O
language	B
modeling	I
approaches	O
to	O
retrieval	O
.	O
in	O
particular	O
we	O
re	O
going	O
to	O
discuss	O
the	O
query	O
likelihood	O
retrieval	B
model	I
which	O
is	O
one	O
of	O
the	O
most	O
effective	O
models	O
in	O
probabilistic	B
models	I
.	O
there	O
is	O
also	O
another	O
line	O
called	O
a	O
divergence-from-randomness	O
model	O
which	O
has	O
latitude	O
the	O
pl	O
function	O
.	O
it	O
s	O
also	O
one	O
of	O
the	O
most	O
effective	O
state	O
of	O
the	O
art	O
attribute	O
functions	O
.	O
in	O
query	O
likelihood	O
our	O
assumption	O
is	O
that	O
this	O
probability	O
readiness	O
can	O
be	O
approximated	O
by	O
the	O
probability	O
of	O
query	O
given	O
a	O
document	O
and	O
readiness	O
.	O
so	O
intuitively	O
this	O
probability	O
just	O
captures	O
the	O
following	O
probability	O
.	O
and	O
that	O
is	O
if	O
a	O
user	O
likes	O
document	O
d	O
how	O
likely	O
would	O
the	O
user	O
enter	O
query	O
q	O
in	O
order	O
to	O
retrieve	O
document	O
d	O
.	O
so	O
we	O
ll	O
assume	O
that	O
the	O
user	O
likes	O
d	O
because	O
we	O
have	O
a	O
relevance	O
value	O
here	O
.	O
and	O
the	O
we	O
ask	O
the	O
question	O
about	O
how	O
likely	O
we	O
will	O
see	O
this	O
particular	O
query	O
from	O
this	O
user	O
so	O
this	O
is	O
the	O
basic	O
idea	O
.	O
now	O
to	O
understand	O
this	O
idea	O
let	O
s	O
take	O
a	O
look	O
at	O
the	O
general	O
idea	O
or	O
the	O
basic	O
idea	O
of	O
probabilistic	B
retrieval	B
models	I
.	O
so	O
here	O
i	O
listed	O
some	O
imagined	O
relevance	O
status	O
values	O
or	O
relevance	B
judgments	I
of	O
queries	O
and	O
documents	O
.	O
for	O
example	O
in	O
this	O
slide	O
it	O
shows	O
that	O
query	O
one	O
is	O
a	O
query	O
that	O
the	O
user	O
typed	O
in	O
and	O
d	O
is	O
a	O
document	O
the	O
user	O
has	O
seen	O
and	O
one	O
means	O
the	O
user	O
thinks	O
d	O
is	O
relevant	O
to	O
to	O
q	O
	O
.	O
so	O
this	O
r	O
here	O
can	O
be	O
also	O
approximated	O
by	O
the	O
clicks	O
little	O
data	O
that	O
the	O
search	B
engine	I
can	O
collect	O
it	O
by	O
watching	O
how	O
you	O
interact	O
with	O
the	O
search	O
results	O
.	O
so	O
in	O
this	O
case	O
let	O
s	O
say	O
the	O
user	O
clicked	O
on	O
this	O
document	O
so	O
there	O
s	O
a	O
one	O
here	O
.	O
similarly	O
the	O
user	O
clicked	O
on	O
d	O
also	O
so	O
there	O
s	O
a	O
one	O
here	O
.	O
in	O
other	O
words	O
d	O
is	O
assumed	O
to	O
relevant	O
at	O
two	O
q	O
	O
.	O
on	O
the	O
other	O
hand	O
d	O
is	O
non	O
relevant	O
there	O
s	O
a	O
zero	O
here	O
.	O
and	O
d	O
is	O
non-relevant	O
and	O
then	O
d	O
is	O
again	O
relevant	O
and	O
so	O
on	O
and	O
so	O
forth	O
.	O
and	O
this	O
part	O
of	O
maybe	O
they	O
are	O
collected	O
from	O
a	O
different	O
user	O
.	O
right	O
.	O
so	O
this	O
user	O
typed	O
in	O
q	O
and	O
then	O
found	O
that	O
d	O
is	O
actually	O
not	O
useful	O
so	O
d	O
is	O
actually	O
non-relevant	O
.	O
in	O
contrast	O
here	O
we	O
see	O
it	O
s	O
relevant	O
and	O
or	O
this	O
could	O
be	O
the	O
same	O
query	O
typing	O
by	O
the	O
same	O
user	O
at	O
different	O
times	O
but	O
d	O
is	O
also	O
relevant	O
et	O
cetera	O
.	O
and	O
then	O
here	O
we	O
can	O
see	O
more	O
data	O
that	O
about	O
other	O
queries	O
.	O
now	O
we	O
can	O
imagine	O
we	O
have	O
a	O
lot	O
of	O
search	O
data	O
.	O
now	O
we	O
can	O
ask	O
the	O
question	O
how	O
can	O
we	O
then	O
estimated	O
the	O
probability	O
of	O
relevance	O
right	O
.	O
so	O
how	O
can	O
we	O
compute	O
this	O
probability	O
of	O
relevance	O
well	O
intuitively	O
that	O
just	O
means	O
if	O
we	O
look	O
at	O
the	O
all	O
the	O
entries	O
where	O
we	O
see	O
this	O
particular	O
d	O
and	O
this	O
particular	O
q	O
how	O
likely	O
will	O
we	O
see	O
a	O
one	O
on	O
the	O
third	O
column	O
basically	O
that	O
just	O
means	O
we	O
can	O
correct	O
the	O
counts	O
.	O
we	O
can	O
first	O
count	O
how	O
many	O
times	O
where	O
we	O
see	O
q	O
and	O
d	O
as	O
a	O
pair	O
in	O
this	O
table	O
and	O
then	O
count	O
how	O
many	O
times	O
we	O
actually	O
have	O
also	O
seen	O
one	O
in	O
the	O
third	O
column	O
and	O
then	O
we	O
just	O
compute	O
the	O
ratio	O
.	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
some	O
specific	O
examples	O
.	O
suppose	O
we	O
are	O
trying	O
to	O
computed	O
this	O
probability	O
for	O
d	O
d	O
and	O
d	O
for	O
q	O
	O
.	O
what	O
is	O
the	O
estimated	O
probability	O
now	O
think	O
about	O
that	O
.	O
you	O
can	O
pause	O
the	O
video	O
if	O
needed	O
.	O
try	O
to	O
take	O
a	O
look	O
at	O
the	O
table	O
and	O
try	O
to	O
give	O
your	O
estimate	O
of	O
the	O
probability	O
.	O
have	O
you	O
seen	O
that	O
if	O
we	O
are	O
interested	O
in	O
q	O
and	O
d	O
we	O
ve	O
been	O
looking	O
at	O
the	O
these	O
two	O
pairs	O
and	O
in	O
both	O
cases	O
or	O
actually	O
in	O
one	O
of	O
the	O
cases	O
the	O
user	O
has	O
said	O
that	O
this	O
is	O
one	O
this	O
is	O
relevant	O
.	O
so	O
r	O
is	O
equal	O
to	O
in	O
only	O
one	O
of	O
the	O
two	O
cases	O
.	O
in	O
the	O
other	O
case	O
this	O
is	O
zero	O
.	O
so	O
that	O
s	O
one	O
out	O
of	O
two	O
.	O
what	O
about	O
the	O
d	O
and	O
the	O
d	O
well	O
they	O
re	O
are	O
here	O
you	O
want	O
d	O
d	O
d	O
	O
.	O
in	O
both	O
cases	O
in	O
this	O
case	O
r	O
is	O
equal	O
to	O
	O
.	O
so	O
it	O
s	O
two	O
out	O
of	O
two	O
and	O
so	O
and	O
so	O
forth	O
.	O
so	O
you	O
can	O
see	O
with	O
this	O
approach	O
we	O
captured	O
it	O
score	O
these	O
documents	O
for	O
the	O
query	O
.	O
right	O
we	O
now	O
have	O
a	O
score	O
for	O
d	O
d	O
and	O
d	O
for	O
this	O
query	O
.	O
we	O
can	O
simply	O
ranked	O
them	O
based	O
on	O
these	O
probabilities	O
and	O
so	O
that	O
s	O
the	O
basic	O
idea	O
of	O
probabilistic	B
retrieval	B
model	I
.	O
and	O
you	O
can	O
see	O
it	O
makes	O
a	O
lot	O
of	O
sense	O
.	O
in	O
this	O
case	O
it	O
s	O
going	O
to	O
rank	O
d	O
above	O
all	O
the	O
other	O
documents	O
.	O
because	O
in	O
all	O
the	O
cases	O
when	O
you	O
have	O
seen	O
q	O
and	O
d	O
r	O
is	O
equal	O
to	O
	O
.	O
the	O
user	O
clicked	O
on	O
this	O
document	O
.	O
so	O
this	O
also	O
showed	O
showed	O
that	O
with	O
a	O
lot	O
of	O
click	O
through	O
data	O
a	O
search	B
engine	I
can	O
learn	O
a	O
lot	O
from	O
the	O
data	O
to	O
improve	O
the	O
search	B
engine	I
.	O
this	O
is	O
a	O
simple	O
example	O
that	O
shows	O
that	O
with	O
even	O
a	O
small	O
number	O
of	O
entries	O
here	O
we	O
can	O
already	O
estimate	O
some	O
probabilities	O
.	O
these	O
probabilities	O
would	O
give	O
us	O
some	O
sense	O
about	O
which	O
document	O
might	O
be	O
more	O
read	O
or	O
more	O
useful	O
to	O
a	O
user	O
for	O
typing	O
this	O
query	O
.	O
now	O
of	O
course	O
the	O
problem	O
is	O
that	O
we	O
don	O
t	O
observe	O
all	O
the	O
queries	O
and	O
all	O
of	O
the	O
documents	O
and	O
all	O
the	O
relevance	O
values	O
.	O
right	O
there	O
will	O
be	O
a	O
lot	O
of	O
unseen	O
documents	O
.	O
in	O
general	O
we	O
can	O
only	O
collect	O
data	O
from	O
the	O
document	O
s	O
that	O
we	O
have	O
shown	O
to	O
the	O
users	O
.	O
there	O
are	O
even	O
more	O
unseen	O
queries	O
because	O
you	O
cannot	O
predict	O
what	O
queries	O
will	O
be	O
typed	O
in	O
by	O
users	O
.	O
so	O
obviously	O
this	O
approach	O
won	O
t	O
work	O
if	O
we	O
apply	O
it	O
to	O
unseen	O
queries	O
or	O
unseen	O
documents	O
.	O
nevertheless	O
this	O
shows	O
the	O
basic	O
idea	O
of	O
the	O
probabilistic	B
retrieval	B
model	I
and	O
it	O
makes	O
sense	O
intuitively	O
.	O
so	O
what	O
do	O
we	O
do	O
in	O
such	O
a	O
case	O
when	O
we	O
have	O
a	O
lot	O
of	O
unseen	O
documents	O
and	O
and	O
unseen	O
queries	O
well	O
the	O
solutions	O
that	O
we	O
have	O
to	O
approximate	O
in	O
some	O
way	O
.	O
right	O
.	O
so	O
in	O
this	O
particular	O
case	O
called	O
the	O
query	O
likelihood	O
retrieval	B
model	I
we	O
just	O
approximate	O
this	O
by	O
another	O
conditional	B
probability	I
p	O
q	O
d	O
r	O
is	O
equal	O
to	O
	O
.	O
so	O
in	O
the	O
condition	O
part	O
we	O
assume	O
that	O
the	O
user	O
likes	O
the	O
document	O
because	O
we	O
have	O
seen	O
that	O
the	O
user	O
clicked	O
on	O
this	O
document	O
.	O
and	O
this	O
part	O
shows	O
that	O
we	O
re	O
interested	O
in	O
how	O
likely	O
the	O
user	O
would	O
actually	O
enter	O
this	O
query	O
.	O
how	O
likely	O
we	O
will	O
see	O
this	O
query	O
in	O
the	O
same	O
row	O
.	O
so	O
note	O
that	O
here	O
we	O
have	O
made	O
an	O
interesting	O
assumption	O
here	O
.	O
basically	O
we	O
we	O
re	O
going	O
to	O
assume	O
that	O
whether	O
the	O
user	O
types	O
in	O
this	O
query	O
has	O
something	O
to	O
do	O
with	O
whether	O
user	O
likes	O
the	O
document	O
.	O
in	O
other	O
words	O
we	O
actually	O
make	O
the	O
foreign	O
assumption	O
and	O
that	O
is	O
a	O
user	O
formula	O
to	O
query	O
based	O
on	O
an	O
imaginary	O
relevant	O
document	O
.	O
well	O
if	O
you	O
just	O
look	O
at	O
this	O
as	O
a	O
conditional	B
probability	I
it	O
s	O
not	O
obvious	O
we	O
are	O
making	O
this	O
assumption	O
.	O
so	O
what	O
i	O
really	O
meant	O
is	O
that	O
to	O
use	O
this	O
new	O
conditional	B
probability	I
to	O
help	O
us	O
score	O
then	O
this	O
new	O
condition	O
of	O
probability	O
.	O
we	O
have	O
to	O
somehow	O
be	O
able	O
to	O
estimate	O
this	O
conditional	B
probability	I
without	O
relying	O
on	O
this	O
big	O
table	O
.	O
otherwise	O
it	O
would	O
be	O
having	O
similar	O
problems	O
as	O
before	O
.	O
and	O
by	O
making	O
this	O
assumption	O
we	O
have	O
some	O
way	O
to	O
bypass	O
this	O
big	O
table	O
and	O
try	O
to	O
just	O
mortar	O
how	O
to	O
use	O
a	O
formula	O
to	O
the	O
query	O
.	O
okay	O
.	O
so	O
this	O
is	O
how	O
you	O
can	O
simplify	O
the	O
the	O
general	O
model	O
so	O
that	O
we	O
can	O
give	O
either	O
specific	O
function	O
later	O
.	O
so	O
let	O
s	O
look	O
at	O
how	O
this	O
model	O
works	O
for	O
our	O
example	O
.	O
and	O
basically	O
what	O
we	O
are	O
going	O
to	O
do	O
in	O
this	O
case	O
is	O
to	O
ask	O
the	O
following	O
question	O
.	O
which	O
of	O
these	O
documents	O
is	O
most	O
likely	O
the	O
imaginary	O
relevant	O
document	O
in	O
the	O
user	O
s	O
mind	O
when	O
the	O
user	O
formulates	O
this	O
query	O
and	O
so	O
we	O
ask	O
this	O
question	O
and	O
we	O
quantify	O
the	O
probability	O
and	O
this	O
probability	O
is	O
a	O
conditional	B
probability	I
of	O
observing	O
this	O
query	O
if	O
a	O
particular	O
document	O
is	O
in	O
fact	O
the	O
imaginary	O
relevant	O
document	O
in	O
the	O
user	O
s	O
mind	O
.	O
here	O
you	O
can	O
see	O
we	O
compute	O
all	O
these	O
query	O
likelihood	O
probabilities	O
the	O
likelihood	O
of	O
queries	O
given	O
each	O
document	O
.	O
once	O
we	O
have	O
these	O
values	O
we	O
can	O
then	O
rank	O
these	O
documents	O
based	O
on	O
these	O
values	O
.	O
so	O
to	O
summarize	O
the	O
general	O
idea	O
of	O
modern	O
relevance	O
in	O
the	O
probability	O
risk	O
model	O
is	O
to	O
assume	O
that	O
we	O
introduce	O
a	O
binary	O
random	O
variable	O
r	O
here	O
.	O
and	O
then	O
let	O
the	O
scoring	O
function	O
be	O
defined	O
based	O
on	O
this	O
conditional	B
probability	I
.	O
we	O
also	O
talked	O
about	O
a	O
proximate	O
in	O
this	O
by	O
using	O
the	O
query	O
likelihood	O
.	O
and	O
in	O
this	O
case	O
we	O
have	O
a	O
ranking	B
function	I
that	O
s	O
basically	O
based	O
on	O
a	O
probability	O
of	O
a	O
query	O
given	O
the	O
document	O
.	O
and	O
this	O
probability	O
should	O
be	O
interpreted	O
as	O
the	O
probability	O
that	O
a	O
user	O
who	O
likes	O
document	O
d	O
would	O
pose	O
query	O
q	O
.	O
now	O
the	O
question	O
of	O
course	O
is	O
how	O
do	O
we	O
compute	O
this	O
additional	O
probability	O
at	O
this	O
in	O
general	O
has	O
to	O
do	O
with	O
how	O
to	O
compute	O
the	O
probability	O
of	O
text	O
because	O
q	O
is	O
a	O
text	O
.	O
and	O
this	O
has	O
to	O
do	O
with	O
a	O
model	O
called	O
a	O
language	B
model	I
.	O
and	O
this	O
kind	O
of	O
models	O
are	O
proposed	O
to	O
model	O
text	O
.	O
so	O
most	O
specifically	O
we	O
will	O
be	O
very	O
interested	O
in	O
the	O
following	O
conditional	B
probability	I
as	O
i	O
show	O
you	O
you	O
this	O
here	O
.	O
if	O
the	O
user	O
like	O
this	O
document	O
how	O
likely	O
the	O
user	O
would	O
approve	O
this	O
query	O
and	O
in	O
the	O
next	O
lecture	O
we	O
re	O
going	O
to	O
give	O
introduction	O
to	O
language	B
model	I
so	O
that	O
we	O
can	O
see	O
how	O
we	O
can	O
model	O
text	O
with	O
a	O
probability	O
risk	O
model	O
in	O
general	O
.	O
	O
.	O
this	O
lecture	O
is	O
about	O
a	O
statistical	B
language	B
model	I
.	O
in	O
this	O
lecture	O
we	O
re	O
go	O
we	O
re	O
going	O
to	O
get	O
an	O
introduction	O
to	O
the	O
probabilistic	B
model	I
.	O
this	O
has	O
to	O
do	O
with	O
how	O
many	O
models	O
have	O
to	O
go	O
into	O
these	O
models	O
.	O
so	O
it	O
s	O
ready	O
to	O
how	O
we	O
model	O
theory	O
based	O
on	O
a	O
document	O
.	O
we	O
re	O
going	O
to	O
talk	O
about	O
what	O
is	O
a	O
language	B
model	I
and	O
then	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
simplest	O
language	B
model	I
called	O
a	O
unigram	O
language	B
model	I
.	O
which	O
also	O
happens	O
to	O
be	O
the	O
most	O
useful	O
model	O
for	O
text	B
retrieval	I
.	O
and	O
finally	O
we	O
ll	O
discuss	O
possible	O
uses	O
of	O
an	O
m	O
model	O
.	O
what	O
is	O
a	O
language	B
model	I
well	O
it	O
s	O
just	O
a	O
probability	O
distribution	O
over	O
word	O
sequences	O
.	O
so	O
here	O
i	O
show	O
one	O
.	O
this	O
model	O
gives	O
the	O
sequence	O
today	O
s	O
wednesday	O
a	O
probability	O
of	O
	O
.	O
it	O
gives	O
today	O
wednesday	O
is	O
a	O
very	O
very	O
small	O
probability	O
because	O
it	O
s	O
algorithmatical	O
.	O
you	O
can	O
see	O
the	O
probabilities	O
given	O
to	O
these	O
sentences	O
or	O
sequences	O
of	O
words	O
can	O
vary	O
a	O
lot	O
depending	O
on	O
the	O
model	O
.	O
therefore	O
it	O
s	O
clearly	O
context-dependent	O
.	O
in	O
ordinary	O
conversation	O
probably	O
today	O
is	O
wednesday	O
is	O
most	O
popular	O
among	O
these	O
sentences	O
.	O
but	O
imagine	O
in	O
the	O
context	O
of	O
discussing	O
a	O
private	O
math	O
maybe	O
the	O
higher	O
values	O
positive	O
would	O
have	O
a	O
higher	O
probability	O
.	O
this	O
means	O
it	O
can	O
be	O
used	O
to	O
represent	O
as	O
a	O
topic	O
of	O
a	O
test	O
.	O
the	O
model	O
can	O
also	O
be	O
regarded	O
as	O
a	O
probabilistic	O
mechanism	O
for	O
generating	O
text	O
and	O
this	O
is	O
why	O
it	O
is	O
often	O
called	O
a	O
generating	O
model	O
.	O
so	O
what	O
does	O
that	O
mean	O
we	O
can	O
image	O
this	O
is	O
a	O
mechanism	O
that	O
s	O
visualized	O
here	O
as	O
a	O
system	O
that	O
can	O
generate	O
a	O
sequences	O
of	O
words	O
.	O
so	O
we	O
can	O
ask	O
for	O
a	O
sequence	O
and	O
it	O
s	O
to	O
sample	O
a	O
sequence	O
from	O
the	O
device	O
if	O
you	O
want	O
.	O
and	O
it	O
might	O
generate	O
for	O
example	O
today	O
is	O
wednesday	O
but	O
it	O
could	O
have	O
generated	O
many	O
other	O
sequences	O
.	O
so	O
for	O
example	O
there	O
are	O
many	O
possibilities	O
right	O
so	O
this	O
in	O
this	O
sense	O
we	O
can	O
view	O
our	O
data	O
as	O
basically	O
a	O
sample	O
observed	O
from	O
such	O
a	O
generated	O
model	O
.	O
so	O
why	O
is	O
such	O
a	O
model	O
useful	O
well	O
it	O
s	O
mainly	O
because	O
it	O
can	O
quantify	O
the	O
uncertainties	O
in	O
natural	B
language	I
.	O
where	O
do	O
uncertainties	O
come	O
from	O
well	O
one	O
source	O
is	O
simply	O
the	O
ambiguity	O
in	O
natural	B
language	I
that	O
we	O
discussed	O
earlier	O
in	O
the	O
lecture	O
.	O
another	O
source	O
is	O
because	O
we	O
don	O
t	O
have	O
complete	O
understanding	O
.	O
we	O
lack	O
all	O
the	O
knowledge	O
to	O
understand	O
language	O
.	O
in	O
that	O
case	O
there	O
will	O
be	O
uncertainties	O
as	O
well	O
.	O
so	O
let	O
me	O
show	O
some	O
examples	O
of	O
questions	O
that	O
we	O
can	O
answer	O
with	O
an	O
average	O
model	O
that	O
would	O
have	O
an	O
interesting	O
application	O
in	O
different	O
ways	O
.	O
given	O
that	O
we	O
see	O
john	O
and	O
feels	O
.	O
how	O
likely	O
will	O
we	O
see	O
happy	O
as	O
opposed	O
to	O
habit	O
as	O
the	O
next	O
word	O
in	O
a	O
sequence	O
of	O
words	O
obviously	O
this	O
would	O
be	O
very	O
useful	O
speech	O
recognition	O
because	O
happy	O
and	O
habit	O
would	O
have	O
similar	O
acoustical	O
sound	O
.	O
acoustic	O
signals	O
.	O
but	O
if	O
we	O
look	O
at	O
the	O
language	B
model	I
we	O
know	O
that	O
john	O
feels	O
happy	O
would	O
be	O
far	O
more	O
likely	O
than	O
john	O
feels	O
habit	O
.	O
another	O
example	O
given	O
that	O
we	O
observe	O
baseball	O
three	O
times	O
and	O
gained	O
once	O
in	O
the	O
news	O
article	O
how	O
likely	O
is	O
it	O
about	O
the	O
sports	O
this	O
obviously	O
is	O
related	O
to	O
text	O
categorization	O
and	O
information	O
.	O
also	O
given	O
that	O
a	O
user	O
is	O
interested	O
in	O
sports	O
news	O
how	O
likely	O
would	O
the	O
user	O
use	O
baseball	O
in	O
a	O
query	O
now	O
this	O
is	O
clearly	O
related	O
to	O
the	O
query	O
that	O
we	O
discussed	O
in	O
the	O
previous	O
lecture	O
.	O
so	O
now	O
let	O
s	O
look	O
at	O
the	O
simplest	O
language	B
model	I
.	O
called	O
a	O
lan	O
unigram	O
language	B
model	I
.	O
in	O
such	O
a	O
case	O
we	O
assume	O
that	O
we	O
generate	O
the	O
text	O
by	O
generating	O
each	O
word	O
independently	O
.	O
so	O
this	O
means	O
the	O
probability	O
of	O
a	O
sequence	O
of	O
words	O
would	O
be	O
then	O
the	O
product	O
of	O
the	O
probability	O
of	O
each	O
word	O
.	O
now	O
normally	O
they	O
are	O
not	O
independent	O
right	O
so	O
if	O
you	O
have	O
seen	O
a	O
word	O
like	O
language	O
.	O
now	O
we	O
ll	O
make	O
it	O
far	O
more	O
likely	O
to	O
observe	O
model	O
than	O
if	O
you	O
haven	O
t	O
seen	O
language	O
.	O
so	O
this	O
assumption	O
is	O
not	O
necessary	O
sure	O
but	O
we	O
ll	O
make	O
this	O
assumption	O
to	O
simplify	O
the	O
model	O
.	O
so	O
now	O
the	O
model	O
has	O
precisely	O
n	O
parameters	O
where	O
n	O
is	O
vocabulary	O
size	O
.	O
we	O
have	O
one	O
probability	O
for	O
each	O
word	O
and	O
all	O
these	O
probabilities	O
must	O
sum	O
to	O
	O
.	O
so	O
strictly	O
speaking	O
we	O
actually	O
have	O
n	O
minus	O
parameters	O
.	O
as	O
i	O
said	O
text	O
can	O
be	O
then	O
be	O
assumed	O
to	O
be	O
a	O
sample	O
drawn	O
from	O
this	O
word	O
distribution	O
.	O
so	O
for	O
example	O
now	O
we	O
can	O
ask	O
the	O
device	O
or	O
the	O
model	O
to	O
stochastically	O
generate	O
the	O
words	O
for	O
us	O
instead	O
of	O
in	O
sequences	O
.	O
so	O
instead	O
of	O
giving	O
a	O
whole	O
sequence	O
like	O
today	O
is	O
wednesday	O
it	O
now	O
gives	O
us	O
just	O
one	O
word	O
.	O
and	O
we	O
can	O
get	O
all	O
kinds	O
of	O
words	O
.	O
and	O
we	O
can	O
assemble	O
these	O
words	O
in	O
a	O
sequence	O
.	O
so	O
that	O
would	O
still	O
allows	O
you	O
to	O
compute	O
the	O
probability	O
of	O
today	O
is	O
wed	O
as	O
the	O
product	O
of	O
the	O
three	O
probabilities	O
.	O
as	O
you	O
can	O
see	O
even	O
though	O
we	O
have	O
not	O
asked	O
the	O
model	O
to	O
generate	O
the	O
the	O
sequences	O
it	O
actually	O
allows	O
us	O
to	O
compute	O
the	O
probability	O
for	O
all	O
the	O
sequences	O
.	O
but	O
this	O
model	O
now	O
only	O
needs	O
n	O
parameters	O
to	O
characterize	O
.	O
that	O
means	O
if	O
we	O
specify	O
all	O
the	O
probabilities	O
for	O
all	O
the	O
words	O
then	O
the	O
model	O
s	O
behavior	O
is	O
completely	O
specified	O
.	O
whereas	O
if	O
you	O
we	O
don	O
t	O
make	O
this	O
assumption	O
we	O
would	O
have	O
to	O
specify	O
.	O
find	O
probabilities	O
for	O
all	O
kinds	O
of	O
combinations	O
of	O
words	O
in	O
sequences	O
.	O
so	O
by	O
making	O
this	O
assumption	O
it	O
makes	O
it	O
much	O
easier	O
to	O
estimate	O
these	O
parameters	O
.	O
so	O
let	O
s	O
see	O
a	O
specific	O
example	O
here	O
.	O
here	O
i	O
show	O
two	O
unigram	O
lambda	O
models	O
with	O
some	O
probabilities	O
and	O
these	O
are	O
high	O
probability	O
words	O
that	O
are	O
shown	O
on	O
top	O
.	O
the	O
first	O
one	O
clearly	O
suggests	O
the	O
topic	O
of	O
text	O
mining	O
because	O
the	O
high	O
probability	O
words	O
are	O
all	O
related	O
to	O
this	O
topic	O
.	O
the	O
second	O
one	O
is	O
more	O
related	O
to	O
health	O
.	O
now	O
we	O
can	O
then	O
ask	O
the	O
question	O
how	O
likely	O
we	O
ll	O
observe	O
a	O
particular	O
text	O
from	O
each	O
of	O
these	O
three	O
models	O
.	O
now	O
suppose	O
with	O
sample	O
words	O
to	O
form	O
the	O
document	O
let	O
s	O
say	O
we	O
take	O
the	O
first	O
distribution	O
which	O
are	O
the	O
sample	O
words	O
.	O
what	O
words	O
do	O
you	O
think	O
it	O
would	O
be	O
generated	O
or	O
maybe	O
text	O
or	O
maybe	O
mining	O
maybe	O
another	O
word	O
even	O
food	O
which	O
has	O
a	O
very	O
small	O
probability	O
might	O
still	O
be	O
able	O
to	O
show	O
up	O
.	O
but	O
in	O
general	O
high	O
probability	O
words	O
will	O
likely	O
show	O
up	O
more	O
often	O
.	O
so	O
we	O
can	O
imagine	O
a	O
generated	O
text	O
that	O
looks	O
like	O
text	O
mining	O
.	O
a	O
factor	O
with	O
a	O
small	O
probability	O
you	O
might	O
be	O
able	O
to	O
actually	O
generate	O
the	O
actual	O
text	O
mining	O
paper	O
that	O
would	O
actually	O
be	O
meaningful	O
although	O
the	O
probability	O
would	O
be	O
very	O
very	O
small	O
.	O
in	O
the	O
extreme	O
case	O
you	O
might	O
imagine	O
we	O
might	O
be	O
able	O
to	O
generate	O
a	O
a	O
text	O
paper	O
text	O
mining	O
paper	O
that	O
would	O
be	O
accepted	O
by	O
a	O
major	O
conference	O
.	O
and	O
in	O
that	O
case	O
the	O
probability	O
would	O
be	O
even	O
smaller	O
.	O
for	O
instance	O
nonzero	O
probability	O
if	O
we	O
assume	O
none	O
of	O
the	O
words	O
will	O
have	O
a	O
nonzero	O
probability	O
.	O
similarly	O
from	O
the	O
second	O
topic	O
we	O
can	O
imagine	O
we	O
can	O
generate	O
a	O
food	O
and	O
nutrition	O
paper	O
.	O
that	O
doesn	O
t	O
mean	O
we	O
cannot	O
generate	O
this	O
paper	O
from	O
text	O
mining	O
distribution	O
.	O
we	O
can	O
but	O
the	O
probability	O
would	O
be	O
very	O
very	O
small	O
maybe	O
smaller	O
than	O
even	O
generating	O
a	O
paper	O
that	O
can	O
be	O
accepted	O
by	O
a	O
major	O
conference	O
on	O
text	O
mining	O
.	O
so	O
the	O
point	O
of	O
here	O
is	O
that	O
given	O
a	O
distribution	O
we	O
can	O
talk	O
about	O
the	O
probability	O
of	O
observing	O
a	O
certain	O
kind	O
of	O
text	O
.	O
some	O
text	O
would	O
have	O
higher	O
probabilities	O
than	O
others	O
.	O
now	O
let	O
s	O
look	O
at	O
the	O
problem	O
in	O
a	O
different	O
way	O
.	O
supposedly	O
we	O
now	O
have	O
available	O
a	O
particular	O
document	O
.	O
in	O
this	O
case	O
maybe	O
the	O
abstract	O
or	O
the	O
text	O
mining	O
paper	O
and	O
we	O
see	O
these	O
word	O
accounts	O
here	O
.	O
the	O
total	O
number	O
of	O
words	O
is	O
	O
.	O
now	O
the	O
question	O
you	O
ask	O
here	O
is	O
a	O
estimation	O
question	O
.	O
we	O
can	O
ask	O
the	O
question	O
which	O
model	O
which	O
word	O
distribution	O
has	O
been	O
used	O
to	O
to	O
generate	O
this	O
text	O
.	O
assuming	O
the	O
text	O
has	O
been	O
generated	O
by	O
assembling	O
words	O
from	O
the	O
distribution	O
.	O
so	O
what	O
would	O
be	O
your	O
guess	O
what	O
have	O
to	O
decide	O
what	O
probabilities	O
test	O
mining	O
et	O
cetera	O
would	O
have	O
.	O
so	O
pause	O
a	O
view	O
for	O
a	O
second	O
and	O
try	O
to	O
think	O
about	O
your	O
best	O
guess	O
.	O
if	O
you	O
re	O
like	O
a	O
lot	O
of	O
people	O
you	O
would	O
have	O
guessed	O
that	O
well	O
my	O
best	O
guess	O
is	O
text	O
has	O
a	O
probability	O
of	O
out	O
of	O
because	O
i	O
have	O
seen	O
text	O
ten	O
times	O
and	O
there	O
are	O
a	O
total	O
of	O
words	O
.	O
so	O
we	O
simply	O
noticed	O
normalize	O
these	O
counts	O
.	O
and	O
that	O
s	O
in	O
fact	O
justified	O
.	O
and	O
your	O
intuition	O
is	O
consistent	O
with	O
mathematical	O
derivation	O
.	O
and	O
this	O
is	O
called	O
a	O
maximum	B
likelihood	I
	O
.	O
in	O
this	O
estimator	O
we	O
ll	O
assume	O
that	O
the	O
parameter	O
settings	O
	O
.	O
are	O
those	O
that	O
would	O
give	O
our	O
observer	O
the	O
maximum	O
probability	O
.	O
that	O
means	O
if	O
we	O
change	O
these	O
probabilities	O
then	O
the	O
probability	O
of	O
observing	O
the	O
particular	O
text	O
would	O
be	O
somewhat	O
smaller	O
.	O
so	O
we	O
can	O
see	O
this	O
has	O
a	O
very	O
simple	O
formula	O
.	O
basically	O
we	O
just	O
need	O
to	O
look	O
at	O
the	O
count	O
of	O
a	O
word	O
in	O
the	O
document	O
and	O
then	O
divide	O
it	O
by	O
the	O
total	O
number	O
of	O
words	O
in	O
the	O
document	O
.	O
about	O
the	O
length	O
.	O
normalize	O
the	O
frequency	O
.	O
well	O
a	O
consequence	O
of	O
this	O
is	O
of	O
course	O
we	O
re	O
going	O
to	O
assign	O
probabilities	O
to	O
unseen	O
words	O
.	O
if	O
we	O
have	O
an	O
observed	O
word	O
there	O
will	O
be	O
no	O
incentive	O
to	O
assign	O
a	O
non-	O
probability	O
using	O
this	O
approach	O
.	O
why	O
because	O
that	O
would	O
take	O
away	O
probability	O
mass	O
for	O
this	O
observed	O
words	O
.	O
and	O
that	O
obviously	O
wouldn	O
t	O
maximize	O
the	O
probability	O
of	O
this	O
particular	O
observed	O
data	O
.	O
but	O
one	O
can	O
still	O
question	O
whether	O
this	O
is	O
our	O
best	O
estimator	O
.	O
well	O
the	O
answer	O
depends	O
on	O
what	O
kind	O
of	O
model	O
you	O
want	O
to	O
find	O
right	O
this	O
is	O
made	O
if	O
it	O
s	O
a	O
best	O
model	O
based	O
on	O
this	O
particular	O
layer	O
.	O
but	O
if	O
you	O
re	O
interested	O
in	O
a	O
model	O
that	O
can	O
explain	O
the	O
content	O
of	O
the	O
four	O
paper	O
of	O
for	O
this	O
abstract	O
then	O
you	O
might	O
have	O
a	O
second	O
thought	O
right	O
so	O
for	O
one	O
thing	O
there	O
should	O
be	O
other	O
things	O
in	O
the	O
body	O
of	O
that	O
article	O
.	O
so	O
they	O
should	O
not	O
have	O
zero	O
probabilities	O
even	O
though	O
they	O
are	O
not	O
observing	O
the	O
abstract	O
.	O
we	O
re	O
going	O
to	O
cover	O
this	O
later	O
in	O
discussing	O
the	O
query	B
model	I
.	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
some	O
possible	O
uses	O
of	O
these	O
language	B
models	I
.	O
one	O
use	O
is	O
simply	O
to	O
use	O
it	O
to	O
represent	O
the	O
topics	O
.	O
so	O
here	O
it	O
shows	O
some	O
general	O
english	O
background	O
that	O
text	O
.	O
we	O
can	O
use	O
this	O
text	O
to	O
estimate	O
a	O
language	B
model	I
.	O
and	O
the	O
model	O
might	O
look	O
like	O
this	O
.	O
right	O
so	O
on	O
the	O
top	O
we	O
ll	O
have	O
those	O
all	O
common	O
words	O
is	O
we	O
is	O
and	O
then	O
we	O
ll	O
see	O
some	O
common	O
words	O
like	O
these	O
and	O
then	O
some	O
very	O
very	O
real	O
words	O
in	O
the	O
bottom	O
.	O
this	O
is	O
the	O
background	O
image	O
model	O
.	O
it	O
represents	O
the	O
frequency	O
on	O
words	O
in	O
english	O
in	O
general	O
right	O
this	O
is	O
the	O
background	O
model	O
.	O
now	O
let	O
s	O
look	O
at	O
another	O
text	O
.	O
maybe	O
this	O
time	O
we	O
ll	O
look	O
at	O
computer	O
science	O
research	O
papers	O
.	O
so	O
we	O
have	O
a	O
correction	O
of	O
computer	O
science	O
research	O
papers	O
we	O
do	O
again	O
we	O
can	O
just	O
use	O
the	O
maximum	O
where	O
we	O
simply	O
normalize	O
the	O
frequencies	O
.	O
now	O
in	O
this	O
case	O
we	O
look	O
at	O
the	O
distribution	O
that	O
looks	O
like	O
this	O
.	O
on	O
the	O
top	O
it	O
looks	O
similar	O
because	O
these	O
words	O
occur	O
everywhere	O
they	O
are	O
very	O
common	O
.	O
but	O
as	O
we	O
go	O
down	O
we	O
ll	O
see	O
words	O
that	O
are	O
more	O
related	O
to	O
computer	O
science	O
.	O
computer	O
or	O
software	O
or	O
text	O
et	O
cetera	O
.	O
so	O
although	O
here	O
we	O
might	O
also	O
see	O
these	O
words	O
for	O
example	O
computer	O
.	O
but	O
we	O
can	O
imagine	O
the	O
probability	O
here	O
is	O
much	O
smaller	O
than	O
the	O
probability	O
here	O
.	O
and	O
we	O
will	O
see	O
many	O
other	O
words	O
here	O
that	O
that	O
would	O
be	O
more	O
common	O
in	O
general	O
in	O
english	O
.	O
so	O
you	O
can	O
see	O
this	O
distribution	O
characterizes	O
a	O
topic	O
of	O
the	O
corresponding	O
text	O
.	O
we	O
can	O
look	O
at	O
the	O
even	O
the	O
smaller	O
text	O
.	O
so	O
in	O
this	O
case	O
let	O
s	O
look	O
at	O
the	O
text	O
mining	O
paper	O
.	O
now	O
if	O
we	O
do	O
the	O
same	O
we	O
have	O
another	O
.	O
distribution	O
again	O
the	O
can	O
be	O
expected	O
to	O
occur	O
on	O
the	O
top	O
.	O
soon	O
we	O
will	O
see	O
text	O
mining	O
association	O
clustering	O
these	O
words	O
have	O
relatively	O
high	O
probabilities	O
in	O
contrast	O
in	O
this	O
distribution	O
has	O
relatively	O
small	O
probability	O
.	O
so	O
this	O
means	O
again	O
based	O
on	O
different	O
text	O
data	O
that	O
we	O
can	O
have	O
a	O
different	O
model	O
.	O
and	O
model	O
captures	O
the	O
topic	O
.	O
so	O
we	O
call	O
this	O
document	O
an	O
lm	B
model	O
and	O
we	O
call	O
this	O
collection	O
lm	B
model	O
.	O
and	O
later	O
we	O
ll	O
see	O
how	O
they	O
re	O
used	O
in	O
a	O
retrieval	O
function	O
.	O
but	O
now	O
let	O
s	O
look	O
at	O
the	O
another	O
use	O
of	O
this	O
model	O
.	O
can	O
we	O
statistically	O
find	O
what	O
words	O
are	O
semantically	O
related	O
to	O
computer	O
now	O
how	O
do	O
we	O
find	O
such	O
words	O
well	O
our	O
first	O
thought	O
is	O
well	O
let	O
s	O
take	O
a	O
look	O
at	O
the	O
text	O
that	O
match	O
.	O
computer	O
.	O
so	O
we	O
can	O
take	O
a	O
look	O
at	O
all	O
the	O
documents	O
that	O
contain	O
the	O
word	O
computer	O
.	O
let	O
s	O
build	O
a	O
language	B
model	I
.	O
okay	O
see	O
what	O
words	O
we	O
see	O
there	O
.	O
well	O
not	O
surprisingly	O
we	O
see	O
these	O
common	O
words	O
on	O
top	O
as	O
we	O
always	O
do	O
.	O
so	O
in	O
this	O
case	O
this	O
language	B
model	I
gives	O
us	O
the	O
.	O
conditional	B
probability	I
of	O
seeing	O
a	O
word	O
in	O
the	O
context	O
of	O
computer	O
.	O
and	O
these	O
common	O
words	O
will	O
naturally	O
have	O
high	O
probabilities	O
.	O
other	O
words	O
will	O
see	O
computer	O
itself	O
and	O
software	O
will	O
have	O
relatively	O
high	O
probabilities	O
.	O
but	O
we	O
if	O
we	O
just	O
use	O
this	O
model	O
we	O
cannot	O
.	O
i	O
just	O
say	O
all	O
these	O
words	O
are	O
semantically	O
related	O
to	O
computer	O
.	O
so	O
intuitively	O
what	O
we	O
d	O
like	O
to	O
get	O
rid	O
of	O
these	O
these	O
common	O
words	O
.	O
how	O
can	O
we	O
do	O
that	O
it	O
turns	O
out	O
that	O
it	O
s	O
possible	O
to	O
use	O
language	B
model	I
to	O
do	O
that	O
.	O
now	O
i	O
suggest	O
you	O
think	O
about	O
that	O
.	O
so	O
how	O
can	O
we	O
know	O
what	O
words	O
are	O
very	O
common	O
so	O
that	O
we	O
want	O
to	O
kind	O
of	O
get	O
rid	O
of	O
them	O
.	O
what	O
model	O
will	O
tell	O
us	O
that	O
well	O
maybe	O
you	O
can	O
think	O
about	O
that	O
.	O
so	O
the	O
background	O
language	B
model	I
precisely	O
tells	O
us	O
this	O
information	O
.	O
it	O
tells	O
us	O
what	O
words	O
are	O
common	O
in	O
general	O
.	O
so	O
if	O
we	O
use	O
this	O
background	O
model	O
we	O
would	O
know	O
that	O
these	O
words	O
are	O
common	O
words	O
in	O
general	O
.	O
so	O
it	O
s	O
not	O
surprising	O
to	O
observe	O
them	O
in	O
the	O
context	O
of	O
computer	O
.	O
whereas	O
computer	O
has	O
a	O
very	O
small	O
probability	O
in	O
general	O
.	O
so	O
it	O
s	O
very	O
surprising	O
that	O
we	O
have	O
seen	O
computer	O
in	O
with	O
this	O
probability	O
.	O
and	O
the	O
same	O
is	O
true	O
for	O
software	O
.	O
so	O
then	O
we	O
can	O
use	O
these	O
two	O
models	O
to	O
somehow	O
figure	O
out	O
.	O
the	O
words	O
that	O
are	O
related	O
to	O
computer	O
.	O
for	O
example	O
we	O
can	O
simply	O
take	O
the	O
ratio	O
of	O
these	O
two	O
probabilities	O
and	O
normalize	O
the	O
top	O
of	O
the	O
model	O
by	O
the	O
probability	O
of	O
the	O
word	O
in	O
the	O
background	O
model	O
.	O
so	O
if	O
we	O
do	O
that	O
we	O
take	O
the	O
ratio	O
we	O
ll	O
see	O
that	O
then	O
on	O
the	O
top	O
computer	O
is	O
ramped	O
and	O
then	O
followed	O
by	O
software	O
program	O
all	O
these	O
words	O
related	O
to	O
computer	O
.	O
because	O
they	O
occur	O
very	O
frequently	O
in	O
the	O
context	O
of	O
computer	O
but	O
not	O
frequently	O
in	O
whole	O
connection	O
.	O
where	O
as	O
these	O
common	O
words	O
will	O
not	O
have	O
a	O
high	O
probability	O
.	O
in	O
fact	O
they	O
have	O
a	O
ratio	O
of	O
about	O
one	O
down	O
there	O
.	O
because	O
they	O
are	O
not	O
really	O
related	O
to	O
computer	O
.	O
by	O
taking	O
the	O
same	O
ball	O
of	O
text	O
that	O
contains	O
the	O
computer	O
we	O
don	O
t	O
really	O
see	O
more	O
occurrences	O
of	O
that	O
in	O
general	O
.	O
so	O
this	O
shows	O
that	O
even	O
with	O
this	O
simple	O
lm	B
models	O
we	O
can	O
do	O
some	O
limited	O
analysis	O
of	O
semantics	O
.	O
so	O
in	O
this	O
lecture	O
we	O
talked	O
about	O
language	B
model	I
which	O
is	O
basically	O
a	O
probability	O
distribution	O
over	O
the	O
text	O
.	O
we	O
talked	O
about	O
the	O
simplistic	O
language	B
model	I
called	O
unigram	O
language	B
model	I
.	O
which	O
is	O
also	O
just	O
a	O
word	O
distribution	O
.	O
we	O
talked	O
about	O
the	O
two	O
uses	O
of	O
a	O
language	B
model	I
.	O
one	O
is	O
to	O
represent	O
the	O
the	O
topic	O
in	O
a	O
document	O
in	O
a	O
classing	O
or	O
in	O
general	O
.	O
the	O
other	O
is	O
discover	O
word	O
associations	O
.	O
in	O
the	O
next	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
how	O
language	B
model	I
can	O
be	O
used	O
to	O
design	O
a	O
retrieval	O
function	O
.	O
here	O
are	O
two	O
additional	O
readings	O
.	O
the	O
first	O
is	O
a	O
textbook	O
on	O
statistical	O
and	O
natural	B
language	I
processing	I
.	O
the	O
second	O
is	O
a	O
article	O
that	O
has	O
a	O
survey	O
of	O
statistical	B
language	B
models	I
with	O
other	O
pointers	O
to	O
research	O
work	O
.	O
this	O
lecture	O
is	O
about	O
query	O
likelihood	O
and	O
probabilistic	B
retrieval	B
model	I
.	O
in	O
this	O
lecture	O
we	O
continue	O
the	O
discussion	O
of	O
probabilistic	B
retrieval	B
model	I
.	O
in	O
particular	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
query	O
likelihood	O
of	O
the	O
retrieval	O
function	O
.	O
in	O
the	O
query	O
of	O
likelihood	O
retrieval	B
model	I
our	O
idea	O
is	O
a	O
model	O
.	O
how	O
a	O
likely	O
a	O
user	O
who	O
likes	O
a	O
document	O
would	O
pose	O
a	O
particular	O
query	O
.	O
so	O
in	O
this	O
case	O
you	O
can	O
imagine	O
if	O
a	O
user	O
likes	O
this	O
particular	O
document	O
about	O
the	O
presidential	O
campaign	O
news	O
.	O
then	O
we	O
can	O
assume	O
the	O
user	O
would	O
use	O
this	O
working	O
as	O
a	O
basis	O
to	O
oppose	O
a	O
query	O
to	O
try	O
and	O
retrieve	O
this	O
doc	O
.	O
so	O
you	O
can	O
imagine	O
the	O
user	O
could	O
use	O
a	O
process	O
that	O
works	O
as	O
follows	O
where	O
we	O
assume	O
that	O
the	O
query	O
is	O
generated	O
by	O
sampling	O
words	O
from	O
the	O
document	O
.	O
so	O
for	O
example	O
a	O
user	O
might	O
pick	O
a	O
word	O
like	O
presidential	O
from	O
this	O
document	O
and	O
then	O
use	O
this	O
as	O
a	O
query	O
word	O
.	O
and	O
then	O
the	O
user	O
would	O
pick	O
another	O
word	O
like	O
campaign	O
and	O
that	O
would	O
be	O
the	O
second	O
query	O
word	O
.	O
now	O
this	O
of	O
course	O
is	O
assumption	O
that	O
we	O
have	O
made	O
about	O
how	O
a	O
user	O
would	O
post	O
a	O
query	O
.	O
whether	O
a	O
user	O
actually	O
followed	O
this	O
process	O
.	O
maybe	O
a	O
different	O
question	O
.	O
but	O
this	O
assumption	O
has	O
allowed	O
us	O
to	O
formally	O
characterize	O
this	O
conditional	B
probability	I
.	O
and	O
this	O
allows	O
to	O
also	O
not	O
rely	O
on	O
the	O
big	O
table	O
that	O
i	O
showed	O
you	O
earlier	O
to	O
use	O
imperative	O
data	O
to	O
estimate	O
this	O
probability	O
.	O
and	O
this	O
is	O
why	O
we	O
can	O
use	O
this	O
idea	O
to	O
then	O
further	O
derive	O
retrieval	O
function	O
that	O
we	O
can	O
implement	O
with	O
the	O
languages	O
.	O
so	O
as	O
you	O
see	O
the	O
assumption	O
that	O
we	O
ve	O
made	O
here	O
is	O
each	O
query	O
word	O
is	O
independent	O
in	O
this	O
sample	O
and	O
also	O
each	O
word	O
is	O
basically	O
obtained	O
from	O
the	O
document	O
.	O
so	O
now	O
let	O
s	O
see	O
how	O
this	O
works	O
exactly	O
.	O
well	O
since	O
we	O
are	O
computing	O
a	O
query	O
likelihood	O
then	O
the	O
probability	O
here	O
is	O
just	O
the	O
probability	O
of	O
this	O
particular	O
query	O
which	O
is	O
a	O
sequence	O
of	O
words	O
.	O
and	O
we	O
make	O
the	O
assumption	O
that	O
each	O
word	O
is	O
generated	O
independently	O
.	O
so	O
as	O
a	O
result	O
the	O
probability	O
of	O
the	O
query	O
is	O
just	O
a	O
product	O
of	O
the	O
probability	O
of	O
each	O
query	O
word	O
.	O
now	O
how	O
do	O
we	O
compute	O
the	O
probability	O
of	O
each	O
query	O
word	O
well	O
based	O
on	O
the	O
assumption	O
that	O
a	O
word	O
is	O
picked	O
from	O
the	O
document	O
that	O
the	O
user	O
has	O
in	O
mind	O
.	O
now	O
we	O
know	O
the	O
probability	O
of	O
each	O
word	O
is	O
just	O
the	O
the	O
relative	O
frequency	O
of	O
the	O
word	O
in	O
the	O
document	O
.	O
so	O
for	O
example	O
the	O
probability	O
of	O
presidential	O
given	O
the	O
document	O
would	O
be	O
just	O
the	O
count	O
of	O
presidential	O
in	O
the	O
document	O
divided	O
by	O
the	O
total	O
number	O
of	O
words	O
in	O
the	O
document	O
or	O
document	O
length	O
.	O
so	O
with	O
this	O
these	O
assumptions	O
we	O
now	O
have	O
actual	O
simple	O
formula	O
for	O
retrieval	O
right	O
we	O
can	O
use	O
this	O
to	O
rank	O
our	O
document	O
.	O
so	O
does	O
this	O
model	O
work	O
let	O
s	O
take	O
a	O
look	O
here	O
are	O
some	O
example	O
documents	O
that	O
you	O
have	O
seen	O
before	O
.	O
suppose	O
now	O
the	O
query	O
is	O
presidential	O
campaign	O
.	O
and	O
we	O
see	O
the	O
formula	O
here	O
on	O
the	O
top	O
.	O
so	O
how	O
do	O
we	O
score	O
these	O
documents	O
well	O
it	O
s	O
very	O
simple	O
right	O
we	O
just	O
count	O
how	O
many	O
times	O
we	O
have	O
seen	O
presidential	O
how	O
many	O
times	O
we	O
have	O
seen	O
campaign	O
etc	O
.	O
and	O
see	O
here	O
and	O
we	O
ve	O
seen	O
president	O
jou	O
tai	O
so	O
that	O
s	O
two	O
over	O
the	O
lands	O
of	O
document	O
the	O
four	O
.	O
multiply	O
by	O
over	O
lands	O
of	O
document	O
of	O
for	O
the	O
probability	O
of	O
campaign	O
and	O
seeming	O
we	O
can	O
probabilities	O
for	O
the	O
other	O
two	O
documents	O
.	O
now	O
if	O
you	O
ll	O
look	O
at	O
this	O
these	O
numbers	O
or	O
these	O
this	O
these	O
formulas	O
for	O
scoring	O
all	O
these	O
documents	O
it	O
seems	O
to	O
make	O
sense	O
because	O
if	O
we	O
assume	O
d	O
and	O
d	O
have	O
about	O
the	O
same	O
length	O
then	O
it	O
looks	O
like	O
we	O
will	O
rank	O
d	O
above	O
d	O
and	O
which	O
is	O
above	O
d	O
right	O
and	O
as	O
we	O
would	O
expect	O
looks	O
like	O
it	O
did	O
capture	O
the	O
tf	B
heuristic	O
.	O
and	O
so	O
this	O
seems	O
to	O
work	O
well	O
.	O
however	O
if	O
we	O
try	O
a	O
different	O
query	O
like	O
this	O
one	O
presidential	O
campaign	O
update	O
then	O
we	O
might	O
see	O
a	O
problem	O
.	O
but	O
what	O
problem	O
well	O
think	O
about	O
update	O
now	O
none	O
of	O
these	O
documents	O
has	O
mentioned	O
update	O
.	O
so	O
according	O
to	O
our	O
assumption	O
that	O
a	O
user	O
would	O
pick	O
a	O
order	O
from	O
a	O
document	O
to	O
generate	O
a	O
query	O
then	O
the	O
probability	O
of	O
obtaining	O
a	O
word	O
like	O
update	O
would	O
be	O
what	O
.	O
would	O
be	O
zero	O
right	O
so	O
that	O
cause	O
a	O
problem	O
because	O
it	O
would	O
cause	O
all	O
these	O
documents	O
to	O
have	O
zero	O
probability	O
of	O
generating	O
this	O
query	O
.	O
now	O
while	O
it	O
s	O
fine	O
to	O
have	O
a	O
zero	O
probability	O
for	O
d	O
which	O
is	O
not	O
relevant	O
.	O
it	O
s	O
not	O
okay	O
to	O
have	O
zero	O
for	O
d	O
and	O
d	O
because	O
now	O
we	O
no	O
longer	O
can	O
distinguish	O
them	O
.	O
what	O
s	O
worse	O
we	O
can	O
t	O
even	O
distinguish	O
them	O
from	O
d	O
	O
.	O
all	O
right	O
so	O
that	O
s	O
obviously	O
not	O
desirable	O
.	O
now	O
when	O
one	O
has	O
such	O
result	O
we	O
should	O
think	O
about	O
what	O
has	O
caused	O
this	O
problem	O
.	O
so	O
we	O
have	O
to	O
examine	O
what	O
assumptions	O
have	O
been	O
made	O
as	O
we	O
derive	O
this	O
ranking	B
function	I
.	O
now	O
if	O
you	O
examine	O
those	O
assumptions	O
carefully	O
you	O
would	O
realize	O
.	O
what	O
has	O
caused	O
this	O
problem	O
right	O
so	O
take	O
a	O
moment	O
to	O
think	O
about	O
what	O
do	O
you	O
think	O
is	O
the	O
reason	O
why	O
update	O
has	O
zero	O
probability	O
and	O
how	O
do	O
we	O
fix	O
it	O
right	O
so	O
if	O
you	O
think	O
about	O
this	O
for	O
the	O
moment	O
that	O
you	O
realize	O
that	O
.	O
that	O
s	O
because	O
we	O
have	O
made	O
an	O
assumption	O
that	O
every	O
query	O
word	O
must	O
be	O
drawn	O
from	O
the	O
document	O
in	O
the	O
user	O
s	O
mind	O
.	O
so	O
in	O
order	O
to	O
fix	O
this	O
we	O
have	O
to	O
assume	O
that	O
the	O
user	O
could	O
have	O
drawn	O
a	O
word	O
not	O
necessarily	O
from	O
the	O
document	O
.	O
so	O
let	O
s	O
see	O
improved	O
model	O
.	O
an	O
improvement	O
here	O
is	O
to	O
say	O
that	O
well	O
instead	O
of	O
drawing	O
a	O
word	O
from	O
the	O
document	O
let	O
s	O
imagine	O
that	O
the	O
user	O
would	O
actually	O
draw	O
a	O
word	O
from	O
a	O
document	O
model	O
and	O
so	O
i	O
show	O
a	O
model	O
here	O
.	O
here	O
we	O
assume	O
that	O
this	O
document	O
is	O
generated	O
by	O
using	O
this	O
unigram	O
image	O
model	O
.	O
now	O
this	O
model	O
doesn	O
t	O
necessarily	O
assign	O
zero	O
probability	O
for	O
update	O
.	O
in	O
fact	O
we	O
assume	O
this	O
model	O
does	O
not	O
assign	O
zero	O
probability	O
for	O
any	O
word	O
.	O
now	O
if	O
we	O
re	O
thinking	O
this	O
way	O
then	O
the	O
generation	O
process	O
is	O
a	O
little	O
bit	O
different	O
.	O
now	O
the	O
user	O
has	O
this	O
model	O
in	O
mind	O
instead	O
of	O
this	O
particular	O
document	O
.	O
although	O
the	O
model	O
has	O
to	O
be	O
estimated	O
based	O
on	O
the	O
document	O
.	O
so	O
the	O
user	O
can	O
again	O
generate	O
the	O
query	O
using	O
a	O
similar	O
process	O
.	O
they	O
may	O
pick	O
a	O
word	O
for	O
example	O
presidential	O
and	O
another	O
word	O
campaign	O
.	O
now	O
the	O
difference	O
is	O
that	O
this	O
time	O
we	O
can	O
also	O
pick	O
a	O
word	O
like	O
update	O
even	O
though	O
update	O
it	O
doesn	O
t	O
occur	O
in	O
the	O
document	O
to	O
potentially	O
generate	O
a	O
query	O
word	O
like	O
update	O
.	O
so	O
that	O
a	O
query	O
was	O
updated	O
we	O
want	O
to	O
have	O
zero	O
probabilities	O
.	O
so	O
this	O
would	O
fix	O
our	O
problem	O
and	O
it	O
s	O
also	O
reasonable	O
because	O
we	O
re	O
now	O
thinking	O
of	O
what	O
the	O
user	O
is	O
looking	O
for	O
in	O
a	O
more	O
general	O
way	O
that	O
is	O
unique	O
language	B
model	I
instead	O
of	O
a	O
fixed	O
document	O
.	O
so	O
how	O
do	O
we	O
compute	O
this	O
query	O
like	O
if	O
we	O
make	O
this	O
sum	O
where	O
it	O
involves	O
two	O
steps	O
right	O
the	O
first	O
is	O
the	O
computer	O
s	O
model	O
and	O
we	O
call	O
it	O
talking	O
the	O
language	B
model	I
here	O
.	O
for	O
example	O
i	O
have	O
shown	O
two	O
possible	O
energy	O
models	O
here	O
.	O
this	O
has	O
been	O
based	O
on	O
two	O
documents	O
.	O
and	O
then	O
given	O
a	O
query	O
and	O
i	O
get	O
a	O
mining	O
algorithms	O
.	O
the	O
second	O
step	O
is	O
just	O
to	O
compute	O
the	O
likelihood	O
of	O
this	O
query	O
.	O
and	O
by	O
making	O
independence	O
assumptions	O
we	O
could	O
then	O
have	O
this	O
probability	O
as	O
a	O
product	O
of	O
the	O
probability	O
of	O
each	O
query	O
word	O
all	O
right	O
but	O
we	O
do	O
this	O
for	O
both	O
documents	O
.	O
and	O
then	O
we	O
re	O
going	O
to	O
score	O
these	O
two	O
documents	O
and	O
then	O
rank	O
them	O
.	O
so	O
that	O
s	O
the	O
basic	O
idea	O
of	O
this	O
query	O
likelihood	O
retrieval	O
function	O
.	O
so	O
more	O
generally	O
than	O
this	O
ranking	B
function	I
would	O
look	O
like	O
the	O
following	O
and	O
here	O
as	O
we	O
assume	O
that	O
query	O
has	O
end	O
words	O
w	O
through	O
wn	O
.	O
and	O
then	O
the	O
scoring	O
function	O
the	O
ranking	B
function	I
is	O
the	O
probability	O
that	O
we	O
observe	O
this	O
query	O
given	O
that	O
the	O
user	O
is	O
thinking	O
of	O
this	O
document	O
.	O
and	O
this	O
assumed	O
to	O
be	O
product	O
of	O
probabilities	O
of	O
all	O
individual	O
words	O
and	O
this	O
is	O
based	O
on	O
the	O
independence	O
assumption	O
.	O
now	O
we	O
actually	O
often	O
score	O
the	O
document	O
for	O
this	O
query	O
by	O
using	O
log	O
of	O
the	O
query	O
likelihood	O
as	O
shown	O
on	O
the	O
sigma	O
line	O
.	O
now	O
we	O
do	O
this	O
to	O
avoid	O
having	O
a	O
lot	O
of	O
small	O
probabilities	O
.	O
m	O
multiplied	O
together	O
.	O
and	O
this	O
could	O
cause	O
underflow	O
and	O
we	O
might	O
lose	O
precision	B
by	O
transforming	O
the	O
value	O
as	O
a	O
logarithm	O
function	O
.	O
we	O
maintain	O
the	O
order	O
of	O
these	O
documents	O
yet	O
we	O
can	O
avoid	O
the	O
end	O
of	O
flow	O
problem	O
.	O
so	O
if	O
we	O
take	O
longer	O
than	O
transformation	O
of	O
coarse	O
the	O
product	O
that	O
would	O
become	O
a	O
sum	O
as	O
you	O
stake	O
in	O
the	O
line	O
here	O
.	O
so	O
it	O
s	O
a	O
sum	O
of	O
all	O
of	O
the	O
query	B
words	I
and	O
inside	O
the	O
sum	O
that	O
is	O
log	O
of	O
the	O
probability	O
of	O
this	O
word	O
given	O
by	O
the	O
document	O
.	O
and	O
then	O
we	O
can	O
further	O
rewrite	O
the	O
sum	O
into	O
a	O
different	O
form	O
.	O
so	O
in	O
the	O
first	O
of	O
the	O
sum	O
here	O
in	O
this	O
sum	O
we	O
have	O
it	O
over	O
all	O
the	O
query	B
words	I
n	O
query	B
words	I
.	O
and	O
in	O
this	O
sum	O
we	O
have	O
a	O
sum	O
of	O
all	O
the	O
possible	O
words	O
but	O
we	O
put	O
a	O
counter	O
here	O
of	O
each	O
word	O
in	O
the	O
query	O
.	O
essentially	O
we	O
are	O
only	O
considering	O
the	O
words	O
in	O
the	O
query	O
because	O
if	O
a	O
word	O
is	O
not	O
in	O
the	O
query	O
it	O
can	O
would	O
be	O
zero	O
.	O
so	O
we	O
re	O
still	O
considering	O
only	O
these	O
end	O
words	O
.	O
but	O
we	O
re	O
using	O
a	O
different	O
form	O
as	O
if	O
we	O
were	O
going	O
to	O
a	O
sum	O
of	O
all	O
the	O
words	O
in	O
the	O
vocabulary	O
.	O
and	O
of	O
course	O
a	O
word	O
might	O
occur	O
multiple	O
times	O
in	O
the	O
query	O
.	O
that	O
s	O
wh	O
why	O
we	O
have	O
a	O
count	O
here	O
.	O
and	O
then	O
this	O
part	O
is	O
log	O
of	O
the	O
probability	O
of	O
the	O
word	O
given	O
by	O
the	O
document	O
mg	O
model	O
.	O
so	O
you	O
can	O
see	O
in	O
this	O
material	O
function	O
we	O
actually	O
know	O
the	O
count	O
of	O
the	O
word	O
in	O
the	O
query	O
.	O
so	O
the	O
only	O
thing	O
that	O
we	O
don	O
t	O
know	O
is	O
this	O
document	O
language	B
model	I
.	O
therefore	O
we	O
can	O
convert	O
through	O
the	O
retrieval	O
problem	O
into	O
the	O
problem	O
of	O
estimating	O
this	O
document	O
language	B
model	I
.	O
so	O
that	O
we	O
can	O
compute	O
the	O
probability	O
of	O
each	O
query	O
we	O
re	O
given	O
by	O
this	O
document	O
.	O
at	O
different	O
estimation	O
methods	O
here	O
would	O
lead	O
to	O
different	O
ranking	B
functions	I
.	O
and	O
this	O
is	O
just	O
like	O
a	O
different	O
a	O
ways	O
to	O
place	O
a	O
doc	O
in	O
the	O
vector	O
in	O
the	O
vector	B
space	I
.	O
would	O
lead	O
it	O
to	O
a	O
different	O
ranking	B
function	I
in	O
the	O
vector	B
space	I
model	I
.	O
here	O
are	O
different	O
ways	O
to	O
estimate	O
this	O
stuff	O
in	O
the	O
language	B
model	I
will	O
lead	O
you	O
to	O
a	O
different	O
ranking	B
function	I
for	O
query	O
likelihood	O
.	O
so	O
i	O
showed	O
you	O
how	O
we	O
rewrite	O
the	O
into	O
a	O
form	O
that	O
looks	O
like	O
a	O
the	O
formula	O
on	O
this	O
slide	O
.	O
after	O
we	O
make	O
the	O
assumption	O
about	O
smoothing	O
the	O
language	B
model	I
based	O
on	O
the	O
collection	O
of	O
the	O
language	B
model	I
.	O
now	O
if	O
we	O
look	O
at	O
the	O
this	O
rewriting	O
it	O
actually	O
would	O
give	O
us	O
two	O
benefits	O
.	O
the	O
first	O
benefit	O
is	O
it	O
helps	O
us	O
better	O
understand	O
that	O
this	O
ranking	B
function	I
.	O
in	O
particular	O
we	O
re	O
going	O
to	O
show	O
that	O
from	O
this	O
formula	O
we	O
can	O
see	O
smoothing	O
is	O
the	O
correction	O
that	O
we	O
model	O
will	O
give	O
us	O
something	O
like	O
a	O
tf-idf	B
weighting	O
and	O
length	B
normalization	I
.	O
the	O
second	O
benefit	O
is	O
that	O
it	O
also	O
allows	O
us	O
to	O
compute	O
the	O
query	O
likelihood	O
more	O
efficiently	O
.	O
in	O
particular	O
we	O
see	O
that	O
the	O
main	O
part	O
of	O
the	O
formula	O
is	O
a	O
sum	O
over	O
the	O
matching	O
query	B
terms	I
.	O
so	O
this	O
is	O
much	O
better	O
than	O
if	O
we	O
take	O
the	O
sum	O
over	O
all	O
the	O
words	O
.	O
after	O
we	O
smooth	O
the	O
document	O
the	O
language	B
model	I
we	O
essentially	O
have	O
nonzero	O
probabilities	O
for	O
all	O
the	O
words	O
.	O
so	O
this	O
new	O
form	O
of	O
the	O
formula	O
is	O
much	O
easier	O
to	O
score	O
or	O
to	O
compute	O
.	O
it	O
s	O
also	O
interesting	O
to	O
note	O
that	O
the	O
last	O
of	O
term	O
here	O
is	O
actually	O
independent	O
of	O
the	O
document	O
.	O
since	O
our	O
goal	O
is	O
to	O
rank	O
the	O
documents	O
for	O
the	O
same	O
query	O
we	O
can	O
ignore	O
this	O
term	O
for	O
ranking	O
.	O
because	O
it	O
s	O
going	O
to	O
be	O
the	O
same	O
for	O
all	O
the	O
documents	O
.	O
ignoring	O
it	O
wouldn	O
t	O
effect	O
the	O
order	O
of	O
the	O
documents	O
.	O
inside	O
the	O
sum	O
we	O
also	O
see	O
that	O
each	O
matched	O
query	B
term	I
would	O
contribute	O
a	O
weight	O
.	O
and	O
this	O
weight	O
actually	O
is	O
very	O
interesting	O
because	O
it	O
looks	O
like	O
tf-idf	B
weighting	O
.	O
first	O
we	O
can	O
already	O
see	O
it	O
has	O
a	O
frequency	O
of	O
the	O
word	O
in	O
the	O
query	O
just	O
like	O
in	O
the	O
vector	B
space	I
model	I
.	O
when	O
we	O
take	O
adult	O
product	O
we	O
see	O
the	O
word	B
frequency	I
in	O
the	O
query	O
to	O
show	O
up	O
in	O
such	O
a	O
sum	O
.	O
and	O
so	O
naturally	O
this	O
part	O
will	O
correspond	O
to	O
the	O
vector	O
element	O
from	O
the	O
document	O
vector	O
.	O
and	O
here	O
indeed	O
we	O
can	O
see	O
it	O
actually	O
encodes	O
a	O
weight	O
that	O
has	O
similar	O
factor	O
to	O
tf-idf	B
weighting	O
.	O
i	O
let	O
you	O
examine	O
it	O
.	O
can	O
you	O
see	O
it	O
can	O
you	O
see	O
which	O
part	O
is	O
capturing	O
tf	B
and	O
which	O
part	O
is	O
capturing	O
idf	B
weighting	O
so	O
if	O
you	O
want	O
you	O
can	O
pause	O
the	O
video	O
to	O
think	O
more	O
about	O
it	O
.	O
so	O
have	O
you	O
noticed	O
that	O
this	O
p	O
sub-seen	O
is	O
related	O
to	O
the	O
term	B
frequency	I
in	O
the	O
sense	O
that	O
if	O
a	O
word	O
occurs	O
very	O
frequently	O
in	O
the	O
document	O
then	O
the	O
s	O
probability	O
here	O
will	O
tend	O
to	O
be	O
larger	O
.	O
right	O
so	O
this	O
means	O
this	O
term	O
is	O
really	O
doing	O
something	O
like	O
tf	B
weighting	O
.	O
have	O
you	O
also	O
noticed	O
that	O
this	O
time	O
in	O
the	O
denominator	O
is	O
actually	O
achieving	O
the	O
factor	O
of	O
idf	B
why	O
because	O
this	O
is	O
the	O
popularity	O
of	O
the	O
term	O
in	O
the	O
collection	O
but	O
it	O
s	O
in	O
the	O
denominator	O
.	O
so	O
if	O
the	O
probability	O
in	O
the	O
collection	O
is	O
larger	O
than	O
the	O
weight	O
is	O
actually	O
smaller	O
.	O
and	O
this	O
means	O
a	O
popular	O
term	O
.	O
we	O
actually	O
have	O
a	O
smaller	O
weight	O
.	O
and	O
this	O
is	O
precisely	O
what	O
idf	B
weighting	O
is	O
doing	O
.	O
only	O
not	O
we	O
now	O
have	O
a	O
different	O
form	O
of	O
tf	B
and	O
idf	B
.	O
remember	O
idf	B
has	O
a	O
log	O
logarithm	O
of	O
document	B
frequency	I
but	O
here	O
we	O
have	O
something	O
different	O
.	O
but	O
intuitively	O
it	O
achieves	O
a	O
similar	O
effect	O
.	O
interestingly	O
we	O
also	O
have	O
something	O
related	O
to	O
the	O
length	B
normalization	I
.	O
again	O
can	O
you	O
see	O
which	O
factor	O
is	O
related	O
to	O
the	O
length	O
in	O
this	O
formula	O
.	O
well	O
i	O
just	O
say	O
that	O
that	O
this	O
term	O
is	O
related	O
to	O
idf	B
weighting	O
.	O
this	O
this	O
collection	O
probability	O
.	O
but	O
it	O
turns	O
out	O
this	O
term	O
here	O
is	O
actually	O
related	O
to	O
a	O
document	O
length	B
normalization	I
.	O
in	O
particular	O
d	O
might	O
be	O
related	O
to	O
document	O
n	O
length	O
.	O
so	O
it	O
it	O
encodes	O
how	O
much	O
probability	O
mass	O
we	O
want	O
to	O
give	O
to	O
unseen	O
words	O
.	O
how	O
much	O
smoothing	O
you	O
are	O
allowed	O
to	O
do	O
.	O
intuitively	O
if	O
a	O
document	O
is	O
long	O
then	O
we	O
need	O
to	O
do	O
less	O
smoothing	O
.	O
because	O
we	O
can	O
assume	O
that	O
it	O
is	O
large	O
enough	O
that	O
we	O
have	O
probably	O
observed	O
all	O
of	O
the	O
words	O
that	O
the	O
author	O
could	O
have	O
written	O
.	O
but	O
if	O
the	O
document	O
is	O
short	O
the	O
unseen	O
are	O
expected	O
to	O
be	O
to	O
be	O
large	O
and	O
we	O
need	O
to	O
do	O
more	O
smoothing	O
.	O
it	O
s	O
like	O
that	O
there	O
are	O
words	O
that	O
have	O
not	O
been	O
retained	O
yet	O
by	O
the	O
author	O
.	O
so	O
this	O
term	O
appears	O
to	O
paralyze	O
long	O
documents	O
tend	O
to	O
be	O
longer	O
than	O
larger	O
than	O
for	O
long	O
document	O
.	O
but	O
note	O
that	O
the	O
also	O
occurs	O
here	O
.	O
and	O
so	O
this	O
may	O
not	O
actually	O
be	O
necessary	O
penalizing	O
long	O
documents	O
and	O
in	O
fact	O
is	O
not	O
so	O
clear	O
here	O
.	O
but	O
as	O
we	O
will	O
see	O
later	O
when	O
we	O
consider	O
some	O
specific	O
smoothing	B
methods	I
it	O
turns	O
out	O
that	O
they	O
do	O
penalize	O
long	O
documents	O
.	O
just	O
like	O
in	O
tf-idf	B
weighting	O
and	O
the	O
document	O
ends	O
formulas	O
in	O
the	O
vector	B
space	I
model	I
.	O
so	O
that	O
s	O
a	O
very	O
interesting	O
observation	O
because	O
it	O
means	O
we	O
don	O
t	O
even	O
have	O
to	O
think	O
about	O
the	O
specific	O
way	O
of	O
doing	O
smoothing	O
.	O
we	O
just	O
need	O
to	O
assume	O
that	O
if	O
we	O
smooth	O
with	O
this	O
language	B
model	I
then	O
we	O
would	O
have	O
a	O
formula	O
that	O
looks	O
like	O
a	O
tf-idf	B
weighting	O
and	O
document	O
length	B
normalization	I
.	O
what	O
s	O
also	O
interesting	O
that	O
we	O
have	O
a	O
very	O
fixed	O
form	O
of	O
the	O
ranking	B
function	I
.	O
and	O
see	O
we	O
have	O
not	O
heuristically	O
put	O
a	O
logarithm	O
here	O
.	O
in	O
fact	O
if	O
you	O
can	O
think	O
about	O
why	O
we	O
would	O
have	O
a	O
logarithm	O
here	O
if	O
you	O
look	O
at	O
the	O
assumptions	O
that	O
we	O
have	O
made	O
it	O
will	O
be	O
clear	O
.	O
it	O
s	O
because	O
we	O
have	O
used	O
a	O
logarithm	O
of	O
query	O
likelihood	O
for	O
scoring	O
.	O
and	O
we	O
turned	O
the	O
product	O
into	O
a	O
sum	O
of	O
logarithm	O
of	O
probability	O
.	O
and	O
that	O
s	O
why	O
we	O
have	O
this	O
logarithm	O
.	O
note	O
that	O
if	O
we	O
only	O
want	O
to	O
heuristically	O
implement	O
a	O
tf	B
weighting	O
and	O
idf	B
weighting	O
we	O
don	O
t	O
necessarily	O
have	O
to	O
have	O
a	O
logarithm	O
here	O
.	O
imagine	O
if	O
we	O
drop	O
this	O
logarithm	O
we	O
would	O
still	O
have	O
tf	B
and	O
idf	B
weighting	O
.	O
but	O
what	O
s	O
nice	O
with	O
probabilistic	O
modeling	O
is	O
that	O
we	O
are	O
automatically	O
given	O
a	O
logarithm	O
function	O
here	O
.	O
and	O
that	O
s	O
basically	O
a	O
fixed	O
reform	O
of	O
the	O
formula	O
that	O
we	O
did	O
not	O
really	O
have	O
to	O
hueristically	O
line	O
.	O
and	O
in	O
this	O
case	O
if	O
you	O
try	O
to	O
drop	O
this	O
logarithm	O
the	O
model	O
probably	O
won	O
t	O
won	O
t	O
work	O
as	O
well	O
as	O
if	O
you	O
keep	O
the	O
logarithm	O
.	O
so	O
a	O
nice	O
property	O
of	O
probabilistic	O
modeling	O
is	O
that	O
by	O
following	O
some	O
assumptions	O
and	O
the	O
probability	O
rules	O
we	O
ll	O
get	O
a	O
formula	O
automatically	O
.	O
and	O
the	O
formula	O
would	O
have	O
a	O
particular	O
form	O
like	O
in	O
this	O
case	O
.	O
and	O
if	O
we	O
hueristically	O
design	O
the	O
formula	O
we	O
may	O
not	O
necessarily	O
end	O
up	O
having	O
such	O
a	O
specific	O
form	O
.	O
so	O
to	O
summarize	O
we	O
talked	O
about	O
the	O
need	O
for	O
smoothing	O
a	O
document	O
and	O
model	O
.	O
otherwise	O
it	O
would	O
give	O
zero	O
probability	O
for	O
unseen	O
words	O
in	O
the	O
document	O
.	O
and	O
that	O
s	O
not	O
good	O
for	O
scoring	O
a	O
query	O
with	O
such	O
an	O
unseen	O
word	O
.	O
it	O
s	O
also	O
necessary	O
in	O
general	O
to	O
improve	O
the	O
acc	O
accuracy	O
of	O
estimating	O
the	O
model	O
representing	O
the	O
topic	O
of	O
this	O
document	O
.	O
the	O
general	O
idea	O
of	O
smoothing	O
in	O
retrieval	O
is	O
to	O
use	O
the	O
connection	O
language	B
model	I
to	O
give	O
us	O
some	O
clue	O
about	O
which	O
unseen	O
word	O
would	O
have	O
a	O
higher	O
probability	O
.	O
that	O
is	O
the	O
probability	O
of	O
the	O
unseen	O
word	O
is	O
assumed	O
to	O
be	O
proportional	O
to	O
its	O
probability	O
in	O
the	O
collection	O
.	O
with	O
this	O
assumption	O
we	O
ve	O
shown	O
that	O
we	O
can	O
derive	O
a	O
general	O
ranking	O
formula	O
for	O
query	O
likelihood	O
.	O
that	O
has	O
a	O
fact	O
of	O
tf-idf	B
waiting	O
and	O
document	O
length	B
normalization	I
.	O
we	O
also	O
see	O
that	O
through	O
some	O
rewriting	O
the	O
scoring	O
of	O
such	O
ranking	B
function	I
is	O
primarily	O
based	O
on	O
sum	O
of	O
weights	O
on	O
matched	O
query	B
terms	I
just	O
like	O
in	O
the	O
vector	B
space	I
model	I
.	O
but	O
the	O
actual	O
ranking	B
function	I
is	O
given	O
us	O
automatically	O
by	O
the	O
probability	O
rules	O
and	O
assumptions	O
we	O
have	O
made	O
.	O
unlike	O
in	O
the	O
vector	B
space	I
model	I
where	O
we	O
have	O
to	O
heuristically	O
think	O
about	O
the	O
form	O
of	O
the	O
function	O
.	O
however	O
we	O
still	O
need	O
to	O
address	O
the	O
question	O
how	O
exactly	O
we	O
should	O
we	O
should	O
smooth	O
a	O
document	O
image	O
model	O
how	O
exactly	O
we	O
should	O
use	O
the	O
reference	O
language	B
model	I
based	O
on	O
the	O
connection	O
to	O
adjusting	O
the	O
probability	O
of	O
the	O
maximum	O
.	O
and	O
this	O
is	O
the	O
topic	O
of	O
the	O
next	O
to	O
that	O
.	O
so	O
let	O
s	O
plug	O
in	O
these	O
model	O
masses	O
into	O
the	O
ranking	B
function	I
to	O
see	O
what	O
we	O
will	O
get	O
okay	O
this	O
is	O
a	O
general	O
smoothing	O
.	O
so	O
a	O
general	O
ranking	B
function	I
for	O
smoothing	O
with	O
subtraction	O
and	O
you	O
have	O
seen	O
this	O
before	O
.	O
and	O
now	O
we	O
have	O
a	O
very	O
specific	O
smoothing	B
method	I
the	O
jm	B
smoothing	B
method	I
.	O
so	O
now	O
let	O
s	O
see	O
what	O
what	O
s	O
a	O
value	O
for	O
office	O
of	O
d	O
here	O
.	O
and	O
what	O
s	O
the	O
value	O
for	O
p	O
sub	O
c	O
here	O
right	O
so	O
we	O
may	O
need	O
to	O
decide	O
this	O
in	O
order	O
to	O
figure	O
out	O
the	O
exact	O
form	O
of	O
the	O
ranking	B
function	I
.	O
and	O
we	O
also	O
need	O
to	O
figure	O
out	O
of	O
course	O
alpha	O
.	O
so	O
let	O
s	O
see	O
.	O
well	O
this	O
ratio	O
is	O
basically	O
this	O
right	O
so	O
here	O
this	O
is	O
the	O
probability	O
of	O
c	O
board	O
on	O
the	O
top	O
and	O
this	O
is	O
the	O
probability	O
of	O
unseen	O
war	O
or	O
in	O
other	O
words	O
basically	O
times	O
basically	O
the	O
alpha	O
here	O
this	O
so	O
it	O
s	O
easy	O
to	O
see	O
that	O
.	O
this	O
can	O
be	O
then	O
rewritten	O
as	O
this	O
.	O
very	O
simple	O
.	O
so	O
we	O
can	O
plug	O
this	O
into	O
here	O
.	O
and	O
then	O
here	O
what	O
s	O
the	O
value	O
for	O
alpha	O
what	O
do	O
you	O
think	O
so	O
it	O
would	O
be	O
just	O
lambda	O
right	O
and	O
what	O
would	O
happen	O
if	O
we	O
plug	O
in	O
this	O
value	O
here	O
if	O
this	O
is	O
lambda	O
.	O
what	O
can	O
we	O
say	O
about	O
this	O
does	O
it	O
depend	O
on	O
the	O
document	O
no	O
so	O
it	O
can	O
be	O
ignored	O
.	O
right	O
so	O
we	O
ll	O
end	O
up	O
having	O
this	O
ranking	B
function	I
shown	O
here	O
.	O
and	O
in	O
this	O
case	O
you	O
can	O
easy	O
to	O
see	O
this	O
a	O
precisely	O
a	O
vector	B
space	I
model	I
because	O
this	O
part	O
is	O
a	O
sum	O
over	O
all	O
the	O
matched	O
query	B
terms	I
this	O
is	O
an	O
element	O
of	O
the	O
query	O
map	B
.	O
what	O
do	O
you	O
think	O
is	O
a	O
element	O
of	O
the	O
document	O
up	O
there	O
well	O
it	O
s	O
this	O
right	O
.	O
so	O
that	O
s	O
our	O
document	O
left	O
element	O
.	O
and	O
let	O
s	O
further	O
examine	O
what	O
s	O
inside	O
of	O
this	O
logarithm	O
.	O
well	O
one	O
plus	O
this	O
.	O
so	O
it	O
s	O
going	O
to	O
be	O
nonnegative	O
this	O
log	O
of	O
this	O
it	O
s	O
going	O
to	O
be	O
at	O
least	O
right	O
and	O
these	O
this	O
is	O
a	O
parameter	O
so	O
lambda	O
is	O
parameter	O
.	O
and	O
let	O
s	O
look	O
at	O
this	O
.	O
now	O
this	O
is	O
a	O
tf	B
.	O
now	O
we	O
see	O
very	O
clearly	O
this	O
tf	B
weighting	O
here	O
.	O
and	O
the	O
larger	O
the	O
count	O
is	O
the	O
higher	O
the	O
weighting	O
will	O
be	O
.	O
we	O
also	O
see	O
idf	B
weighting	O
which	O
is	O
given	O
by	O
this	O
.	O
and	O
we	O
see	O
docking	O
the	O
lan	O
s	O
relationship	O
here	O
.	O
so	O
all	O
these	O
heuristics	O
are	O
captured	O
in	O
this	O
formula	O
.	O
what	O
s	O
interesting	O
that	O
we	O
kind	O
of	O
have	O
got	O
this	O
weighting	O
function	O
automatically	O
by	O
making	O
various	O
assumptions	O
.	O
whereas	O
in	O
the	O
vector	B
space	I
model	I
we	O
had	O
to	O
go	O
through	O
those	O
heuristic	O
design	O
in	O
order	O
to	O
get	O
this	O
.	O
and	O
in	O
this	O
case	O
note	O
that	O
there	O
s	O
a	O
specific	O
form	O
.	O
and	O
when	O
you	O
see	O
whether	O
this	O
form	O
actually	O
makes	O
sense	O
.	O
all	O
right	O
so	O
what	O
do	O
you	O
think	O
is	O
the	O
denominator	O
here	O
hm	O
this	O
is	O
a	O
math	O
of	O
document	O
.	O
total	O
number	O
of	O
words	O
multiplied	O
by	O
the	O
probability	O
of	O
the	O
word	O
given	O
by	O
the	O
collection	O
right	O
so	O
this	O
actually	O
can	O
be	O
interpreted	O
as	O
expected	O
account	O
over	O
word	O
.	O
if	O
we	O
re	O
going	O
to	O
draw	O
a	O
word	O
from	O
the	O
connection	O
that	O
we	O
model	O
.	O
and	O
we	O
re	O
going	O
to	O
draw	O
as	O
many	O
as	O
the	O
number	O
of	O
words	O
in	O
the	O
document	O
.	O
if	O
you	O
do	O
that	O
the	O
expected	O
account	O
of	O
a	O
word	O
w	O
would	O
be	O
precisely	O
given	O
by	O
this	O
denominator	O
.	O
so	O
this	O
ratio	O
basically	O
is	O
comparing	O
the	O
actual	O
count	O
here	O
.	O
the	O
actual	O
count	O
of	O
the	O
word	O
in	O
the	O
document	O
with	O
expected	O
count	O
given	O
by	O
this	O
product	O
if	O
the	O
word	O
is	O
in	O
fact	O
following	O
the	O
distribution	O
in	O
the	O
clutch	O
this	O
.	O
and	O
if	O
this	O
counter	O
is	O
larger	O
than	O
the	O
expected	O
counter	O
in	O
this	O
part	O
this	O
ratio	O
would	O
be	O
larger	O
than	O
one	O
.	O
so	O
that	O
s	O
actually	O
a	O
very	O
interesting	O
interpretation	O
right	O
it	O
s	O
very	O
natural	O
and	O
intuitive	O
it	O
makes	O
a	O
lot	O
of	O
sense	O
.	O
and	O
this	O
is	O
one	O
advantage	O
of	O
using	O
this	O
kind	O
of	O
probabilistic	O
reasoning	O
where	O
we	O
have	O
made	O
explicit	O
assumptions	O
.	O
and	O
we	O
know	O
precisely	O
why	O
we	O
have	O
a	O
logarithm	O
here	O
.	O
and	O
why	O
we	O
have	O
these	O
probabilities	O
here	O
.	O
and	O
we	O
also	O
have	O
a	O
formula	O
that	O
intuitively	O
makes	O
a	O
lot	O
of	O
sense	O
and	O
does	O
tf-idf	B
weighting	O
and	O
documenting	O
and	O
some	O
others	O
.	O
let	O
s	O
look	O
at	O
the	O
the	O
dirichlet	B
prior	I
smoothing	I
.	O
it	O
s	O
very	O
similar	O
to	O
the	O
case	O
of	O
jm	B
smoothing	O
.	O
in	O
this	O
case	O
the	O
smoothing	O
parameter	O
is	O
mu	O
and	O
that	O
s	O
different	O
from	O
lambda	O
that	O
we	O
saw	O
before	O
.	O
but	O
the	O
format	O
looks	O
very	O
similar	O
.	O
the	O
form	O
of	O
the	O
function	O
looks	O
very	O
similar	O
.	O
so	O
we	O
still	O
have	O
linear	O
operation	O
here	O
.	O
and	O
when	O
we	O
compute	O
this	O
ratio	O
one	O
will	O
find	O
that	O
is	O
that	O
the	O
ratio	O
is	O
equal	O
to	O
this	O
.	O
and	O
what	O
s	O
interesting	O
here	O
is	O
that	O
we	O
are	O
doing	O
another	O
comparison	O
here	O
now	O
.	O
we	O
re	O
comparing	O
the	O
actual	O
count	O
.	O
which	O
is	O
the	O
expected	O
account	O
of	O
the	O
world	O
if	O
we	O
sampled	O
meal	O
worlds	O
according	O
to	O
the	O
collection	O
world	O
probability	O
.	O
so	O
note	O
that	O
it	O
s	O
interesting	O
we	O
don	O
t	O
even	O
see	O
docking	O
the	O
lens	O
here	O
and	O
lighter	O
in	O
the	O
jms	O
model	O
.	O
all	O
right	O
so	O
this	O
of	O
course	O
should	O
be	O
plugged	O
into	O
this	O
part	O
.	O
so	O
you	O
might	O
wonder	O
so	O
where	O
is	O
docking	O
lens	O
.	O
interestingly	O
the	O
docking	O
lens	O
is	O
here	O
in	O
alpha	O
sub	O
d	O
so	O
this	O
would	O
be	O
plugged	O
into	O
this	O
part	O
.	O
as	O
a	O
result	O
what	O
we	O
get	O
is	O
the	O
following	O
function	O
here	O
and	O
this	O
is	O
again	O
a	O
sum	O
over	O
all	O
the	O
match	O
query	B
words	I
.	O
and	O
we	O
re	O
against	O
the	O
queer	O
the	O
query	O
time	O
frequency	O
here	O
.	O
and	O
you	O
can	O
interpret	O
this	O
as	O
the	O
element	O
of	O
a	O
document	O
vector	O
but	O
this	O
is	O
no	O
longer	O
a	O
single	O
dot	B
product	I
right	O
because	O
we	O
have	O
this	O
part	O
i	O
know	O
that	O
n	O
is	O
the	O
name	O
of	O
the	O
query	O
right	O
so	O
that	O
just	O
means	O
if	O
we	O
score	O
this	O
function	O
we	O
have	O
to	O
take	O
a	O
sum	O
over	O
all	O
the	O
query	B
words	I
and	O
then	O
do	O
some	O
adjustment	O
of	O
the	O
score	O
based	O
on	O
the	O
document	O
.	O
but	O
it	O
s	O
still	O
it	O
s	O
still	O
clear	O
that	O
it	O
does	O
documents	O
lens	O
modulation	O
because	O
this	O
lens	O
is	O
in	O
the	O
denominator	O
so	O
a	O
longer	O
document	O
will	O
have	O
a	O
lower	O
weight	O
here	O
.	O
and	O
we	O
can	O
also	O
see	O
it	O
has	O
tf	B
here	O
and	O
now	O
idf	B
.	O
only	O
that	O
this	O
time	O
the	O
form	O
of	O
the	O
formula	O
is	O
different	O
from	O
the	O
previous	O
one	O
in	O
jms	O
one	O
.	O
but	O
intuitively	O
it	O
still	O
implements	O
tfidf	B
waiting	O
and	O
document	O
lens	O
rendition	O
again	O
the	O
form	O
of	O
the	O
function	O
is	O
dictated	O
by	O
the	O
probabilistic	O
reasoning	O
and	O
assumptions	O
that	O
we	O
have	O
made	O
.	O
now	O
there	O
are	O
also	O
disadvantages	O
of	O
this	O
approach	O
.	O
and	O
that	O
is	O
there	O
s	O
no	O
guarantee	O
that	O
there	O
s	O
such	O
a	O
form	O
of	O
the	O
formula	O
will	O
actually	O
work	O
well	O
.	O
so	O
if	O
we	O
look	O
about	O
at	O
this	O
geo	O
function	O
all	O
those	O
tf-idf	B
waiting	O
and	O
document	O
lens	O
rendition	O
for	O
example	O
it	O
s	O
unclear	O
whether	O
we	O
have	O
sub-linear	O
transformation	O
.	O
unfortunately	O
we	O
can	O
see	O
here	O
there	O
is	O
a	O
logarithm	O
function	O
here	O
.	O
so	O
we	O
do	O
have	O
also	O
the	O
so	O
it	O
s	O
here	O
right	O
so	O
we	O
do	O
have	O
the	O
sublinear	O
transformation	O
but	O
we	O
do	O
not	O
intentionally	O
do	O
that	O
.	O
that	O
means	O
there	O
s	O
no	O
guarantee	O
that	O
we	O
will	O
end	O
up	O
in	O
this	O
in	O
this	O
way	O
.	O
suppose	O
we	O
don	O
t	O
have	O
logarithm	O
then	O
there	O
s	O
no	O
sub-linear	O
transformation	O
.	O
as	O
we	O
discussed	O
before	O
perhaps	O
the	O
formula	O
is	O
not	O
going	O
to	O
work	O
so	O
well	O
.	O
so	O
that	O
s	O
an	O
example	O
of	O
the	O
gap	B
between	O
a	O
formal	O
model	O
like	O
this	O
and	O
the	O
relevance	O
that	O
we	O
have	O
to	O
model	O
which	O
is	O
really	O
a	O
subject	O
motion	O
that	O
is	O
tied	O
to	O
users	O
.	O
so	O
it	O
doesn	O
t	O
mean	O
we	O
cannot	O
fix	O
this	O
.	O
for	O
example	O
imagine	O
if	O
we	O
did	O
not	O
have	O
this	O
logarithm	O
right	O
so	O
we	O
can	O
take	O
a	O
risk	O
and	O
we	O
re	O
going	O
to	O
add	O
one	O
or	O
we	O
can	O
even	O
add	O
double	O
logarithm	O
.	O
but	O
then	O
it	O
would	O
mean	O
that	O
the	O
function	O
is	O
no	O
longer	O
a	O
proper	O
risk	O
model	O
.	O
so	O
the	O
consequence	O
of	O
the	O
modification	O
is	O
no	O
longer	O
as	O
predictable	O
as	O
what	O
we	O
have	O
been	O
doing	O
now	O
.	O
so	O
that	O
s	O
also	O
why	O
for	O
example	O
pm	O
remains	O
very	O
competitive	O
and	O
still	O
open	O
channel	O
how	O
to	O
use	O
public	O
risk	O
models	O
as	O
they	O
arrive	O
better	O
model	O
than	O
the	O
pm	O
	O
.	O
in	O
particular	O
how	O
do	O
we	O
use	O
query	O
like	O
how	O
to	O
derive	O
a	O
model	O
and	O
that	O
would	O
work	O
consistently	O
better	O
than	O
dm	O
	O
.	O
currently	O
we	O
still	O
cannot	O
do	O
that	O
.	O
still	O
interesting	O
open	O
question	O
.	O
so	O
to	O
summarize	O
this	O
part	O
we	O
ve	O
talked	O
about	O
the	O
two	O
smoothing	B
methods	I
.	O
jelinek-mercer	B
which	O
is	O
doing	O
the	O
fixed	O
coefficient	O
linear	B
interpolation	B
.	O
dirichlet	B
prior	I
this	O
is	O
what	O
add	O
a	O
pseudo	O
counts	O
to	O
every	O
word	O
and	O
is	O
doing	O
adaptive	O
interpolation	B
in	O
that	O
the	O
coefficient	O
would	O
be	O
larger	O
for	O
shorter	O
documents	O
.	O
in	O
most	O
cases	O
we	O
can	O
see	O
by	O
using	O
these	O
smoothing	B
methods	I
we	O
will	O
be	O
able	O
to	O
reach	O
a	O
retrieval	O
function	O
where	O
the	O
assumptions	O
are	O
clearly	O
articulate	O
.	O
so	O
they	O
are	O
less	O
heuristic	O
.	O
explaining	O
the	O
results	O
also	O
show	O
that	O
these	O
retrieval	O
functions	O
.	O
also	O
are	O
very	O
effective	O
and	O
they	O
are	O
comparable	O
to	O
bm	O
or	O
pm	O
lens	O
adultation	O
.	O
so	O
this	O
is	O
a	O
major	O
advantage	O
of	O
probably	O
smaller	O
where	O
we	O
don	O
t	O
have	O
to	O
do	O
a	O
lot	O
of	O
heuristic	O
design	O
.	O
yet	O
in	O
the	O
end	O
that	O
we	O
naturally	O
implemented	O
tf-idf	B
weighting	O
and	O
doc	O
length	B
normalization	I
.	O
each	O
of	O
these	O
functions	O
also	O
has	O
precise	O
ones	O
smoothing	O
parameter	O
.	O
in	O
this	O
case	O
of	O
course	O
we	O
still	O
need	O
to	O
set	O
this	O
smoothing	O
parameter	O
.	O
there	O
are	O
also	O
methods	O
that	O
can	O
be	O
used	O
to	O
estimate	O
these	O
parameters	O
.	O
so	O
overall	O
this	O
shows	O
by	O
using	O
a	O
probabilistic	B
model	I
we	O
follow	O
very	O
different	O
strategies	O
then	O
the	O
vector	B
space	I
model	I
.	O
yet	O
in	O
the	O
end	O
we	O
end	O
up	O
uh	O
with	O
some	O
retrievable	O
functions	O
that	O
look	O
very	O
similar	O
to	O
the	O
vector	B
space	I
model	I
.	O
with	O
some	O
advantages	O
in	O
having	O
assumptions	O
clearly	O
stated	O
.	O
and	O
then	O
the	O
form	O
dictated	O
by	O
a	O
probabilistic	B
model	I
.	O
now	O
this	O
also	O
concludes	O
our	O
discussion	O
of	O
the	O
query	O
likelihood	O
probabilistic	B
model	I
.	O
and	O
let	O
s	O
recall	B
what	O
assumptions	O
we	O
have	O
made	O
in	O
order	O
to	O
derive	O
the	O
functions	O
that	O
we	O
have	O
seen	O
in	O
this	O
lecture	O
.	O
well	O
we	O
basically	O
have	O
made	O
four	O
assumptions	O
that	O
i	O
listed	O
here	O
.	O
the	O
first	O
assumption	O
is	O
that	O
the	O
relevance	O
can	O
be	O
modeled	O
by	O
the	O
query	O
likelihood	O
.	O
and	O
the	O
second	O
assumption	O
with	O
med	O
is	O
are	O
query	B
words	I
are	O
generated	O
independently	O
that	O
allows	O
us	O
to	O
decompose	O
the	O
probability	O
of	O
the	O
whole	O
query	O
into	O
a	O
product	O
of	O
probabilities	O
of	O
old	O
words	O
in	O
the	O
query	O
.	O
and	O
then	O
the	O
third	O
assumption	O
that	O
we	O
have	O
made	O
is	O
if	O
a	O
word	O
is	O
not	O
seen	O
the	O
document	O
or	O
in	O
the	O
late	O
its	O
probability	O
proportional	O
to	O
its	O
probability	O
in	O
the	O
collection	O
.	O
that	O
s	O
a	O
smoothing	O
with	O
a	O
collection	O
ama	O
model	O
.	O
and	O
finally	O
we	O
made	O
one	O
of	O
these	O
two	O
assumptions	O
about	O
the	O
smoothing	O
.	O
so	O
we	O
either	O
used	O
jm	B
smoothing	O
or	O
dirichlet	B
prior	I
smoothing	I
.	O
if	O
we	O
make	O
these	O
four	O
assumptions	O
then	O
we	O
have	O
no	O
choice	O
but	O
to	O
take	O
the	O
form	O
of	O
the	O
retrieval	O
function	O
that	O
we	O
have	O
seen	O
earlier	O
.	O
fortunately	O
the	O
function	O
has	O
a	O
nice	O
property	O
in	O
that	O
it	O
implements	O
tf-idf	B
weighting	O
and	O
document	O
machine	O
and	O
these	O
functions	O
also	O
work	O
very	O
well	O
.	O
so	O
in	O
that	O
sense	O
these	O
functions	O
are	O
less	O
heuristic	O
compared	O
with	O
the	O
vector	B
space	I
model	I
.	O
and	O
there	O
are	O
many	O
extensions	O
of	O
this	O
this	O
basic	O
model	O
and	O
you	O
can	O
find	O
the	O
discussion	O
of	O
them	O
in	O
the	O
reference	O
at	O
the	O
end	O
of	O
this	O
lecture	O
.	O
this	O
lecture	O
is	O
about	O
the	O
feedback	O
in	O
text	B
retrieval	I
.	O
so	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
continue	O
the	O
discussion	O
on	O
text	B
retrieval	B
methods	I
.	O
in	O
particular	O
we	O
re	O
going	O
to	O
talk	O
about	O
feedback	O
in	O
text	B
retrieval	I
.	O
this	O
is	O
a	O
diagram	O
that	O
shows	O
the	O
retrieval	O
process	O
.	O
we	O
can	O
see	O
the	O
user	O
would	O
typed	O
in	O
a	O
query	O
and	O
then	O
the	O
query	O
would	O
be	O
sent	O
to	O
a	O
retrieval	O
engine	O
or	O
search	B
engine	I
and	O
the	O
engine	O
would	O
return	O
results	O
.	O
these	O
results	O
would	O
be	O
shown	O
to	O
the	O
user	O
.	O
after	O
the	O
user	O
has	O
seen	O
these	O
results	O
the	O
user	O
can	O
actually	O
make	O
judgments	O
.	O
so	O
for	O
example	O
the	O
user	O
has	O
say	O
well	O
this	O
is	O
good	O
and	O
this	O
document	O
is	O
not	O
very	O
useful	O
.	O
this	O
is	O
good	O
again	O
et	O
cetera	O
.	O
now	O
this	O
is	O
called	O
a	O
relevance	B
judgment	I
or	O
relevance	B
feedback	I
because	O
we	O
ve	O
got	O
some	O
feedback	O
information	O
from	O
the	O
user	O
based	O
on	O
the	O
judgments	O
.	O
this	O
can	O
be	O
very	O
useful	O
to	O
the	O
system	O
.	O
learn	O
what	O
exactly	O
is	O
interesting	O
to	O
the	O
user	O
.	O
so	O
the	O
feedback	O
module	O
would	O
then	O
take	O
this	O
as	O
input	O
and	O
also	O
use	O
the	O
document	O
collection	O
to	O
try	O
to	O
improve	O
ranking	O
.	O
typically	O
it	O
would	O
involve	O
updating	O
the	O
query	O
.	O
so	O
the	O
system	O
can	O
now	O
rank	O
the	O
results	O
more	O
accurately	O
for	O
the	O
user	O
.	O
so	O
this	O
is	O
called	O
relevance	B
feedback	I
.	O
the	O
feedback	O
is	O
based	O
on	O
relevance	O
judgements	O
made	O
by	O
the	O
users	O
.	O
now	O
these	O
judgements	O
are	O
reliable	O
but	O
the	O
users	O
generally	O
don	O
t	O
want	O
to	O
make	O
extra	O
effort	O
unless	O
they	O
have	O
to	O
.	O
so	O
the	O
downside	O
s	O
that	O
involves	O
some	O
extra	O
effort	O
by	O
the	O
user	O
.	O
there	O
is	O
another	O
form	O
of	O
feedback	O
called	O
a	O
pseudo	B
relevance	B
feedback	I
or	O
a	O
blind	B
feedback	I
also	O
called	O
an	O
automatic	B
feedback	I
.	O
in	O
this	O
case	O
you	O
can	O
see	O
once	O
the	O
user	O
has	O
got	O
without	O
an	O
effect	O
we	O
don	O
t	O
have	O
to	O
involve	O
users	O
.	O
so	O
you	O
can	O
see	O
there	O
s	O
no	O
user	O
involved	O
here	O
.	O
and	O
we	O
simply	O
assume	O
that	O
the	O
top	O
ranked	O
documents	O
to	O
be	O
relevant	O
.	O
let	O
s	O
say	O
we	O
have	O
assumed	O
the	O
top	O
ten	O
is	O
relevant	O
.	O
and	O
then	O
we	O
will	O
then	O
use	O
these	O
assumed	O
documents	O
to	O
learn	O
and	O
to	O
improve	O
the	O
query	O
.	O
now	O
you	O
might	O
wonder	O
you	O
know	O
how	O
could	O
this	O
help	O
if	O
we	O
simply	O
assume	O
the	O
top	O
rank	O
documents	O
would	O
be	O
random	O
.	O
well	O
you	O
can	O
imagine	O
these	O
top	O
rank	O
documents	O
are	O
actually	O
similar	O
to	O
relevant	B
documents	I
even	O
if	O
they	O
are	O
not	O
relevant	O
they	O
look	O
like	O
relevant	B
documents	I
.	O
so	O
it	O
s	O
possible	O
to	O
learn	O
some	O
related	O
terms	O
to	O
the	O
query	O
from	O
this	O
set	O
.	O
in	O
fact	O
you	O
may	O
recall	B
that	O
we	O
talked	O
about	O
using	O
language	B
model	I
to	O
analyze	O
word	O
association	O
to	O
learn	O
related	O
words	O
to	O
the	O
word	O
computer	O
.	O
right	O
and	O
then	O
what	O
we	O
did	O
is	O
first	O
use	O
computer	O
to	O
retrieve	O
all	O
the	O
documents	O
that	O
contain	O
computer	O
.	O
so	O
imagine	O
now	O
the	O
query	O
here	O
is	O
a	O
computer	O
.	O
right	O
and	O
then	O
the	O
results	O
will	O
be	O
those	O
documents	O
that	O
contain	O
computer	O
.	O
and	O
what	O
we	O
can	O
do	O
then	O
is	O
to	O
take	O
the	O
top	O
end	O
results	O
.	O
they	O
can	O
match	O
computer	O
very	O
well	O
and	O
we	O
re	O
going	O
to	O
count	O
the	O
terms	O
in	O
this	O
set	O
and	O
then	O
we	O
re	O
going	O
to	O
then	O
use	O
the	O
background	O
language	B
model	I
to	O
choose	O
the	O
terms	O
that	O
are	O
frequent	O
the	O
in	O
this	O
set	O
but	O
not	O
frequent	O
the	O
in	O
the	O
whole	O
collection	O
.	O
so	O
if	O
we	O
make	O
a	O
contrast	O
between	O
these	O
two	O
what	O
we	O
can	O
find	O
is	O
that	O
we	O
ll	O
learn	O
some	O
related	O
terms	O
too	O
the	O
work	O
computer	O
as	O
what	O
i	O
ve	O
seen	O
before	O
.	O
and	O
these	O
related	O
words	O
can	O
then	O
be	O
added	O
to	O
the	O
original	O
query	O
to	O
expand	O
the	O
query	O
.	O
and	O
this	O
would	O
help	O
us	O
free	O
documents	O
that	O
don	O
t	O
necessarily	O
match	O
computer	O
but	O
match	O
other	O
words	O
like	O
program	O
and	O
software	O
.	O
so	O
this	O
is	O
factored	O
for	O
improving	O
the	O
search	O
doubt	O
.	O
but	O
of	O
course	O
pseudo	O
relevancy	O
feedback	O
is	O
completely	O
unreliable	O
.	O
we	O
have	O
to	O
arbitrarily	O
set	O
a	O
cutoff	O
.	O
so	O
there	O
is	O
also	O
something	O
in	O
between	O
called	O
implicit	B
feedback	I
.	O
in	O
this	O
case	O
what	O
we	O
do	O
we	O
do	O
involve	O
users	O
but	O
we	O
don	O
t	O
have	O
to	O
ask	O
users	O
to	O
make	O
judgements	O
.	O
instead	O
we	O
are	O
going	O
to	O
observe	O
how	O
the	O
user	O
interacts	O
with	O
the	O
search	O
results	O
.	O
so	O
in	O
this	O
case	O
we	O
re	O
going	O
to	O
look	O
at	O
the	O
clickthroughs	O
.	O
so	O
the	O
user	O
clicked	O
on	O
this	O
one	O
and	O
the	O
the	O
user	O
viewed	O
this	O
one	O
.	O
and	O
the	O
user	O
skipped	O
this	O
one	O
and	O
the	O
user	O
viewed	O
this	O
one	O
again	O
.	O
now	O
this	O
also	O
is	O
a	O
clue	O
about	O
whether	O
a	O
document	O
is	O
useful	O
to	O
the	O
user	O
and	O
we	O
can	O
even	O
assume	O
that	O
we	O
re	O
going	O
to	O
use	O
only	O
the	O
snippet	O
here	O
in	O
this	O
document	O
.	O
the	O
text	O
that	O
s	O
actually	O
seen	O
by	O
the	O
user	O
instead	O
of	O
the	O
actual	O
document	O
of	O
this	O
entry	O
in	O
the	O
link	O
.	O
there	O
that	O
same	O
web	B
search	I
may	O
be	O
broken	O
but	O
that	O
it	O
doesn	O
t	O
matter	O
.	O
if	O
the	O
user	O
tries	O
to	O
fetch	O
this	O
document	O
that	O
because	O
of	O
the	O
displayed	O
text	O
we	O
can	O
assume	O
this	O
displayed	O
text	O
is	O
probably	O
relevant	O
is	O
interesting	O
to	O
user	O
so	O
we	O
can	O
learn	O
from	O
such	O
information	O
.	O
and	O
this	O
is	O
called	O
implicit	B
feedback	I
and	O
we	O
can	O
again	O
use	O
the	O
information	O
to	O
update	O
the	O
query	O
.	O
this	O
is	O
a	O
very	O
important	O
technique	O
used	O
in	O
modern	O
search	B
engines	I
.	O
you	O
know	O
think	O
about	O
google	O
and	O
bing	O
and	O
they	O
can	O
collect	O
a	O
lot	O
of	O
user	O
activities	O
.	O
why	O
they	O
are	O
serverless	O
right	O
.	O
so	O
they	O
would	O
observe	O
what	O
documents	O
we	O
click	O
on	O
what	O
documents	O
we	O
skip	O
.	O
and	O
this	O
information	O
is	O
very	O
valuable	O
and	O
they	O
can	O
use	O
this	O
to	O
encode	O
the	O
search	B
engine	I
.	O
so	O
to	O
summarize	O
we	O
would	O
talk	O
about	O
the	O
three	O
kinds	O
of	O
feedback	O
here	O
rather	O
than	O
feedback	O
.	O
where	O
the	O
use	O
exquisite	O
judgement	O
it	O
takes	O
some	O
used	O
effort	O
but	O
the	O
judgement	O
that	O
information	O
is	O
reliable	O
.	O
we	O
talked	O
about	O
the	O
pseudo	B
feedback	I
where	O
we	O
simply	O
assumed	O
top	O
random	O
documents	O
.	O
we	O
get	O
random	O
we	O
don	O
t	O
have	O
to	O
involve	O
the	O
user	O
.	O
therefore	O
we	O
could	O
do	O
that	O
actually	O
before	O
we	O
we	O
return	O
the	O
results	O
to	O
the	O
user	O
.	O
and	O
the	O
third	O
is	O
implicit	B
feedback	I
where	O
we	O
use	O
clickthroughs	O
.	O
where	O
we	O
don	O
t	O
we	O
involve	O
users	O
but	O
the	O
user	O
doesn	O
t	O
have	O
to	O
make	O
explicit	O
effort	O
to	O
make	O
judgement	O
.	O
this	O
lecture	O
is	O
about	O
the	O
feedback	O
in	O
the	O
vector	B
space	I
model	I
.	O
in	O
this	O
lecture	O
we	O
continue	O
talking	O
about	O
the	O
feedback	O
and	O
text	B
retrieval	I
.	O
particularly	O
we	O
re	O
going	O
to	O
talk	O
about	O
feedback	O
in	O
the	O
vector	B
space	I
model	I
.	O
as	O
we	O
have	O
discussed	O
before	O
in	O
the	O
case	O
of	O
feedback	O
the	O
task	O
of	O
a	O
text	B
retrieval	I
system	O
is	O
relearned	O
from	O
examples	O
to	O
improve	O
retrieval	O
accuracy	O
.	O
we	O
will	O
have	O
positive	O
examples	O
those	O
are	O
the	O
documents	O
that	O
are	O
assumed	O
that	O
will	O
be	O
random	O
or	O
judged	O
with	O
being	O
random	O
and	O
all	O
the	O
documents	O
that	O
are	O
viewed	O
by	O
users	O
.	O
we	O
also	O
have	O
negative	O
examples	O
those	O
are	O
documents	O
known	O
to	O
be	O
non-relevant	O
.	O
they	O
can	O
also	O
be	O
the	O
documents	O
that	O
are	O
escaped	O
by	O
users	O
.	O
the	O
general	O
method	O
in	O
the	O
vector	B
space	I
model	I
for	O
feedback	O
is	O
to	O
modify	O
our	O
query	O
vector	O
.	O
now	O
we	O
want	O
to	O
place	O
the	O
query	O
vector	O
in	O
a	O
better	O
position	O
to	O
make	O
that	O
accurate	O
and	O
what	O
does	O
that	O
mean	O
exactly	O
well	O
if	O
you	O
think	O
about	O
the	O
query	O
vector	O
that	O
would	O
mean	O
you	O
would	O
have	O
to	O
do	O
something	O
to	O
vector	O
elements	O
.	O
and	O
in	O
general	O
that	O
would	O
mean	O
we	O
might	O
add	O
new	O
terms	O
.	O
we	O
might	O
adjust	O
weights	O
of	O
old	O
terms	O
or	O
assign	O
weights	O
to	O
new	O
terms	O
.	O
and	O
as	O
a	O
result	O
in	O
general	O
the	O
query	O
will	O
have	O
more	O
terms	O
so	O
we	O
often	O
call	O
this	O
query	O
expansion	O
.	O
the	O
most	O
effective	O
method	O
in	O
the	O
vector	B
space	I
model	I
of	O
feedback	O
is	O
called	O
rocchio	B
feedback	O
which	O
was	O
actually	O
proposed	O
several	O
decades	O
ago	O
.	O
so	O
the	O
idea	O
is	O
quite	O
simple	O
we	O
illustrate	O
this	O
idea	O
by	O
using	O
a	O
two-dimensional	O
display	O
of	O
all	O
the	O
documents	O
in	O
the	O
collection	O
and	O
also	O
the	O
query	O
vector	O
.	O
so	O
now	O
we	O
can	O
see	O
the	O
query	O
vector	O
is	O
here	O
in	O
the	O
center	O
and	O
these	O
are	O
all	O
of	O
the	O
documents	O
.	O
so	O
when	O
we	O
use	O
a	O
query	O
vector	O
and	O
use	O
a	O
similarity	O
function	O
to	O
find	O
the	O
most	O
similar	O
documents	O
.	O
we	O
are	O
basically	O
drawing	O
a	O
circle	O
here	O
and	O
then	O
these	O
documents	O
would	O
be	O
basically	O
the	O
top-ranked	O
documents	O
.	O
and	O
this	O
process	O
of	O
relevant	B
documents	I
right	O
and	O
these	O
are	O
random	O
documents	O
for	O
example	O
that	O
s	O
relevant	O
etc	O
.	O
and	O
then	O
these	O
minuses	O
are	O
negative	O
documents	O
like	O
this	O
.	O
so	O
our	O
goal	O
here	O
is	O
trying	O
to	O
move	O
this	O
query	O
vector	O
to	O
some	O
position	O
to	O
improve	O
the	O
retrieval	O
accuracy	O
.	O
by	O
looking	O
at	O
this	O
diagram	O
what	O
do	O
you	O
think	O
where	O
should	O
we	O
move	O
the	O
query	O
vector	O
so	O
that	O
we	O
can	O
improve	O
the	O
retrieval	O
accuracy	O
.	O
intuitively	O
where	O
do	O
you	O
want	O
to	O
move	O
the	O
query	O
back	O
to	O
if	O
you	O
want	O
to	O
think	O
more	O
you	O
can	O
pause	O
the	O
video	O
.	O
now	O
if	O
you	O
think	O
about	O
this	O
picture	O
you	O
can	O
realize	O
that	O
in	O
order	O
to	O
work	O
well	O
in	O
this	O
case	O
you	O
want	O
the	O
query	O
vector	O
to	O
be	O
as	O
close	O
to	O
the	O
positive	O
vectors	O
as	O
possible	O
.	O
that	O
means	O
ideally	O
you	O
want	O
to	O
place	O
the	O
query	O
vector	O
somewhere	O
here	O
or	O
we	O
want	O
to	O
move	O
the	O
query	O
vector	O
closer	O
to	O
this	O
point	O
.	O
now	O
so	O
what	O
exactly	O
at	O
this	O
point	O
well	O
if	O
you	O
want	O
these	O
relevant	B
documents	I
to	O
be	O
ranked	O
on	O
the	O
top	O
you	O
want	O
this	O
to	O
be	O
in	O
the	O
center	O
of	O
all	O
of	O
these	O
relevant	B
documents	I
right	O
because	O
then	O
if	O
you	O
draw	O
a	O
circle	O
around	O
this	O
one	O
you	O
get	O
all	O
these	O
relevant	B
documents	I
.	O
so	O
that	O
means	O
we	O
can	O
move	O
the	O
query	O
back	O
toward	O
the	O
centroid	O
of	O
all	O
the	O
relevant	O
document	O
vectors	O
.	O
and	O
this	O
is	O
basically	O
the	O
idea	O
of	O
rocchio	B
of	O
course	O
you	O
then	O
can	O
see	O
that	O
the	O
centroid	O
of	O
negative	O
documents	O
.	O
and	O
one	O
move	O
away	O
from	O
the	O
negative	O
documents	O
.	O
now	O
geometrically	O
we	O
re	O
talking	O
about	O
a	O
moving	O
vector	O
closer	O
to	O
some	O
other	O
vector	O
and	O
away	O
from	O
other	O
vectors	O
.	O
algebraically	O
it	O
just	O
means	O
we	O
have	O
this	O
formula	O
.	O
here	O
you	O
can	O
see	O
this	O
is	O
original	O
query	O
vector	O
and	O
this	O
average	O
basically	O
is	O
the	O
centroid	O
vector	O
of	O
relevant	B
documents	I
.	O
when	O
we	O
take	O
the	O
average	O
over	O
these	O
vectors	O
then	O
we	O
re	O
computing	O
the	O
centroid	O
of	O
these	O
vectors	O
.	O
and	O
similarly	O
this	O
is	O
the	O
average	O
in	O
that	O
non-relevant	O
document	O
of	O
vectors	O
so	O
it	O
s	O
essentially	O
of	O
now	O
random	O
documents	O
.	O
and	O
we	O
have	O
these	O
three	O
parameters	O
here	O
alpha	O
beta	B
and	O
gamma	B
.	O
they	O
re	O
controlling	O
the	O
amount	O
of	O
movement	O
.	O
when	O
we	O
add	O
these	O
two	O
vectors	O
together	O
we	O
re	O
moving	O
the	O
query	O
at	O
the	O
closer	O
to	O
the	O
centroid	O
alright	O
so	O
when	O
we	O
add	O
them	O
together	O
.	O
when	O
we	O
subtracted	O
this	O
part	O
we	O
kind	O
of	O
move	O
the	O
query	O
vector	O
away	O
from	O
that	O
centroid	O
so	O
this	O
is	O
the	O
main	O
idea	O
of	O
rocchio	B
feedback	O
.	O
and	O
after	O
we	O
have	O
done	O
this	O
we	O
will	O
get	O
a	O
new	O
query	O
vector	O
which	O
can	O
use	O
it	O
to	O
store	O
documents	O
.	O
this	O
new	O
new	O
query	O
vector	O
will	O
then	O
reflect	O
the	O
move	O
of	O
this	O
original	O
query	O
vector	O
toward	O
this	O
relevant	O
centroid	O
vector	O
and	O
away	O
from	O
the	O
non-relevant	O
centroid	O
vector	O
okay	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
example	O
right	O
this	O
is	O
the	O
example	O
that	O
we	O
have	O
seen	O
earlier	O
only	O
that	O
i	O
in	O
the	O
the	O
display	O
of	O
the	O
actual	O
documents	O
i	O
only	O
showed	O
the	O
vector	O
representation	O
of	O
these	O
documents	O
.	O
we	O
have	O
five	O
documents	O
here	O
and	O
we	O
have	O
true	O
red	O
in	O
the	O
documents	O
here	O
right	O
they	O
are	O
displayed	O
in	O
red	O
and	O
these	O
are	O
the	O
term	O
vectors	O
.	O
now	O
i	O
just	O
assumed	O
an	O
idea	O
of	O
weights	O
a	O
lot	O
of	O
times	O
we	O
have	O
zero	O
weights	O
of	O
course	O
.	O
these	O
are	O
negative	O
documents	O
there	O
are	O
two	O
here	O
there	O
is	O
another	O
one	O
here	O
.	O
now	O
in	O
this	O
rocchio	B
method	O
we	O
first	O
compute	O
the	O
centroid	O
of	O
each	O
category	O
and	O
so	O
let	O
s	O
see	O
.	O
look	O
at	O
the	O
centroid	O
of	O
the	O
positive	O
document	O
but	O
we	O
simply	O
just	O
so	O
it	O
s	O
very	O
easy	O
to	O
see	O
.	O
we	O
just	O
add	O
this	O
with	O
this	O
one	O
the	O
corresponding	O
element	O
and	O
that	O
s	O
down	O
here	O
and	O
take	O
the	O
average	O
.	O
and	O
then	O
we	O
re	O
going	O
to	O
add	O
the	O
corresponding	O
elements	O
and	O
then	O
just	O
take	O
the	O
average	O
right	O
so	O
we	O
do	O
this	O
for	O
all	O
these	O
.	O
in	O
the	O
end	O
what	O
we	O
have	O
is	O
this	O
one	O
.	O
this	O
is	O
the	O
average	O
vector	O
of	O
these	O
two	O
so	O
it	O
s	O
a	O
centroid	O
of	O
these	O
two	O
right	O
let	O
s	O
also	O
look	O
at	O
the	O
centroid	O
of	O
the	O
nested	O
documents	O
.	O
this	O
is	O
basically	O
the	O
same	O
we	O
re	O
going	O
to	O
take	O
the	O
average	O
of	O
three	O
elements	O
.	O
and	O
these	O
are	O
the	O
corresponding	O
elements	O
in	O
these	O
three	O
vectors	O
and	O
so	O
on	O
and	O
so	O
forth	O
.	O
so	O
in	O
the	O
end	O
that	O
we	O
have	O
this	O
one	O
.	O
now	O
in	O
the	O
rocchio	B
feedback	O
method	O
we	O
re	O
going	O
to	O
combine	O
all	O
these	O
with	O
original	O
query	O
vector	O
which	O
is	O
this	O
.	O
so	O
now	O
let	O
s	O
see	O
how	O
we	O
combine	O
them	O
together	O
.	O
well	O
that	O
s	O
basically	O
this	O
right	O
so	O
we	O
have	O
a	O
parameter	O
outlier	O
controlling	O
the	O
original	O
query	B
term	I
weight	O
that	O
s	O
	O
.	O
and	O
now	O
i	O
ve	O
beta	B
to	O
control	O
the	O
inference	B
of	O
the	O
positive	O
centroid	O
vector	O
weight	O
that	O
s	O
	O
.	O
that	O
comes	O
from	O
here	O
right	O
so	O
this	O
goes	O
here	O
and	O
we	O
also	O
have	O
this	O
negative	O
wait	O
here	O
.	O
conduit	O
by	O
a	O
gamma	B
here	O
and	O
this	O
weight	O
has	O
come	O
from	O
of	O
course	O
the	O
nective	O
centroid	O
here	O
.	O
and	O
we	O
do	O
exactly	O
the	O
same	O
for	O
other	O
terms	O
each	O
is	O
for	O
one	O
term	O
.	O
and	O
this	O
is	O
our	O
new	O
vector	O
.	O
and	O
we	O
re	O
going	O
to	O
use	O
this	O
new	O
query	O
vector	O
this	O
one	O
to	O
run	O
the	O
documents	O
.	O
you	O
can	O
imagine	O
what	O
would	O
happen	O
right	O
because	O
of	O
the	O
movement	O
that	O
this	O
one	O
or	O
the	O
match	O
of	O
these	O
red	O
documents	O
much	O
better	O
.	O
because	O
we	O
move	O
this	O
vector	O
closer	O
to	O
them	O
and	O
it	O
s	O
going	O
to	O
penalize	O
these	O
black	O
documents	O
these	O
non-relevant	O
documents	O
.	O
so	O
this	O
is	O
precisely	O
what	O
we	O
want	O
from	O
feedback	O
.	O
now	O
of	O
course	O
if	O
we	O
apply	O
this	O
method	O
in	O
practice	O
we	O
will	O
see	O
one	O
potential	O
problem	O
and	O
that	O
is	O
the	O
original	O
query	O
has	O
only	O
four	O
times	O
that	O
are	O
not	O
zero	O
.	O
but	O
after	O
we	O
do	O
queries	O
imagine	O
you	O
can	O
imagine	O
we	O
ll	O
have	O
many	O
terms	O
that	O
would	O
have	O
a	O
number	O
of	O
weights	O
.	O
so	O
the	O
calculation	O
would	O
have	O
to	O
involve	O
more	O
terms	O
.	O
in	O
practice	O
we	O
often	O
truncate	O
this	O
vector	O
and	O
only	O
retain	O
the	O
terms	O
which	O
is	O
the	O
highest	O
weight	O
.	O
so	O
let	O
s	O
talk	O
about	O
how	O
we	O
use	O
this	O
method	O
in	O
practice	O
.	O
i	O
just	O
mentioned	O
that	O
we	O
often	O
truncate	O
the	O
vector	O
consider	O
only	O
a	O
small	O
number	O
of	O
words	O
that	O
have	O
highest	O
weights	O
in	O
the	O
centroid	O
vector	O
.	O
this	O
is	O
for	O
efficiency	O
concern	O
.	O
i	O
also	O
say	O
that	O
here	O
that	O
a	O
negative	O
examples	O
or	O
non-relevant	O
examples	O
tend	O
not	O
to	O
be	O
very	O
useful	O
especially	O
compared	O
with	O
positive	O
examples	O
.	O
now	O
you	O
can	O
think	O
about	O
the	O
why	O
.	O
one	O
reason	O
is	O
because	O
negative	O
documents	O
tend	O
to	O
distract	O
the	O
query	O
in	O
all	O
directions	O
so	O
when	O
you	O
take	O
the	O
average	O
it	O
doesn	O
t	O
really	O
tell	O
you	O
where	O
exactly	O
it	O
should	O
be	O
moving	O
to	O
.	O
whereas	O
positive	O
documents	O
tend	O
to	O
be	O
clustered	O
together	O
and	O
they	O
respond	O
to	O
you	O
to	O
consistent	O
the	O
direction	O
.	O
so	O
that	O
also	O
means	O
that	O
sometimesw	O
we	O
don	O
t	O
have	O
those	O
negative	O
examples	O
but	O
note	O
that	O
in	O
in	O
some	O
cases	O
in	O
difficult	O
queries	O
where	O
most	O
top	O
random	O
results	O
are	O
negative	O
.	O
negative	O
feedback	O
afterwards	O
is	O
very	O
useful	O
.	O
another	O
thing	O
is	O
to	O
avoid	O
over-fitting	O
that	O
means	O
we	O
have	O
to	O
keep	O
relatively	O
high	O
weight	O
on	O
the	O
original	O
query	B
terms	I
.	O
why	O
because	O
the	O
sample	O
that	O
we	O
see	O
in	O
feedback	O
is	O
a	O
relatively	O
small	O
sample	O
.	O
we	O
don	O
t	O
want	O
to	O
overly	O
trust	O
the	O
small	O
sample	O
and	O
the	O
original	O
query	B
terms	I
are	O
still	O
very	O
important	O
.	O
those	O
terms	O
are	O
typed	O
in	O
by	O
the	O
user	O
and	O
the	O
user	O
has	O
decided	O
that	O
those	O
terms	O
are	O
most	O
important	O
.	O
so	O
in	O
order	O
to	O
prevent	O
the	O
us	O
from	O
over-fitting	O
or	O
drifting	O
.	O
a	O
type	O
of	O
drift	O
prevent	O
type	O
of	O
drifting	O
due	O
to	O
the	O
bias	O
toward	O
the	O
the	O
feedback	O
examples	O
.	O
we	O
generally	O
would	O
have	O
to	O
keep	O
a	O
pretty	O
high	O
weight	O
on	O
the	O
original	O
terms	O
so	O
it	O
is	O
safe	O
to	O
do	O
that	O
.	O
and	O
this	O
is	O
especially	O
true	O
for	O
pseudo	O
awareness	O
feedback	O
.	O
now	O
this	O
method	O
can	O
be	O
used	O
for	O
both	O
relevance	B
feedback	I
and	O
pseudo	B
relevance	B
feedback	I
.	O
in	O
the	O
case	O
of	O
pseudo	B
feedback	I
the	O
parameter	O
beta	B
should	O
be	O
set	O
to	O
a	O
a	O
smaller	O
value	O
because	O
the	O
random	O
examples	O
are	O
assumed	O
to	O
be	O
random	O
there	O
not	O
as	O
reliable	O
as	O
your	O
relevance	B
feedback	I
right	O
in	O
the	O
case	O
of	O
relevance	B
feedback	I
we	O
obviously	O
could	O
use	O
a	O
larger	O
value	O
.	O
so	O
those	O
parameters	O
still	O
have	O
to	O
be	O
set	O
and	O
.	O
and	O
the	O
ro	O
rocchio	B
method	O
is	O
usually	O
robust	O
and	O
effective	O
.	O
it	O
s	O
it	O
s	O
still	O
a	O
very	O
popular	O
method	O
for	O
feedback	O
.	O
this	O
lecture	O
is	O
about	O
the	O
feedback	O
in	O
the	O
language	B
modeling	I
approach	I
.	O
in	O
this	O
lecture	O
we	O
will	O
continue	O
the	O
discussion	O
of	O
feedback	O
in	O
text	B
retrieval	I
.	O
in	O
particular	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
feedback	O
in	O
language	B
modeling	I
approaches	O
.	O
so	O
we	O
derive	O
the	O
query	O
likelihood	O
ranking	B
function	I
by	O
making	O
various	O
assumptions	O
.	O
as	O
a	O
basic	O
retrieval	O
function	O
that	O
formula	O
or	O
those	O
formulas	O
worked	O
well	O
.	O
but	O
if	O
we	O
think	O
about	O
the	O
feedback	O
information	O
it	O
s	O
a	O
little	O
bit	O
awkward	O
to	O
use	O
query	O
likelihood	O
to	O
perform	O
feedback	O
because	O
a	O
lot	O
of	O
times	O
the	O
feedback	O
information	O
is	O
additional	O
information	O
about	O
the	O
query	O
.	O
but	O
we	O
assume	O
the	O
query	O
is	O
generated	O
by	O
assembling	O
words	O
from	O
a	O
language	B
model	I
in	O
the	O
query	O
likelihood	O
method	O
.	O
it	O
s	O
kind	O
of	O
unnatural	O
to	O
sample	O
words	O
that	O
form	O
feedback	O
documents	O
.	O
as	O
a	O
result	O
then	O
research	O
is	O
proposed	O
a	O
way	O
to	O
generalize	O
query	O
likelihood	O
function	O
.	O
it	O
s	O
called	O
a	O
kullback-leibler	B
divergence	B
retrieval	B
model	I
.	O
and	O
this	O
model	O
is	O
actually	O
going	O
to	O
make	O
the	O
query	O
likelihood	O
our	O
retrieval	O
function	O
much	O
closer	O
to	O
vector	B
space	I
model	I
.	O
yet	O
this	O
form	O
of	O
the	O
language	B
model	I
can	O
be	O
regarded	O
as	O
a	O
generalization	O
of	O
query	O
likelihood	O
in	O
the	O
sense	O
that	O
if	O
it	O
can	O
cover	O
query	O
likelihood	O
as	O
a	O
special	O
case	O
.	O
and	O
in	O
this	O
case	O
the	O
feedback	O
can	O
be	O
achieved	O
through	O
simply	O
query	B
model	I
estimation	O
or	O
updating	O
.	O
this	O
is	O
very	O
similar	O
to	O
rocchio	B
which	O
updates	O
the	O
query	O
vector	O
.	O
so	O
let	O
s	O
see	O
what	O
the	O
is	O
the	O
scale	O
of	O
divergence	B
which	O
we	O
will	O
model	O
.	O
so	O
on	O
the	O
top	O
what	O
you	O
see	O
is	O
query	O
likelihood	O
retrieval	O
function	O
all	O
right	O
this	O
one	O
.	O
and	O
then	O
kl-divergence	B
or	O
also	O
called	O
cross	O
entropy	O
retrieval	B
model	I
is	O
basically	O
to	O
generalize	O
the	O
frequency	O
part	O
here	O
into	O
a	O
layered	O
model	O
.	O
so	O
basically	O
it	O
s	O
the	O
difference	O
given	O
by	O
the	O
probabilistic	B
model	I
here	O
to	O
characterize	O
what	O
the	O
user	O
s	O
looking	O
for	O
versus	O
the	O
kind	O
of	O
query	B
words	I
there	O
.	O
and	O
this	O
difference	O
allows	O
us	O
to	O
plotting	O
various	O
different	O
ways	O
to	O
estimate	O
this	O
.	O
so	O
this	O
can	O
be	O
estimated	O
in	O
many	O
different	O
ways	O
including	O
using	O
feedback	O
information	O
.	O
now	O
this	O
is	O
called	O
a	O
kl-divergence	B
because	O
this	O
can	O
be	O
interpreted	O
as	O
measuring	O
the	O
kl-divergence	B
of	O
two	O
distributions	O
.	O
one	O
is	O
the	O
query	B
model	I
denoted	O
by	O
this	O
distribution	O
.	O
one	O
is	O
the	O
talking	O
the	O
language	B
model	I
here	O
.	O
and	O
though	O
is	O
a	O
language	B
model	I
of	O
course	O
.	O
and	O
we	O
are	O
not	O
going	O
to	O
talk	O
about	O
the	O
detail	O
of	O
that	O
and	O
you	O
ll	O
find	O
the	O
things	O
in	O
references	O
.	O
it	O
s	O
also	O
called	O
cross	O
entropy	O
because	O
in	O
in	O
fact	O
we	O
can	O
ignore	O
some	O
terms	O
in	O
the	O
kl-divergence	B
function	O
and	O
we	O
will	O
end	O
up	O
having	O
actually	O
cross	O
entropy	O
and	O
that	O
both	O
are	O
terms	O
in	O
information	O
theory	O
.	O
but	O
anyway	O
for	O
our	O
purposes	O
here	O
you	O
can	O
just	O
see	O
the	O
two	O
formulas	O
look	O
almost	O
identical	O
except	O
that	O
here	O
we	O
have	O
a	O
probability	O
of	O
a	O
word	O
given	O
by	O
a	O
query	O
language	B
model	I
.	O
this	O
and	O
here	O
the	O
sum	O
is	O
over	O
all	O
the	O
words	O
that	O
are	O
in	O
the	O
document	O
and	O
also	O
with	O
the	O
non-zero	O
probability	O
for	O
the	O
query	B
model	I
.	O
so	O
it	O
s	O
kind	O
of	O
again	O
a	O
generalization	O
of	O
sum	O
over	O
all	O
the	O
matching	O
query	B
words	I
.	O
now	O
you	O
can	O
also	O
easy	O
to	O
see	O
we	O
can	O
recover	O
the	O
query	O
likelihood	O
which	O
we	O
will	O
find	O
here	O
by	O
as	O
simple	O
as	O
setting	O
this	O
query	B
model	I
to	O
the	O
relative	O
frequency	O
of	O
a	O
word	O
in	O
the	O
query	O
right	O
this	O
is	O
very	O
to	O
easy	O
see	O
once	O
you	O
practice	O
this	O
.	O
and	O
to	O
here	O
you	O
can	O
eliminate	O
this	O
query	O
lens	O
that	O
s	O
a	O
constant	O
and	O
then	O
you	O
get	O
exactly	O
like	O
that	O
.	O
so	O
you	O
can	O
see	O
the	O
equivalence	O
.	O
and	O
that	O
s	O
also	O
why	O
this	O
kl-divergence	B
model	O
can	O
be	O
regarded	O
as	O
a	O
generalization	O
of	O
query	O
likelihood	O
because	O
we	O
can	O
cover	O
query	O
likelihood	O
as	O
a	O
special	O
case	O
but	O
it	O
would	O
also	O
allow	O
it	O
to	O
do	O
much	O
more	O
than	O
that	O
.	O
so	O
this	O
is	O
how	O
we	O
use	O
the	O
kl-divergence	B
model	O
to	O
then	O
do	O
feedback	O
.	O
the	O
picture	O
shows	O
that	O
we	O
first	O
estimate	O
a	O
document	O
language	B
model	I
then	O
we	O
estimate	O
a	O
query	O
language	B
model	I
and	O
we	O
compute	O
the	O
kl-divergence	B
this	O
is	O
often	O
denoted	O
by	O
a	O
d	O
here	O
.	O
but	O
this	O
basically	O
means	O
this	O
was	O
exactly	O
like	O
in	O
vector	B
space	I
model	I
because	O
we	O
compute	O
the	O
vector	O
for	O
the	O
document	O
in	O
the	O
computer	O
and	O
not	O
the	O
vector	O
for	O
the	O
query	O
and	O
then	O
we	O
compute	O
the	O
distance	O
.	O
only	O
that	O
these	O
vectors	O
are	O
of	O
special	O
forms	O
they	O
have	O
probability	O
distributions	O
.	O
and	O
then	O
we	O
get	O
the	O
results	O
and	O
we	O
can	O
find	O
some	O
feedback	O
documents	O
.	O
let	O
s	O
assume	O
they	O
are	O
more	O
selective	O
sorry	O
mostly	O
positive	O
documents	O
.	O
although	O
we	O
could	O
also	O
consider	O
both	O
kinds	O
of	O
documents	O
.	O
so	O
what	O
we	O
could	O
do	O
is	O
like	O
in	O
rocchio	B
we	O
can	O
compute	O
another	O
language	B
model	I
called	O
feedback	O
language	B
model	I
here	O
.	O
again	O
this	O
is	O
going	O
to	O
be	O
another	O
vector	O
just	O
like	O
a	O
computing	O
centroid	O
vector	O
in	O
rocchio	B
.	O
and	O
then	O
this	O
model	O
can	O
be	O
combined	O
with	O
the	O
original	O
query	B
model	I
using	O
a	O
linear	B
interpolation	B
.	O
and	O
this	O
would	O
then	O
give	O
us	O
an	O
updated	O
model	O
just	O
like	O
again	O
in	O
rocchio	B
.	O
right	O
so	O
here	O
we	O
can	O
see	O
the	O
parameter	O
of	O
our	O
controlling	O
amount	O
of	O
feedback	O
if	O
it	O
s	O
set	O
to	O
then	O
it	O
says	O
here	O
there	O
s	O
no	O
feedback	O
.	O
after	O
set	O
to	O
we	O
ve	O
got	O
full	O
feedback	O
we	O
can	O
ignore	O
the	O
original	O
query	O
.	O
and	O
this	O
is	O
generally	O
not	O
desirable	O
right	O
.	O
so	O
this	O
unless	O
you	O
are	O
absolutely	O
sure	O
you	O
have	O
seen	O
a	O
lot	O
of	O
relevant	B
documents	I
and	O
the	O
query	B
terms	I
are	O
not	O
important	O
.	O
so	O
of	O
course	O
the	O
main	O
question	O
here	O
is	O
how	O
do	O
you	O
compute	O
this	O
theta	O
f	O
this	O
is	O
the	O
big	O
question	O
here	O
.	O
and	O
once	O
you	O
can	O
do	O
that	O
the	O
rest	O
is	O
easy	O
.	O
so	O
here	O
we	O
ll	O
talk	O
about	O
one	O
of	O
the	O
approaches	O
.	O
and	O
there	O
are	O
many	O
approaches	O
of	O
course	O
.	O
this	O
approach	O
is	O
based	O
on	O
generative	O
model	O
and	O
i	O
m	O
going	O
to	O
show	O
you	O
how	O
it	O
works	O
.	O
this	O
is	O
a	O
user	O
generative	O
mixture	O
model	O
.	O
so	O
this	O
picture	O
shows	O
that	O
the	O
we	O
have	O
this	O
model	O
here	O
the	O
feedback	O
model	O
that	O
we	O
want	O
to	O
estimate	O
.	O
and	O
we	O
the	O
basis	O
is	O
the	O
feedback	O
options	O
.	O
let	O
s	O
say	O
we	O
are	O
observing	O
the	O
positive	O
documents	O
.	O
these	O
are	O
the	O
collected	O
documents	O
by	O
users	O
or	O
random	O
documents	O
judged	O
by	O
users	O
or	O
simply	O
top	O
ranked	O
documents	O
that	O
we	O
assumed	O
to	O
be	O
random	O
.	O
now	O
imagine	O
how	O
we	O
can	O
compute	O
a	O
centroid	O
for	O
these	O
documents	O
by	O
using	O
language	B
model	I
.	O
one	O
approach	O
is	O
simply	O
to	O
assume	O
these	O
documents	O
are	O
generated	O
from	O
this	O
language	B
model	I
as	O
we	O
did	O
before	O
.	O
what	O
we	O
could	O
do	O
is	O
do	O
it	O
just	O
normalize	O
the	O
word	B
frequency	I
here	O
.	O
and	O
then	O
we	O
we	O
ll	O
get	O
this	O
word	O
distribution	O
.	O
now	O
the	O
question	O
is	O
whether	O
this	O
distribution	O
is	O
good	O
for	O
feedback	O
.	O
well	O
you	O
can	O
imagine	O
well	O
the	O
top	O
rank	O
of	O
the	O
words	O
would	O
be	O
what	O
what	O
do	O
you	O
think	O
well	O
those	O
words	O
would	O
be	O
common	O
words	O
right	O
as	O
well	O
we	O
see	O
in	O
in	O
the	O
language	B
model	I
in	O
the	O
top	O
right	O
the	O
words	O
are	O
actually	O
common	O
words	O
like	O
the	O
et	O
cetera	O
.	O
so	O
it	O
s	O
not	O
very	O
good	O
for	O
feedback	O
because	O
we	O
will	O
be	O
adding	O
a	O
lot	O
of	O
such	O
words	O
to	O
our	O
query	O
when	O
we	O
interpret	O
this	O
was	O
the	O
original	O
query	B
model	I
.	O
so	O
this	O
is	O
not	O
good	O
so	O
we	O
need	O
to	O
do	O
something	O
in	O
particular	O
we	O
are	O
trying	O
to	O
get	O
rid	O
of	O
those	O
common	O
words	O
.	O
and	O
we	O
all	O
we	O
have	O
seen	O
actually	O
one	O
way	O
to	O
do	O
that	O
by	O
using	O
background	O
language	B
model	I
in	O
the	O
case	O
of	O
learning	O
the	O
associations	O
with	O
of	O
words	O
right	O
.	O
the	O
words	O
that	O
are	O
related	O
to	O
the	O
word	O
computer	O
.	O
we	O
could	O
do	O
that	O
and	O
that	O
would	O
be	O
another	O
way	O
to	O
do	O
this	O
.	O
but	O
here	O
we	O
re	O
going	O
to	O
talk	O
about	O
another	O
approach	O
which	O
is	O
a	O
more	O
principled	O
approach	O
.	O
in	O
this	O
case	O
we	O
re	O
going	O
to	O
say	O
well	O
you	O
you	O
said	O
that	O
there	O
are	O
common	O
words	O
here	O
in	O
this	O
these	O
documents	O
that	O
should	O
not	O
belong	O
to	O
this	O
top	O
model	O
right	O
so	O
now	O
what	O
we	O
can	O
do	O
is	O
to	O
assume	O
that	O
well	O
those	O
words	O
are	O
generally	O
from	O
background	O
language	B
model	I
so	O
they	O
will	O
generate	O
a	O
those	O
words	O
like	O
the	O
for	O
example	O
.	O
and	O
if	O
we	O
use	O
maximum	B
likelihood	I
estimated	O
note	O
that	O
if	O
all	O
the	O
words	O
here	O
must	O
be	O
generated	O
from	O
this	O
model	O
then	O
this	O
model	O
is	O
forced	O
to	O
assign	O
high	O
probabilities	O
to	O
a	O
word	O
like	O
the	O
because	O
it	O
occurs	O
so	O
frequently	O
here	O
.	O
note	O
that	O
in	O
order	O
to	O
reduce	O
its	O
probability	O
in	O
this	O
model	O
we	O
have	O
to	O
have	O
another	O
model	O
which	O
is	O
this	O
one	O
to	O
help	O
explain	O
the	O
word	O
the	O
here	O
.	O
and	O
in	O
this	O
case	O
it	O
s	O
not	O
appropriate	O
to	O
use	O
the	O
background	O
language	B
model	I
to	O
achieve	O
this	O
goal	O
because	O
this	O
model	O
will	O
assign	O
high	O
probabilities	O
to	O
these	O
common	O
words	O
.	O
so	O
in	O
this	O
approach	O
then	O
we	O
assume	O
this	O
machine	O
that	O
which	O
generated	O
these	O
words	O
would	O
work	O
as	O
follows	O
.	O
we	O
have	O
a	O
source	O
controller	O
here	O
.	O
imagine	O
we	O
flip	O
a	O
coin	O
here	O
to	O
decide	O
what	O
distribution	O
to	O
use	O
.	O
with	O
the	O
probability	O
of	O
lambda	O
the	O
coin	O
shows	O
up	O
as	O
head	O
.	O
and	O
then	O
we	O
re	O
going	O
to	O
use	O
the	O
background	O
language	B
model	I
.	O
and	O
we	O
can	O
do	O
then	O
sample	O
word	O
from	O
that	O
model	O
.	O
with	O
probability	O
of	O
minus	O
lambda	O
now	O
we	O
now	O
decide	O
to	O
use	O
a	O
unknown	O
topic	O
model	O
here	O
that	O
we	O
will	O
try	O
to	O
estimate	O
.	O
and	O
we	O
re	O
going	O
to	O
then	O
generate	O
a	O
word	O
here	O
.	O
if	O
we	O
make	O
this	O
assumption	O
and	O
this	O
whole	O
thing	O
will	O
be	O
just	O
one	O
model	O
and	O
we	O
call	O
this	O
a	O
mixture	O
model	O
because	O
there	O
are	O
two	O
distributions	O
that	O
are	O
mixed	O
here	O
together	O
.	O
and	O
we	O
actually	O
don	O
t	O
know	O
when	O
each	O
distribution	O
is	O
used	O
.	O
right	O
so	O
again	O
think	O
of	O
this	O
whole	O
thing	O
as	O
one	O
model	O
.	O
and	O
we	O
can	O
still	O
ask	O
it	O
for	O
words	O
and	O
it	O
will	O
still	O
give	O
us	O
a	O
word	O
in	O
a	O
random	O
method	O
right	O
and	O
of	O
course	O
which	O
word	O
will	O
show	O
up	O
will	O
depend	O
on	O
both	O
this	O
distribution	O
and	O
that	O
distribution	O
.	O
in	O
addition	O
it	O
would	O
also	O
depend	O
on	O
this	O
lambda	O
because	O
if	O
you	O
say	O
lambda	O
is	O
very	O
high	O
and	O
it	O
s	O
going	O
to	O
always	O
use	O
the	O
background	O
distribution	O
you	O
ll	O
get	O
different	O
words	O
.	O
if	O
you	O
say	O
well	O
our	O
lambda	O
is	O
very	O
small	O
we	O
re	O
going	O
to	O
use	O
this	O
all	O
right	O
so	O
all	O
these	O
are	O
parameters	O
in	O
this	O
model	O
.	O
and	O
then	O
if	O
you	O
re	O
thinking	O
this	O
way	O
basically	O
we	O
can	O
do	O
exactly	O
the	O
same	O
as	O
what	O
we	O
did	O
before	O
we	O
re	O
going	O
to	O
use	O
maximum	B
likelihood	I
estimator	I
to	O
adjust	O
this	O
model	O
to	O
estimate	O
the	O
parameters	O
.	O
basically	O
we	O
re	O
going	O
to	O
adjust	O
well	O
this	O
parameter	O
so	O
that	O
we	O
can	O
best	O
explain	O
all	O
the	O
data	O
.	O
the	O
difference	O
now	O
is	O
that	O
we	O
are	O
not	O
asking	O
this	O
model	O
alone	O
to	O
explain	O
this	O
.	O
but	O
rather	O
we	O
re	O
going	O
to	O
ask	O
this	O
whole	O
model	O
mixture	O
model	O
to	O
explain	O
the	O
data	O
because	O
it	O
has	O
got	O
some	O
help	O
from	O
the	O
background	O
model	O
.	O
it	O
doesn	O
t	O
have	O
to	O
assign	O
high	O
probabilities	O
towards	O
like	O
the	O
as	O
a	O
result	O
.	O
it	O
would	O
then	O
assign	O
high	O
probabilities	O
to	O
other	O
words	O
that	O
are	O
common	O
here	O
but	O
not	O
having	O
high	O
probability	O
here	O
.	O
so	O
those	O
would	O
be	O
common	O
here	O
.	O
right	O
and	O
if	O
they	O
re	O
common	O
they	O
would	O
have	O
to	O
have	O
high	O
probabilities	O
according	O
to	O
a	O
maximum	B
likelihood	I
estimator	I
.	O
and	O
if	O
they	O
are	O
rare	O
here	O
all	O
right	O
so	O
if	O
they	O
are	O
rare	O
here	O
then	O
you	O
don	O
t	O
get	O
much	O
help	O
from	O
this	O
background	O
model	O
.	O
as	O
a	O
result	O
this	O
topic	O
model	O
must	O
assign	O
high	O
probabilities	O
.	O
so	O
the	O
higher	O
probability	O
words	O
according	O
to	O
the	O
topic	O
model	O
will	O
be	O
those	O
that	O
are	O
common	O
here	O
but	O
rare	O
in	O
the	O
background	O
.	O
okay	O
so	O
this	O
is	O
basically	O
a	O
little	O
bit	O
like	O
a	O
idea	O
for	O
weighting	O
here	O
.	O
this	O
would	O
allow	O
us	O
to	O
achieve	O
the	O
effect	O
of	O
removing	O
these	O
top	O
words	O
that	O
are	O
meaningless	O
in	O
the	O
feedback	O
.	O
so	O
mathematically	O
what	O
we	O
have	O
is	O
to	O
compute	O
the	O
likelihood	O
again	O
local	O
likelihood	O
of	O
the	O
feedback	O
documents	O
.	O
and	O
and	O
note	O
that	O
we	O
also	O
have	O
another	O
parameter	O
lambda	O
here	O
.	O
but	O
we	O
assume	O
that	O
lambda	O
denotes	O
noise	O
in	O
the	O
feedback	O
document	O
.	O
so	O
we	O
are	O
going	O
to	O
let	O
s	O
say	O
set	O
this	O
to	O
a	O
parameter	O
let	O
s	O
say	O
say	O
of	O
the	O
words	O
are	O
noise	O
or	O
are	O
noise	O
.	O
and	O
this	O
can	O
then	O
be	O
assume	O
it	O
will	O
be	O
fixed	O
.	O
if	O
we	O
assume	O
this	O
is	O
fixed	O
then	O
we	O
only	O
have	O
these	O
probabilities	O
as	O
parameters	O
just	O
like	O
in	O
the	O
simplest	O
unigram	O
language	B
model	I
we	O
have	O
n	O
parameters	O
.	O
n	O
is	O
the	O
number	O
of	O
words	O
and	O
then	O
the	O
likelihood	O
function	O
will	O
look	O
like	O
this	O
.	O
it	O
s	O
very	O
similar	O
to	O
the	O
likelihood	O
function	O
normal	O
likelihood	O
function	O
we	O
see	O
before	O
except	O
that	O
inside	O
the	O
logarithm	O
there	O
s	O
a	O
sum	O
in	O
here	O
.	O
and	O
this	O
sum	O
is	O
because	O
we	O
can	O
see	O
the	O
two	O
distributions	O
.	O
and	O
which	O
ones	O
used	O
would	O
depend	O
on	O
lambda	O
and	O
that	O
s	O
why	O
we	O
have	O
this	O
form	O
.	O
but	O
mathematically	O
this	O
is	O
the	O
function	O
with	O
theta	O
as	O
unknown	O
variables	O
right	O
so	O
this	O
is	O
just	O
a	O
function	O
.	O
all	O
the	O
other	O
variables	O
are	O
known	O
except	O
for	O
this	O
guy	O
.	O
so	O
we	O
can	O
then	O
choose	O
this	O
probability	O
distribution	O
to	O
maximize	O
this	O
log	O
likelihood	O
.	O
the	O
same	O
idea	O
as	O
the	O
maximum	B
likelihood	I
estimator	I
.	O
as	O
a	O
mathematical	O
problem	O
which	O
is	O
to	O
we	O
just	O
have	O
to	O
solve	O
this	O
optimization	O
problem	O
.	O
we	O
said	O
we	O
would	O
try	O
all	O
of	O
the	O
theta	O
values	O
and	O
here	O
we	O
find	O
one	O
that	O
gives	O
this	O
whole	O
thing	O
the	O
maximum	O
probability	O
.	O
so	O
it	O
s	O
a	O
well-defined	O
math	O
problem	O
.	O
once	O
we	O
have	O
done	O
that	O
we	O
obtain	O
this	O
theta	O
f	O
that	O
can	O
be	O
the	O
interpreter	O
with	O
the	O
original	O
query	B
model	I
to	O
do	O
feedback	O
.	O
so	O
here	O
are	O
some	O
examples	O
of	O
the	O
feedback	O
model	O
learned	O
from	O
a	O
web	O
document	O
collection	O
and	O
we	O
do	O
pseudo-feedback	B
.	O
we	O
just	O
use	O
the	O
top	O
documents	O
and	O
we	O
use	O
this	O
mixture	O
model	O
.	O
so	O
the	O
query	O
is	O
airport	O
security	O
.	O
what	O
we	O
do	O
is	O
we	O
first	O
retrieve	O
ten	O
documents	O
from	O
the	O
web	O
database	O
.	O
and	O
this	O
is	O
of	O
course	O
pseudo-feedback	B
right	O
and	O
then	O
we	O
re	O
going	O
to	O
feed	O
to	O
that	O
mixture	O
model	O
to	O
this	O
ten	O
document	O
set	O
.	O
and	O
these	O
are	O
the	O
words	O
learned	O
using	O
this	O
approach	O
.	O
this	O
is	O
the	O
probability	O
of	O
a	O
word	O
given	O
by	O
the	O
feedback	O
model	O
in	O
both	O
cases	O
.	O
so	O
in	O
both	O
cases	O
you	O
can	O
see	O
the	O
highest	O
probability	O
of	O
words	O
include	O
very	O
random	O
words	O
to	O
the	O
query	O
.	O
so	O
airport	O
security	O
for	O
example	O
these	O
query	B
words	I
still	O
show	O
up	O
as	O
high	O
probabilities	O
in	O
each	O
case	O
naturally	O
because	O
they	O
occur	O
frequently	O
in	O
the	O
top	O
rank	O
of	O
documents	O
.	O
but	O
we	O
also	O
see	O
beverage	O
alcohol	O
bomb	O
terrorist	O
et	O
cetera	O
.	O
right	O
so	O
these	O
are	O
relevant	O
to	O
this	O
topic	O
and	O
they	O
if	O
combined	O
with	O
original	O
query	O
can	O
help	O
us	O
match	O
more	O
accurately	O
on	O
documents	O
.	O
and	O
also	O
they	O
can	O
help	O
us	O
bring	O
up	O
documents	O
that	O
only	O
managing	O
the	O
some	O
of	O
these	O
other	O
words	O
.	O
and	O
maybe	O
for	O
example	O
just	O
airport	O
and	O
then	O
bomb	O
for	O
example	O
.	O
these	O
so	O
this	O
is	O
how	O
pseudo-feedback	B
works	O
.	O
it	O
shows	O
that	O
this	O
model	O
really	O
works	O
and	O
picks	O
up	O
mm	O
some	O
related	O
words	O
to	O
the	O
query	O
.	O
what	O
s	O
also	O
interesting	O
is	O
that	O
if	O
you	O
look	O
at	O
the	O
two	O
tables	O
here	O
and	O
you	O
compare	O
them	O
and	O
you	O
see	O
in	O
this	O
case	O
when	O
lambda	O
is	O
set	O
to	O
a	O
small	O
value	O
and	O
we	O
ll	O
still	O
see	O
some	O
common	O
words	O
here	O
and	O
that	O
means	O
.	O
when	O
we	O
don	O
t	O
use	O
the	O
background	O
model	O
often	O
remember	O
lambda	O
can	O
use	O
the	O
probability	O
of	O
using	O
the	O
background	O
model	O
to	O
generate	O
to	O
the	O
text	O
.	O
if	O
we	O
don	O
t	O
rely	O
much	O
on	O
background	O
model	O
we	O
still	O
have	O
to	O
use	O
this	O
topped	O
model	O
to	O
account	O
for	O
the	O
common	O
words	O
.	O
whereas	O
if	O
we	O
set	O
lambda	O
to	O
a	O
very	O
high	O
value	O
we	O
would	O
use	O
the	O
background	O
model	O
very	O
often	O
to	O
explain	O
these	O
words	O
then	O
there	O
is	O
no	O
burden	O
on	O
expanding	O
those	O
common	O
words	O
in	O
the	O
feedback	O
documents	O
by	O
the	O
topping	O
model	O
.	O
so	O
as	O
a	O
result	O
the	O
top	O
of	O
the	O
model	O
here	O
is	O
very	O
discriminative	O
.	O
it	O
contains	O
all	O
the	O
relevant	O
words	O
without	O
common	O
words	O
.	O
so	O
this	O
can	O
be	O
added	O
to	O
the	O
original	O
query	O
to	O
achieve	O
feedback	O
.	O
so	O
to	O
summarize	O
in	O
this	O
lecture	O
we	O
have	O
talked	O
about	O
the	O
feedback	O
in	O
language	B
model	I
approach	O
.	O
in	O
general	O
feedback	O
is	O
to	O
learn	O
from	O
examples	O
.	O
these	O
examples	O
can	O
be	O
assumed	O
examples	O
can	O
be	O
pseudo-examples	O
like	O
assume	O
the	O
the	O
top	O
ten	O
documents	O
are	O
assumed	O
to	O
be	O
random	O
.	O
they	O
could	O
be	O
based	O
on	O
using	O
fractions	O
like	O
feedback	O
based	O
on	O
quick	O
sorts	O
or	O
implicit	B
feedback	I
.	O
we	O
talked	O
about	O
the	O
three	O
major	O
feedback	O
scenarios	O
relevance	B
feedback	I
pseudo-feedback	B
and	O
implicit	B
feedback	I
.	O
we	O
talked	O
about	O
how	O
to	O
use	O
rocchio	B
to	O
do	O
feedback	O
in	O
vector-space	O
model	O
and	O
how	O
to	O
use	O
query	B
model	I
estimation	O
for	O
feedback	O
in	O
language	B
model	I
.	O
and	O
we	O
briefly	O
talked	O
about	O
the	O
mixture	O
model	O
and	O
the	O
basic	O
idea	O
and	O
there	O
are	O
many	O
other	O
methods	O
.	O
for	O
example	O
the	O
relevance	O
model	O
is	O
a	O
very	O
effective	O
model	O
for	O
estimating	O
query	B
model	I
.	O
so	O
you	O
can	O
read	O
more	O
about	O
the	O
these	O
methods	O
in	O
the	O
references	O
that	O
are	O
listed	O
at	O
the	O
end	O
of	O
this	O
lecture	O
.	O
so	O
there	O
are	O
two	O
additional	O
readings	O
here	O
.	O
the	O
first	O
one	O
is	O
a	O
book	O
that	O
has	O
a	O
systematic	O
review	O
and	O
discussion	O
of	O
language	B
models	I
of	O
more	O
information	B
retrieval	I
.	O
and	O
the	O
second	O
one	O
is	O
an	O
important	O
research	O
paper	O
that	O
s	O
about	O
relevance	O
based	O
language	B
models	I
and	O
it	O
s	O
a	O
very	O
effective	O
way	O
of	O
computing	O
query	B
model	I
.	O
this	O
lecture	O
is	O
about	O
the	O
web	B
search	I
.	O
in	O
this	O
lecture	O
we	O
are	O
going	O
to	O
talk	O
about	O
one	O
of	O
the	O
most	O
important	O
applications	O
of	O
text	B
retrieval	I
web	B
search	B
engines	I
.	O
so	O
let	O
s	O
first	O
look	O
at	O
some	O
general	O
challenges	O
and	O
opportunities	O
in	O
web	B
search	I
.	O
now	O
many	O
information	B
retrieval	I
algorithms	O
had	O
been	O
developed	O
at	O
the	O
before	O
the	O
web	O
was	O
born	O
.	O
so	O
when	O
the	O
web	O
was	O
born	O
it	O
created	O
the	O
best	O
opportunity	O
to	O
apply	O
those	O
algorithms	O
to	O
major	O
application	O
problem	O
that	O
everyone	O
would	O
care	O
about	O
.	O
so	O
naturally	O
there	O
had	O
to	O
be	O
some	O
further	O
extensions	O
of	O
the	O
classical	O
search	O
algorithms	O
to	O
address	O
some	O
new	O
challenges	O
encountered	O
in	O
web	B
search	I
.	O
so	O
here	O
are	O
some	O
general	O
challenges	O
.	O
firstly	O
this	O
is	O
a	O
scalability	O
challenge	O
.	O
how	O
we	O
handle	O
the	O
size	O
of	O
the	O
web	O
and	O
ensure	O
completeness	O
of	O
coverage	O
of	O
all	O
the	O
information	O
.	O
how	O
to	O
serve	O
many	O
users	O
quickly	O
and	O
by	O
answering	O
all	O
their	O
queries	O
.	O
all	O
right	O
so	O
that	O
s	O
one	O
major	O
challenge	O
.	O
and	O
before	O
the	O
web	O
was	O
born	O
the	O
scale	O
of	O
search	O
was	O
relatively	O
small	O
.	O
the	O
second	O
problem	O
is	O
that	O
there	O
is	O
low	O
quality	O
information	O
.	O
and	O
there	O
are	O
often	O
spams	O
.	O
the	O
third	O
challenge	O
is	O
dynamics	O
of	O
the	O
web	O
.	O
the	O
new	O
pages	O
are	O
constantly	O
created	O
and	O
some	O
pages	O
may	O
be	O
updated	O
eve-	O
very	O
quickly	O
.	O
so	O
it	O
makes	O
it	O
harder	O
to	O
keep	O
the	O
index	O
fresh	O
.	O
so	O
these	O
are	O
some	O
of	O
the	O
challenges	O
that	O
the	O
we	O
have	O
to	O
solve	O
in	O
order	O
to	O
build	O
a	O
high	O
quality	O
web	B
search	B
engine	I
.	O
on	O
the	O
other	O
hand	O
there	O
are	O
also	O
some	O
interesting	O
opportunities	O
that	O
we	O
can	O
leverage	O
to	O
improve	O
search	O
results	O
.	O
there	O
are	O
many	O
additional	O
heuristics	O
.	O
for	O
example	O
you	O
know	O
using	O
links	O
that	O
we	O
can	O
leverage	O
to	O
improve	O
scoring	O
.	O
now	O
the	O
errors	O
that	O
we	O
talked	O
about	O
such	O
as	O
the	O
vector	B
space	I
model	I
are	O
general	O
algorithms	O
.	O
and	O
they	O
can	O
be	O
applied	O
to	O
any	O
search	O
applications	O
so	O
that	O
s	O
the	O
advantage	O
.	O
on	O
the	O
other	O
hand	O
they	O
also	O
don	O
t	O
take	O
advantage	O
of	O
special	O
characteristics	O
of	O
pages	O
or	O
documents	O
in	O
the	O
specific	O
applications	O
such	O
as	O
web	B
search	I
.	O
web	O
pages	O
are	O
linked	O
with	O
each	O
other	O
so	O
obviously	O
the	O
linking	O
is	O
something	O
that	O
we	O
can	O
also	O
leverage	O
.	O
so	O
because	O
of	O
these	O
challenges	O
and	O
opportunities	O
there	O
are	O
new	O
techniques	O
that	O
have	O
been	O
developed	O
for	O
web	B
search	I
or	O
due	O
to	O
the	O
need	O
of	O
a	O
web	B
search	I
.	O
one	O
is	O
parallel	O
indexing	O
and	O
searching	O
and	O
this	O
is	O
to	O
address	O
the	O
issue	O
of	O
scalability	O
in	O
particular	O
google	O
s	O
imaging	O
of	O
mapreduce	B
is	O
very	O
inferential	O
and	O
has	O
been	O
very	O
helpful	O
in	O
that	O
aspect	O
.	O
second	O
there	O
are	O
techniques	O
that	O
are	O
developed	O
for	O
addressing	O
the	O
problem	O
of	O
spams	O
.	O
so	O
spam	O
detection	O
.	O
we	O
ll	O
have	O
to	O
prevent	O
those	O
spam	O
pages	O
from	O
being	O
ranked	O
high	O
.	O
and	O
there	O
are	O
also	O
techniques	O
to	O
achieve	O
robust	O
ranking	O
.	O
and	O
we	O
re	O
going	O
to	O
use	O
a	O
lot	O
of	O
signals	O
to	O
rank	O
pages	O
so	O
that	O
it	O
s	O
not	O
easy	O
to	O
spam	O
the	O
search	B
engine	I
with	O
particular	O
tricks	O
.	O
and	O
the	O
third	O
line	O
of	O
techniques	O
is	O
link	B
analysis	I
.	O
and	O
these	O
are	O
techniques	O
that	O
can	O
allow	O
us	O
to	O
to	O
improve	O
search	O
results	O
by	O
leveraging	O
extra	O
information	O
.	O
and	O
in	O
general	O
in	O
web	B
search	I
we	O
re	O
going	O
to	O
use	O
multiple	O
features	O
for	O
ranking	O
.	O
not	O
just	O
link	B
analysis	I
but	O
also	O
exploiting	O
all	O
kinds	O
of	O
crawls	O
like	O
the	O
layout	O
of	O
web	O
pages	O
or	O
anchor	B
text	I
that	O
describes	O
a	O
link	O
to	O
another	O
page	O
.	O
so	O
here	O
s	O
a	O
picture	O
showing	O
the	O
basic	O
search	B
engine	I
technologies	O
.	O
basically	O
this	O
is	O
the	O
web	O
on	O
the	O
left	O
and	O
then	O
user	O
on	O
the	O
right	O
side	O
.	O
and	O
we	O
re	O
going	O
to	O
help	O
these	O
this	O
user	O
get	O
access	O
to	O
the	O
web	O
information	O
.	O
and	O
the	O
first	O
component	O
is	O
the	O
crawler	O
that	O
with	O
the	O
crawl	O
pages	O
and	O
the	O
second	O
component	O
is	O
indexer	B
.	O
that	O
will	O
take	O
these	O
pages	O
create	O
an	O
invert	O
index	O
.	O
the	O
third	O
component	O
that	O
is	O
a	O
retrieval	O
not	O
with	O
the	O
using	O
but	O
the	O
index	O
to	O
answer	O
user	O
s	O
query	O
by	O
talking	O
to	O
the	O
user	O
s	O
browser	O
.	O
and	O
then	O
the	O
search	O
results	O
would	O
be	O
given	O
to	O
the	O
user	O
.	O
and	O
and	O
then	O
the	O
browser	O
will	O
show	O
those	O
results	O
and	O
to	O
allow	O
the	O
user	O
to	O
interact	O
with	O
the	O
web	O
.	O
so	O
we	O
re	O
going	O
to	O
talk	O
about	O
each	O
of	O
these	O
component	O
.	O
first	O
we	O
re	O
going	O
to	O
talk	O
about	O
the	O
crawler	O
also	O
called	O
a	O
spider	O
or	O
a	O
software	O
robot	O
that	O
would	O
do	O
something	O
like	O
a	O
crawling	B
pages	O
on	O
the	O
web	O
.	O
to	O
build	O
a	O
toy	O
crawler	O
is	O
relatively	O
easy	O
because	O
you	O
just	O
need	O
to	O
start	O
with	O
a	O
set	O
of	O
seed	O
pages	O
and	O
then	O
fetch	O
pages	O
from	O
the	O
web	O
and	O
parse	O
these	O
pages	O
new	O
links	O
.	O
and	O
then	O
add	O
them	O
to	O
the	O
priority	O
of	O
q	O
and	O
then	O
just	O
explore	O
those	O
additional	O
links	O
right	O
.	O
but	O
to	O
build	O
a	O
real	O
crawler	O
actually	O
is	O
tricky	O
and	O
there	O
are	O
some	O
complicated	O
issues	O
that	O
we	O
have	O
do	O
deal	O
with	O
.	O
for	O
example	O
robustness	O
what	O
if	O
the	O
server	O
doesn	O
t	O
respond	O
.	O
what	O
if	O
there	O
s	O
a	O
trap	O
that	O
generates	O
dynamically	O
generated	O
webpages	O
that	O
might	O
attract	O
your	O
crawler	O
to	O
keep	O
crawling	B
the	O
same	O
site	O
and	O
to	O
fetch	O
dynamically	O
generated	O
pages	O
.	O
the	O
results	O
of	O
this	O
issue	O
of	O
crawling	B
and	O
you	O
don	O
t	O
want	O
to	O
overload	O
one	O
particular	O
server	O
with	O
many	O
crawling	B
requests	O
.	O
and	O
you	O
have	O
to	O
respect	O
the	O
the	O
robot	O
exclusion	O
protocol	O
.	O
you	O
also	O
need	O
to	O
handle	O
different	O
types	O
of	O
files	O
.	O
there	O
are	O
images	O
pdf	O
files	O
all	O
kinds	O
of	O
formats	O
on	O
the	O
web	O
.	O
and	O
you	O
have	O
to	O
also	O
consider	O
url	O
extensions	O
.	O
so	O
sometimes	O
those	O
are	O
cgi	O
scripts	O
and	O
you	O
know	O
internal	O
references	O
etc	O
.	O
and	O
sometimes	O
you	O
have	O
javascripts	O
on	O
the	O
page	O
that	O
they	O
also	O
create	O
challenges	O
.	O
and	O
you	O
ideally	O
should	O
also	O
recognize	O
the	O
pages	O
because	O
you	O
don	O
t	O
have	O
to	O
duplicate	O
to	O
the	O
those	O
pages	O
.	O
and	O
finally	O
you	O
may	O
be	O
interesting	O
to	O
discover	O
hidden	O
urls	O
.	O
those	O
are	O
urls	O
that	O
may	O
not	O
be	O
linked	O
to	O
any	O
page	O
.	O
but	O
if	O
you	O
truncate	O
the	O
url	O
to	O
shorter	O
pass	O
you	O
might	O
be	O
able	O
to	O
get	O
some	O
additional	O
pages	O
.	O
so	O
what	O
are	O
the	O
major	O
crawling	B
strategies	O
in	O
general	O
breadth-first	O
is	O
most	O
common	O
because	O
it	O
naturally	O
balance	O
balances	O
server	O
load	O
.	O
you	O
would	O
not	O
keep	O
probing	O
a	O
particular	O
server	O
	O
.	O
also	O
parallel	O
crawling	B
is	O
very	O
natural	O
because	O
this	O
task	O
is	O
very	O
easy	O
to	O
parallelise	O
and	O
there	O
are	O
some	O
variations	O
of	O
the	O
crawling	B
task	O
.	O
one	O
interesting	O
variation	O
is	O
called	O
focused	B
crawling	B
.	O
in	O
this	O
kind	O
we	O
re	O
going	O
to	O
crawl	O
just	O
some	O
pages	O
about	O
a	O
particular	O
topic	O
.	O
for	O
example	O
all	O
pages	O
about	O
automobiles	O
.	O
and	O
and	O
this	O
is	O
typically	O
going	O
to	O
start	O
with	O
a	O
query	O
and	O
then	O
you	O
can	O
use	O
the	O
query	O
to	O
get	O
some	O
results	O
.	O
from	O
the	O
major	O
search	B
engine	I
.	O
and	O
then	O
you	O
can	O
start	O
it	O
with	O
those	O
results	O
and	O
gradually	O
crawl	O
more	O
.	O
so	O
one	O
challenge	O
in	O
crawling	B
is	O
to	O
find	O
the	O
new	O
pages	O
that	O
people	O
have	O
created	O
and	O
people	O
probably	O
are	O
creating	O
new	O
pages	O
all	O
the	O
time	O
and	O
this	O
is	O
very	O
challenging	O
if	O
the	O
new	O
pages	O
have	O
not	O
been	O
actually	O
linked	O
to	O
any	O
old	O
page	O
.	O
if	O
they	O
are	O
then	O
you	O
can	O
probably	O
refine	O
them	O
by	O
recrawling	O
the	O
older	O
page	O
.	O
so	O
these	O
are	O
also	O
some	O
um	O
interesting	O
challenges	O
that	O
have	O
to	O
be	O
solved	O
.	O
and	O
finally	O
we	O
might	O
face	O
the	O
scenario	O
of	O
incremental	O
crawling	B
or	O
repeated	O
crawling	B
.	O
right	O
so	O
your	O
first	O
let	O
s	O
say	O
if	O
you	O
want	O
to	O
be	O
able	O
to	O
web	B
search	B
engine	I
.	O
and	O
you	O
were	O
the	O
first	O
to	O
crawl	O
a	O
lot	O
of	O
data	O
from	O
the	O
web	O
.	O
and	O
then	O
but	O
then	O
once	O
you	O
have	O
collected	O
all	O
the	O
data	O
and	O
in	O
future	O
we	O
just	O
need	O
to	O
crawl	O
the	O
the	O
update	O
pages	O
.	O
you	O
you	O
in	O
general	O
you	O
don	O
t	O
have	O
to	O
re-crawl	O
everything	O
right	O
or	O
it	O
s	O
not	O
necessary	O
.	O
so	O
in	O
this	O
case	O
you	O
you	O
go	O
as	O
you	O
minimize	O
a	O
resource	O
overhead	O
by	O
using	O
minimum	O
resource	O
to	O
to	O
just	O
still	O
crawl	O
updated	O
pages	O
.	O
so	O
this	O
is	O
after	O
a	O
very	O
interesting	O
research	O
question	O
here	O
.	O
and	O
research	O
question	O
is	O
that	O
there	O
aren	O
t	O
many	O
standard	O
algorithms	O
for	O
doing	O
this	O
this	O
task	O
.	O
right	O
but	O
in	O
general	O
you	O
can	O
imagine	O
you	O
can	O
learn	O
from	O
the	O
past	O
experience	O
.	O
right	O
.	O
so	O
the	O
two	O
major	O
factors	O
that	O
you	O
have	O
to	O
consider	O
are	O
first	O
will	O
this	O
page	O
be	O
updated	O
frequently	O
and	O
do	O
i	O
have	O
to	O
crawl	O
this	O
page	O
again	O
if	O
the	O
page	O
is	O
a	O
static	O
page	O
that	O
hasn	O
t	O
been	O
changed	O
for	O
months	O
you	O
probably	O
don	O
t	O
have	O
to	O
re-crawl	O
it	O
everyday	O
right	O
because	O
it	O
s	O
unlikely	O
that	O
it	O
will	O
be	O
changed	O
frequently	O
.	O
on	O
the	O
other	O
hand	O
if	O
it	O
s	O
you	O
know	O
sports	O
score	O
page	O
that	O
gets	O
updated	O
very	O
frequently	O
and	O
you	O
may	O
need	O
to	O
re-crawl	O
it	O
maybe	O
even	O
multiple	O
times	O
on	O
the	O
same	O
day	O
.	O
the	O
other	O
factor	O
to	O
consider	O
is	O
is	O
this	O
page	O
frequently	O
accessed	O
by	O
users	O
if	O
it	O
if	O
it	O
is	O
that	O
means	O
it	O
s	O
a	O
high	O
utility	O
page	O
and	O
then	O
thus	O
it	O
s	O
more	O
important	O
to	O
ensure	O
such	O
a	O
page	O
to	O
be	O
fresh	O
.	O
compare	O
it	O
with	O
another	O
page	O
that	O
has	O
never	O
been	O
fetched	O
by	O
any	O
users	O
for	O
a	O
year	O
.	O
than	O
even	O
though	O
that	O
page	O
has	O
been	O
changed	O
a	O
lot	O
then	O
it	O
s	O
probably	O
not	O
necessary	O
to	O
crawl	O
that	O
page	O
or	O
at	O
least	O
it	O
s	O
not	O
as	O
urgent	O
as	O
to	O
maintain	O
the	O
freshness	O
of	O
frequently	O
accessed	O
page	O
by	O
users	O
.	O
so	O
to	O
summarize	O
web	B
search	I
is	O
one	O
of	O
the	O
most	O
important	O
applications	O
of	O
text	B
retrieval	I
.	O
and	O
there	O
are	O
some	O
new	O
challenges	O
particularly	O
scalability	O
efficiency	O
quality	O
information	O
.	O
there	O
are	O
also	O
new	O
opportunities	O
particularly	O
rich	O
link	O
information	O
and	O
layout	O
et	O
cetera	O
.	O
crawler	O
is	O
an	O
essential	O
component	O
of	O
web	B
search	I
applications	O
.	O
and	O
in	O
general	O
we	O
can	O
classify	O
two	O
scenarios	O
.	O
once	O
is	O
initial	O
crawling	B
and	O
here	O
we	O
want	O
to	O
have	O
complete	O
crawling	B
of	O
the	O
web	O
if	O
you	O
are	O
doing	O
a	O
general	O
search	B
engine	I
or	O
focus	O
crawling	B
if	O
you	O
want	O
to	O
just	O
target	O
it	O
at	O
a	O
certain	O
type	O
of	O
pages	O
.	O
and	O
then	O
there	O
is	O
another	O
scenario	O
that	O
s	O
incremental	O
updating	O
of	O
the	O
crawl	O
data	O
or	O
incremental	O
crawling	B
.	O
in	O
this	O
case	O
you	O
need	O
to	O
optimize	O
the	O
resource	O
.	O
for	O
to	O
use	O
minimum	O
resource	O
we	O
get	O
the	O
	O
.	O
	O
.	O
this	O
lecture	O
is	O
about	O
web	O
indexing	O
.	O
in	O
this	O
lecture	O
we	O
will	O
continue	O
talking	O
about	O
web	B
search	I
and	O
we	O
re	O
going	O
to	O
talk	O
about	O
how	O
to	O
create	O
a	O
web	O
scale	O
index	O
.	O
so	O
once	O
we	O
crawl	O
the	O
web	O
we	O
ve	O
got	O
a	O
lot	O
of	O
web	O
pages	O
.	O
the	O
next	O
step	O
is	O
we	O
use	O
the	O
indexer	B
to	O
create	O
the	O
inverted	B
index	I
.	O
in	O
general	O
we	O
can	O
use	O
the	O
standard	O
information	B
retrieval	B
techniques	I
for	O
creating	O
the	O
index	O
and	O
that	O
is	O
what	O
we	O
talked	O
about	O
in	O
the	O
previous	O
lecture	O
.	O
but	O
there	O
are	O
new	O
challenges	O
that	O
we	O
have	O
to	O
solve	O
for	O
web	O
scale	O
indexing	O
and	O
the	O
two	O
main	O
challenges	O
of	O
scalability	O
and	O
efficiency	O
.	O
the	O
index	O
will	O
be	O
so	O
large	O
that	O
it	O
cannot	O
actually	O
fit	O
into	O
any	O
single	O
machine	O
or	O
single	O
disk	O
so	O
we	O
have	O
to	O
store	O
the	O
data	O
on	O
multiple	O
machines	O
.	O
also	O
because	O
the	O
data	O
is	O
so	O
large	O
it	O
s	O
beneficial	O
to	O
process	O
the	O
data	O
in	O
parallel	O
so	O
that	O
we	O
can	O
produce	O
the	O
index	O
quickly	O
.	O
to	O
address	O
these	O
challenges	O
google	O
has	O
made	O
a	O
number	O
of	O
innovations	O
.	O
one	O
is	O
the	O
google	B
file	B
system	I
that	O
s	O
a	O
general	O
distributed	O
file	B
system	I
that	O
can	O
help	O
programmers	O
manage	O
files	O
stored	O
on	O
a	O
cluster	O
of	O
machines	O
.	O
the	O
second	O
is	O
mapreduce	B
.	O
this	O
is	O
a	O
general	O
software	O
framework	O
for	O
supporting	O
parallel	O
computation	O
.	O
hadoop	O
is	O
the	O
most	O
well	O
known	O
open	O
source	O
implementation	O
of	O
mapreduce	B
now	O
used	O
in	O
many	O
applications	O
.	O
so	O
this	O
is	O
the	O
architecture	O
of	O
the	O
google	B
file	B
system	I
.	O
it	O
uses	O
a	O
very	O
simple	O
centralized	O
management	O
mechanism	O
to	O
manage	O
all	O
the	O
specific	O
locations	O
of	O
files	O
.	O
so	O
it	O
maintains	O
the	O
file	O
namespace	O
and	O
look	O
up	O
table	O
to	O
know	O
where	O
exactly	O
each	O
file	O
is	O
stored	O
.	O
the	O
application	O
client	O
would	O
then	O
talk	O
to	O
this	O
gfs	B
master	O
.	O
and	O
that	O
obtains	O
specific	O
locations	O
of	O
the	O
files	O
that	O
they	O
want	O
to	O
process	O
.	O
and	O
once	O
the	O
gfs	B
client	O
obtained	O
the	O
specific	O
information	O
about	O
the	O
files	O
then	O
the	O
application	O
client	O
can	O
talk	O
to	O
the	O
specific	O
servers	O
where	O
the	O
data	O
actually	O
sits	O
directly	O
.	O
so	O
that	O
you	O
can	O
avoid	O
avoid	O
involving	O
other	O
nodes	O
in	O
the	O
network	O
.	O
so	O
when	O
this	O
file	B
system	I
stores	O
the	O
files	O
on	O
machines	O
the	O
system	O
also	O
would	O
create	O
a	O
fixed	O
sizes	O
of	O
chunks	O
.	O
so	O
the	O
data	O
files	O
are	O
separate	O
into	O
many	O
chunks	O
each	O
chunk	O
is	O
megabytes	O
so	O
it	O
s	O
pretty	O
big	O
.	O
and	O
that	O
s	O
appropriate	O
for	O
large	O
data	O
processing	O
.	O
these	O
chunks	O
are	O
replicated	O
to	O
ensure	O
reliability	O
.	O
so	O
this	O
is	O
something	O
that	O
the	O
the	O
programmer	O
doesn	O
t	O
have	O
to	O
worry	O
about	O
and	O
it	O
s	O
all	O
taken	O
care	O
of	O
by	O
this	O
file	B
system	I
.	O
so	O
from	O
the	O
application	O
perspective	O
the	O
programmer	O
would	O
see	O
this	O
as	O
if	O
it	O
s	O
a	O
normal	O
file	O
.	O
the	O
program	O
doesn	O
t	O
have	O
to	O
know	O
where	O
exactly	O
it	O
s	O
stored	O
and	O
can	O
just	O
invoke	O
high	O
level	O
operators	O
to	O
process	O
the	O
file	O
.	O
and	O
another	O
feature	O
is	O
that	O
the	O
data	O
transfer	O
is	O
directly	O
between	O
application	O
and	O
chunk	O
servers	O
so	O
it	O
s	O
it	O
s	O
efficient	O
in	O
this	O
sense	O
.	O
on	O
top	O
of	O
the	O
google	B
file	B
system	I
and	O
google	O
also	O
proposed	O
mapreduce	B
as	O
a	O
general	O
framework	O
for	O
parallel	O
programming	O
.	O
now	O
this	O
is	O
very	O
useful	O
to	O
support	O
a	O
task	O
like	O
building	O
inverted	B
index	I
.	O
and	O
so	O
this	O
framework	O
is	O
hiding	O
a	O
lot	O
of	O
low	O
level	O
features	O
from	O
the	O
programmer	O
.	O
as	O
a	O
result	O
the	O
programmer	O
can	O
make	O
minimum	O
effort	O
to	O
create	O
a	O
application	O
that	O
can	O
be	O
run	O
on	O
a	O
large	O
cluster	O
in	O
parallel	O
.	O
so	O
some	O
of	O
the	O
low	O
level	O
details	O
hidden	O
in	O
the	O
framework	O
including	O
the	O
specific	O
natural	O
communications	O
or	O
load	O
balancing	O
or	O
where	O
the	O
tasks	O
are	O
executed	O
all	O
these	O
details	O
are	O
hidden	O
from	O
the	O
programmer	O
.	O
there	O
is	O
also	O
a	O
nice	O
feature	O
which	O
is	O
the	O
built-in	O
fault	O
tolerance	O
.	O
if	O
one	O
server	O
is	O
broken	O
let	O
s	O
say	O
so	O
it	O
s	O
down	O
and	O
then	O
some	O
tasks	O
may	O
not	O
be	O
finished	O
then	O
the	O
mapreduce	B
mechanism	O
would	O
know	O
that	O
the	O
task	O
has	O
not	O
been	O
done	O
.	O
so	O
it	O
would	O
automatically	O
dispatch	O
the	O
task	O
on	O
other	O
servers	O
that	O
can	O
do	O
the	O
job	O
.	O
and	O
therefore	O
again	O
the	O
programmer	O
doesn	O
t	O
have	O
to	O
worry	O
about	O
that	O
.	O
so	O
here	O
s	O
how	O
mapreduce	B
works	O
.	O
the	O
input	O
data	O
will	O
be	O
separated	O
into	O
a	O
number	O
of	O
key	O
value	O
pairs	O
.	O
now	O
what	O
exactly	O
is	O
in	O
the	O
value	O
will	O
depend	O
on	O
the	O
data	O
.	O
and	O
it	O
s	O
actually	O
a	O
fairly	O
general	O
framework	O
to	O
allow	O
you	O
to	O
just	O
partition	O
the	O
data	O
into	O
different	O
parts	O
.	O
and	O
each	O
part	O
can	O
be	O
then	O
processed	O
in	O
parallel	O
.	O
each	O
key	O
value	O
pair	O
will	O
be	O
then	O
sent	O
to	O
a	O
map	B
function	O
.	O
the	O
programmer	O
will	O
write	O
the	O
map	B
function	O
of	O
course	O
.	O
and	O
then	O
the	O
map	B
function	O
will	O
then	O
process	O
this	O
key	O
value	O
pair	O
and	O
generate	O
the	O
a	O
number	O
of	O
other	O
key	O
value	O
pairs	O
.	O
of	O
course	O
the	O
new	O
key	O
is	O
usually	O
different	O
from	O
the	O
old	O
key	O
that	O
s	O
given	O
to	O
the	O
map	B
as	O
input	O
.	O
and	O
these	O
key	O
value	O
pairs	O
are	O
the	O
output	O
of	O
the	O
map	B
function	O
.	O
and	O
all	O
the	O
outputs	O
of	O
all	O
the	O
map	B
functions	O
will	O
be	O
then	O
collected	O
.	O
and	O
then	O
they	O
will	O
be	O
further	O
sorted	O
based	O
on	O
the	O
key	O
.	O
and	O
the	O
result	O
is	O
that	O
all	O
the	O
values	O
that	O
are	O
associated	O
with	O
the	O
same	O
key	O
will	O
be	O
then	O
grouped	O
together	O
.	O
so	O
now	O
we	O
ve	O
got	O
a	O
pair	O
of	O
a	O
key	O
and	O
a	O
set	O
of	O
values	O
that	O
are	O
attached	O
to	O
this	O
key	O
.	O
so	O
this	O
will	O
then	O
be	O
sent	O
to	O
a	O
reduce	O
function	O
.	O
now	O
of	O
course	O
each	O
reduce	O
function	O
will	O
handle	O
a	O
different	O
each	O
a	O
different	O
key	O
.	O
so	O
we	O
will	O
send	O
this	O
these	O
output	O
values	O
to	O
multiple	O
reduce	O
functions	O
each	O
handling	O
a	O
unique	O
key	O
.	O
a	O
reduce	O
function	O
would	O
then	O
process	O
the	O
input	O
which	O
is	O
a	O
key	O
and	O
a	O
set	O
of	O
values	O
to	O
produce	O
another	O
set	O
of	O
key	O
values	O
as	O
the	O
output	O
.	O
so	O
these	O
output	O
values	O
would	O
be	O
then	O
collected	O
together	O
to	O
form	O
the	O
the	O
final	O
output	O
.	O
right	O
so	O
this	O
is	O
the	O
the	O
general	O
framework	O
of	O
mapreduce	B
.	O
now	O
the	O
programmer	O
only	O
needs	O
to	O
write	O
the	O
the	O
map	B
function	O
and	O
the	O
reduce	O
function	O
.	O
everything	O
else	O
is	O
actually	O
taken	O
care	O
of	O
by	O
the	O
mapreduce	B
framework	I
.	O
so	O
you	O
can	O
see	O
the	O
programmer	O
really	O
only	O
needs	O
to	O
do	O
minimum	O
work	O
.	O
and	O
with	O
such	O
a	O
framework	O
the	O
input	O
data	O
can	O
be	O
partitioned	O
into	O
multiple	O
parts	O
.	O
each	O
is	O
processed	O
in	O
parallel	O
first	O
by	O
map	B
and	O
then	O
in	O
the	O
process	O
after	O
we	O
reach	O
the	O
reduce	O
stage	O
then	O
much	O
more	O
reduce	O
functions	O
can	O
also	O
further	O
process	O
the	O
different	O
keys	O
and	O
their	O
associated	O
values	O
in	O
parallel	O
.	O
so	O
it	O
achieves	O
some	O
it	O
achieves	O
the	O
purpose	O
of	O
parallel	O
processing	O
of	O
a	O
large	O
dataset	O
.	O
so	O
let	O
s	O
take	O
a	O
look	O
at	O
a	O
simple	O
example	O
and	O
that	O
s	O
word	O
counting	O
.	O
the	O
input	O
is	O
is	O
files	O
containing	O
words	O
.	O
and	O
the	O
output	O
that	O
we	O
want	O
to	O
generate	O
is	O
the	O
number	O
of	O
occurrences	O
of	O
each	O
word	O
so	O
it	O
s	O
the	O
word	O
count	O
.	O
right	O
we	O
know	O
this	O
this	O
kind	O
of	O
counting	O
would	O
be	O
useful	O
to	O
for	O
example	O
assess	O
the	O
popularity	O
of	O
a	O
word	O
in	O
a	O
large	O
collection	O
.	O
and	O
this	O
is	O
useful	O
for	O
achieving	O
a	O
factor	O
of	O
idf	B
weighting	O
for	O
search	O
.	O
so	O
how	O
can	O
we	O
solve	O
this	O
problem	O
well	O
one	O
natural	O
thought	O
is	O
that	O
well	O
this	O
task	O
can	O
be	O
done	O
in	O
parallel	O
by	O
simply	O
counting	O
different	O
parts	O
of	O
the	O
file	O
in	O
parallel	O
and	O
then	O
in	O
the	O
end	O
we	O
just	O
combine	O
all	O
the	O
counts	O
.	O
and	O
that	O
s	O
precisely	O
the	O
idea	O
of	O
what	O
we	O
can	O
do	O
with	O
mapreduce	B
.	O
we	O
can	O
parallelize	O
lines	O
in	O
this	O
input	O
file	O
.	O
so	O
more	O
specifically	O
we	O
can	O
assume	O
the	O
input	O
to	O
each	O
map	B
function	O
is	O
a	O
key	O
value	O
pair	O
that	O
represents	O
the	O
line	O
number	O
and	O
the	O
stream	O
on	O
that	O
line	O
.	O
so	O
the	O
first	O
line	O
for	O
example	O
has	O
a	O
key	O
of	O
one	O
.	O
and	O
the	O
value	O
is	O
hello	O
world	O
bye	O
world	O
and	O
just	O
four	O
words	O
on	O
that	O
line	O
.	O
so	O
this	O
key-value	O
pair	O
will	O
be	O
sent	O
to	O
a	O
map	B
function	O
.	O
the	O
map	B
function	O
would	O
then	O
just	O
count	O
the	O
words	O
in	O
this	O
line	O
.	O
and	O
in	O
this	O
case	O
of	O
course	O
there	O
are	O
only	O
four	O
words	O
.	O
each	O
word	O
gets	O
a	O
count	O
of	O
one	O
.	O
and	O
these	O
are	O
the	O
output	O
that	O
you	O
see	O
here	O
on	O
this	O
slide	O
from	O
this	O
map	B
function	O
.	O
so	O
the	O
map	B
function	O
is	O
really	O
very	O
simple	O
.	O
if	O
you	O
look	O
at	O
the	O
what	O
the	O
pseudocode	O
looks	O
like	O
on	O
the	O
right	O
side	O
you	O
see	O
it	O
simply	O
needs	O
to	O
iterate	O
over	O
all	O
the	O
words	O
in	O
this	O
line	O
and	O
then	O
just	O
call	O
a	O
collect	O
function	O
which	O
means	O
it	O
would	O
then	O
send	O
the	O
word	O
and	O
the	O
counter	O
to	O
the	O
collector	O
.	O
the	O
collector	O
would	O
then	O
try	O
to	O
sort	O
all	O
these	O
key	O
value	O
pairs	O
from	O
different	O
map	B
functions	O
.	O
right	O
so	O
the	O
functions	O
are	O
very	O
simple	O
.	O
and	O
the	O
programmer	O
specifies	O
this	O
function	O
as	O
a	O
way	O
to	O
process	O
each	O
part	O
of	O
the	O
data	O
.	O
of	O
course	O
the	O
second	O
line	O
will	O
be	O
handled	O
by	O
a	O
different	O
map	B
function	O
which	O
will	O
produce	O
a	O
similar	O
output	O
.	O
okay	O
now	O
the	O
output	O
from	O
the	O
map	B
functions	O
will	O
be	O
then	O
sent	O
to	O
a	O
collector	O
.	O
and	O
the	O
collector	O
will	O
do	O
the	O
internal	O
grouping	O
or	O
sorting	O
.	O
so	O
at	O
this	O
stage	O
you	O
can	O
see	O
we	O
have	O
collected	O
multiple	O
pairs	O
.	O
each	O
pair	O
is	O
a	O
word	O
and	O
its	O
count	O
in	O
the	O
line	O
.	O
so	O
once	O
we	O
see	O
all	O
these	O
these	O
pairs	O
then	O
we	O
can	O
sort	O
them	O
based	O
on	O
the	O
key	O
which	O
is	O
the	O
word	O
.	O
so	O
we	O
will	O
collect	O
all	O
the	O
counts	O
of	O
a	O
word	O
like	O
bye	O
here	O
together	O
.	O
and	O
similarly	O
we	O
do	O
that	O
for	O
other	O
words	O
.	O
like	O
hadoop	O
hello	O
etc	O
.	O
so	O
each	O
word	O
now	O
is	O
attached	O
to	O
a	O
number	O
of	O
values	O
a	O
number	O
of	O
counts	O
.	O
and	O
these	O
counts	O
represent	O
the	O
occurrences	O
of	O
this	O
word	O
in	O
different	O
lines	O
.	O
so	O
now	O
we	O
have	O
got	O
a	O
new	O
pair	O
of	O
a	O
key	O
and	O
a	O
set	O
of	O
values	O
and	O
this	O
pair	O
will	O
then	O
be	O
fed	O
into	O
a	O
reduce	O
function	O
.	O
so	O
the	O
reduce	O
function	O
now	O
will	O
have	O
to	O
finish	O
the	O
job	O
of	O
counting	O
the	O
total	O
occurrences	O
of	O
this	O
word	O
.	O
now	O
it	O
has	O
already	O
got	O
all	O
these	O
partial	O
counts	O
so	O
all	O
it	O
needs	O
to	O
do	O
is	O
simply	O
to	O
add	O
them	O
up	O
.	O
so	O
the	O
reduce	O
function	O
shown	O
here	O
is	O
very	O
simple	O
as	O
well	O
.	O
you	O
have	O
a	O
counter	O
and	O
then	O
iterate	O
over	O
all	O
the	O
words	O
that	O
you	O
see	O
in	O
this	O
array	O
and	O
then	O
you	O
just	O
accumulate	O
these	O
counts	O
right	O
.	O
and	O
then	O
finally	O
you	O
output	O
the	O
key	O
and	O
and	O
the	O
total	O
count	O
and	O
that	O
s	O
precisely	O
what	O
we	O
want	O
as	O
the	O
output	O
of	O
this	O
whole	O
program	O
.	O
so	O
you	O
can	O
see	O
this	O
is	O
already	O
very	O
similar	O
to	O
building	O
a	O
inverted	B
index	I
and	O
if	O
you	O
think	O
about	O
it	O
the	O
output	O
here	O
is	O
indexed	O
by	O
a	O
word	O
and	O
we	O
have	O
already	O
got	O
a	O
dictionary	O
basically	O
.	O
we	O
have	O
got	O
the	O
count	O
.	O
but	O
what	O
s	O
missing	O
is	O
the	O
document	O
ids	O
and	O
the	O
specific	O
frequency	O
counts	O
of	O
words	O
in	O
those	O
documents	O
.	O
so	O
we	O
can	O
modify	O
this	O
slightly	O
to	O
actually	O
build	O
a	O
inverted	B
index	I
in	O
parallel	O
.	O
so	O
here	O
s	O
one	O
way	O
to	O
do	O
that	O
.	O
so	O
in	O
this	O
case	O
we	O
can	O
assume	O
the	O
input	O
to	O
a	O
map	B
function	O
is	O
a	O
pair	O
of	O
a	O
key	O
which	O
denotes	O
the	O
document	O
id	O
and	O
the	O
value	O
denoting	O
the	O
string	O
for	O
that	O
document	O
.	O
so	O
it	O
s	O
all	O
the	O
words	O
in	O
that	O
document	O
.	O
and	O
so	O
the	O
map	B
function	O
will	O
do	O
something	O
very	O
similar	O
to	O
what	O
we	O
have	O
seen	O
in	O
the	O
water	O
company	O
example	O
.	O
it	O
simply	O
groups	O
all	O
the	O
counts	O
of	O
this	O
word	O
in	O
this	O
document	O
together	O
.	O
and	O
it	O
will	O
then	O
generate	O
a	O
set	O
of	O
key	O
value	O
pairs	O
.	O
each	O
key	O
is	O
a	O
word	O
.	O
and	O
the	O
value	O
is	O
the	O
count	O
of	O
this	O
word	O
in	O
this	O
document	O
plus	O
the	O
document	O
id	O
.	O
now	O
you	O
can	O
easily	O
see	O
why	O
we	O
need	O
to	O
add	O
document	O
id	O
here	O
.	O
of	O
course	O
later	O
in	O
the	O
inverted	B
index	I
we	O
would	O
like	O
to	O
keep	O
this	O
information	O
so	O
the	O
map	B
function	O
should	O
keep	O
track	O
of	O
it	O
.	O
and	O
this	O
can	O
then	O
be	O
sent	O
to	O
the	O
reduce	O
function	O
later	O
.	O
now	O
similarly	O
another	O
document	O
d	O
can	O
be	O
processed	O
in	O
the	O
same	O
way	O
.	O
so	O
in	O
the	O
end	O
again	O
there	O
is	O
a	O
sorting	O
mechanism	O
that	O
would	O
group	O
them	O
together	O
.	O
and	O
then	O
we	O
will	O
have	O
just	O
a	O
key	O
like	O
java	O
associated	O
with	O
all	O
the	O
documents	O
that	O
match	O
this	O
key	O
or	O
all	O
the	O
documents	O
where	O
java	O
occurred	O
and	O
their	O
counts	O
right	O
so	O
the	O
counts	O
of	O
java	O
in	O
those	O
documents	O
.	O
and	O
this	O
will	O
be	O
collected	O
together	O
.	O
and	O
this	O
will	O
be	O
so	O
fed	O
into	O
the	O
reduced	O
function	O
.	O
so	O
now	O
you	O
can	O
see	O
the	O
reduce	O
function	O
has	O
already	O
got	O
input	O
that	O
looks	O
like	O
a	O
inverted	B
index	I
entry	O
right	O
so	O
it	O
s	O
just	O
the	O
word	O
and	O
all	O
the	O
documents	O
that	O
contain	O
the	O
word	O
and	O
the	O
frequency	O
of	O
the	O
word	O
in	O
those	O
documents	O
.	O
so	O
all	O
you	O
need	O
to	O
do	O
is	O
simply	O
to	O
concatenate	O
them	O
into	O
a	O
continuous	O
chunk	O
of	O
data	O
and	O
this	O
can	O
be	O
then	O
retained	O
into	O
a	O
file	B
system	I
.	O
so	O
basically	O
the	O
reduce	O
function	O
is	O
going	O
to	O
do	O
very	O
minimal	O
work	O
.	O
and	O
so	O
this	O
is	O
pseudo-code	O
for	O
inverted	B
index	B
construction	I
.	O
here	O
we	O
see	O
two	O
functions	O
procedure	O
map	B
and	O
procedure	O
reduce	O
.	O
and	O
a	O
programmer	O
would	O
specify	O
these	O
two	O
functions	O
to	O
program	O
on	O
top	O
of	O
mapreduce	B
.	O
and	O
you	O
can	O
see	O
basically	O
they	O
are	O
doing	O
what	O
i	O
just	O
described	O
.	O
in	O
the	O
case	O
of	O
map	B
it	O
s	O
going	O
to	O
count	O
the	O
occurrences	O
of	O
a	O
word	O
using	O
an	O
associative	O
array	O
and	O
will	O
output	O
all	O
the	O
counts	O
together	O
with	O
the	O
document	O
id	O
here	O
.	O
right	O
so	O
this	O
the	O
reduce	O
function	O
on	O
the	O
other	O
hand	O
simply	O
concatenates	O
all	O
the	O
input	O
that	O
it	O
has	O
been	O
given	O
and	O
then	O
put	O
them	O
together	O
as	O
one	O
single	O
entry	O
for	O
this	O
key	O
.	O
so	O
this	O
is	O
a	O
very	O
simple	O
mapreduce	B
function	O
yet	O
it	O
would	O
allow	O
us	O
to	O
construct	O
an	O
inverted	B
index	I
at	O
a	O
very	O
large	O
scale	O
and	O
data	O
can	O
be	O
processed	O
by	O
different	O
machines	O
.	O
the	O
program	O
doesn	O
t	O
have	O
to	O
take	O
care	O
of	O
the	O
details	O
.	O
so	O
this	O
is	O
how	O
we	O
can	O
do	O
parallel	O
index	B
construction	I
for	O
web	B
search	I
.	O
so	O
to	O
summarize	O
web	O
scale	O
indexing	O
requires	O
some	O
new	O
techniques	O
that	O
go	O
beyond	O
the	O
standard	O
traditional	O
indexing	B
techniques	I
.	O
mainly	O
we	O
have	O
to	O
store	O
index	O
on	O
multiple	O
machines	O
and	O
this	O
is	O
usually	O
done	O
by	O
using	O
a	O
file	B
system	I
like	O
google	B
file	B
system	I
a	O
distributed	O
file	B
system	I
.	O
and	O
secondly	O
it	O
requires	O
creating	O
the	O
index	O
in	O
parallel	O
because	O
it	O
s	O
so	O
large	O
it	O
takes	O
a	O
long	O
time	O
to	O
create	O
an	O
index	O
for	O
all	O
the	O
documents	O
.	O
so	O
if	O
we	O
can	O
do	O
it	O
in	O
parallel	O
it	O
would	O
be	O
much	O
faster	O
and	O
this	O
is	O
done	O
by	O
using	O
the	O
mapreduce	B
framework	I
.	O
note	O
that	O
the	O
both	O
the	O
gfs	B
and	O
mapreduce	B
frameworks	O
are	O
very	O
general	O
so	O
they	O
can	O
also	O
support	O
many	O
other	O
applications	O
.	O
so	O
we	O
talked	O
about	O
a	O
page	B
rank	I
as	O
a	O
way	O
to	O
to	O
capture	O
the	O
authorities	O
.	O
now	O
we	O
also	O
looked	O
at	O
the	O
some	O
other	O
examples	O
where	O
a	O
hub	O
might	O
be	O
interesting	O
.	O
so	O
there	O
is	O
another	O
algorithm	O
called	O
the	O
hits	B
and	O
that	O
s	O
going	O
to	O
do	O
compute	O
the	O
scores	O
for	O
us	O
.	O
authorities	O
hubs	O
.	O
intuitions	O
of	O
pages	O
that	O
are	O
widely	O
cited	O
good	O
sorry	O
there	O
is	O
then	O
there	O
is	O
pages	O
that	O
are	O
cited	O
.	O
many	O
other	O
pages	O
are	O
good	O
hubs	O
right	O
but	O
there	O
i	O
think	O
that	O
the	O
.	O
most	O
interesting	O
idea	O
of	O
this	O
algorithm	O
hits	B
is	O
it	O
s	O
going	O
to	O
use	O
a	O
reinforcement	O
mechanism	O
to	O
kind	O
of	O
help	O
improve	O
the	O
scoring	O
for	O
hubs	O
and	O
the	O
authorities	O
.	O
and	O
here	O
so	O
here	O
s	O
the	O
idea	O
it	O
will	O
assume	O
that	O
good	O
authorities	O
are	O
cited	O
by	O
good	O
hubs	O
.	O
that	O
means	O
if	O
you	O
re	O
cited	O
by	O
many	O
pages	O
with	O
good	O
hub	O
scores	O
then	O
that	O
increases	O
your	O
authority	O
score	O
.	O
and	O
similarly	O
good	O
hubs	O
are	O
those	O
that	O
pointed	O
to	O
good	O
authorities	O
.	O
so	O
if	O
you	O
get	O
you	O
point	O
it	O
to	O
a	O
lot	O
of	O
good	O
authority	O
pages	O
then	O
your	O
hub	O
score	O
would	O
be	O
increased	O
.	O
so	O
you	O
then	O
you	O
would	O
have	O
iterative	O
reinforce	O
each	O
other	O
because	O
you	O
can	O
point	O
it	O
to	O
some	O
good	O
hubs	O
.	O
sorry	O
you	O
can	O
point	O
it	O
to	O
some	O
good	O
authorities	O
.	O
to	O
get	O
a	O
good	O
hub	O
score	O
.	O
whereas	O
those	O
authority	O
scores	O
would	O
be	O
also	O
improved	O
because	O
they	O
are	O
pointed	O
to	O
by	O
a	O
good	O
hub	O
.	O
and	O
this	O
hub	O
is	O
also	O
general	O
it	O
can	O
have	O
many	O
applications	O
in	O
graph	O
and	O
network	O
analysis	O
.	O
so	O
just	O
briefly	O
here	O
s	O
how	O
it	O
works	O
.	O
we	O
first	O
also	O
construct	O
the	O
matrix	O
but	O
this	O
time	O
we	O
re	O
going	O
to	O
construct	O
the	O
adjacency	O
matrix	O
.	O
we	O
re	O
not	O
going	O
to	O
normalize	O
the	O
values	O
so	O
if	O
there	O
s	O
a	O
link	O
there	O
s	O
a	O
y	O
.	O
if	O
there	O
s	O
no	O
link	O
that	O
s	O
zero	O
.	O
right	O
again	O
it	O
s	O
the	O
same	O
graph	O
and	O
then	O
we	O
re	O
going	O
to	O
define	O
the	O
top	O
score	O
of	O
page	O
as	O
a	O
sum	O
of	O
the	O
authority	O
scores	O
of	O
all	O
the	O
pages	O
that	O
it	O
appoints	O
to	O
.	O
so	O
whether	O
you	O
are	O
hub	O
that	O
really	O
depends	O
on	O
whether	O
you	O
are	O
pointing	O
to	O
a	O
lot	O
of	O
good	O
authority	O
pages	O
.	O
that	O
s	O
what	O
it	O
says	O
in	O
the	O
first	O
equation	O
.	O
your	O
second	O
equation	O
will	O
define	O
the	O
authority	O
score	O
of	O
a	O
page	O
as	O
a	O
sum	O
of	O
the	O
hub	O
scores	O
of	O
all	O
those	O
pages	O
.	O
that	O
they	O
point	O
to	O
so	O
whether	O
you	O
are	O
a	O
good	O
authority	O
would	O
depend	O
on	O
whether	O
those	O
pages	O
that	O
are	O
pointing	O
to	O
you	O
are	O
good	O
hubs	O
.	O
so	O
you	O
can	O
see	O
this	O
a	O
forms	O
a	O
iterative	O
reinforcement	O
mechanism	O
.	O
now	O
these	O
two	O
equations	O
can	O
be	O
also	O
written	O
.	O
in	O
the	O
matrix	O
fo-	O
format	O
.	O
right	O
so	O
what	O
we	O
get	O
here	O
is	O
then	O
the	O
hub	O
vector	O
is	O
equal	O
to	O
the	O
product	O
of	O
the	O
adjacency	O
matrix	O
.	O
and	O
the	O
authority	O
vector	O
.	O
and	O
this	O
is	O
basically	O
the	O
first	O
equation	O
.	O
right	O
.	O
and	O
similarly	O
the	O
second	O
equation	O
can	O
be	O
returned	O
as	O
the	O
authority	O
vector	O
is	O
equal	O
to	O
the	O
product	O
of	O
a	O
transpose	O
multiplied	O
by	O
the	O
hub	O
vector	O
.	O
and	O
these	O
are	O
just	O
different	O
ways	O
of	O
expressing	O
these	O
equations	O
.	O
but	O
what	O
s	O
interesting	O
is	O
that	O
if	O
you	O
look	O
at	O
to	O
the	O
matrix	O
form	O
.	O
you	O
can	O
also	O
plug-in	O
the	O
authority	O
equation	O
into	O
the	O
first	O
one	O
.	O
so	O
if	O
you	O
do	O
that	O
you	O
can	O
actually	O
make	O
it	O
limited	O
to	O
the	O
authority	O
vector	O
completely	O
and	O
you	O
get	O
the	O
equation	O
of	O
only	O
hub	O
scores	O
.	O
right	O
the	O
hub	O
score	O
vector	O
is	O
equal	O
to	O
a	O
multiplied	O
by	O
a	O
transpose	O
.	O
multiplied	O
by	O
the	O
hub	O
score	O
vector	O
again	O
.	O
and	O
similarly	O
we	O
can	O
do	O
a	O
transformation	O
to	O
have	O
equation	O
for	O
just	O
the	O
authorities	O
scores	O
.	O
so	O
although	O
we	O
framed	O
the	O
problem	O
as	O
computing	O
hubs	O
authorities	O
we	O
can	O
actually	O
eliminate	O
the	O
one	O
of	O
them	O
to	O
obtain	O
equation	O
just	O
for	O
one	O
of	O
them	O
.	O
now	O
the	O
difference	O
between	O
this	O
and	O
page	O
is	O
that	O
now	O
the	O
matrix	O
is	O
actually	O
a	O
multiplication	O
of	O
the	O
mer-	O
adjacency	O
matrix	O
and	O
its	O
transpose	O
.	O
so	O
this	O
is	O
different	O
from	O
page	B
rank	I
.	O
right	O
but	O
mathematically	O
then	O
we	O
would	O
be	O
computing	O
the	O
same	O
problem	O
.	O
so	O
in	O
ha	O
in	O
hits	B
we	O
re	O
keeping	O
would	O
initialize	O
the	O
values	O
that	O
state	O
one	O
for	O
all	O
these	O
values	O
.	O
and	O
then	O
with	O
the	O
algorithm	O
will	O
apply	O
these	O
these	O
equations	O
essentially	O
and	O
this	O
is	O
equivalent	O
if	O
you	O
multiply	O
that	O
.	O
by	O
by	O
the	O
matrix	O
.	O
a	O
and	O
a	O
transpose	O
.	O
right	O
.	O
and	O
so	O
the	O
arrows	O
of	O
these	O
are	O
exactly	O
the	O
same	O
in	O
the	O
debate	O
rank	O
.	O
but	O
here	O
because	O
the	O
adjacency	O
matrix	O
is	O
not	O
normalized	O
so	O
what	O
we	O
have	O
to	O
do	O
is	O
to	O
what	O
we	O
have	O
to	O
do	O
is	O
after	O
each	O
iteration	O
we	O
have	O
to	O
do	O
normalize	O
.	O
and	O
this	O
would	O
allow	O
us	O
to	O
control	O
the	O
grooves	O
of	O
value	O
.	O
otherwise	O
they	O
would	O
grew	O
larger	O
and	O
larger	O
.	O
and	O
if	O
we	O
do	O
that	O
and	O
then	O
we	O
will	O
basically	O
get	O
a	O
hits	B
.	O
i	O
was	O
in	O
the	O
computer	O
the	O
hub	O
scores	O
and	O
also	O
the	O
scores	O
for	O
all	O
of	O
the	O
pages	O
.	O
and	O
these	O
scores	O
can	O
then	O
be	O
used	O
in	O
ranging	O
to	O
start	O
the	O
pagerank	B
scores	O
.	O
so	O
to	O
summarize	O
in	O
this	O
lecture	O
we	O
have	O
seen	O
that	O
link	O
information	O
is	O
very	O
useful	O
.	O
in	O
particular	O
the	O
anchor	B
text	I
base	O
is	O
very	O
useful	O
.	O
to	O
increase	O
the	O
the	O
text	O
representation	O
of	O
a	O
page	O
.	O
and	O
we	O
also	O
talk	O
about	O
the	O
pagerank	B
and	O
hits	B
algorithm	I
as	O
two	O
major	O
link	B
analysis	I
algorithms	O
.	O
both	O
can	O
generate	O
scores	O
for	O
.	O
what	O
pages	O
that	O
can	O
be	O
used	O
for	O
the	O
the	O
ranking	B
function	I
.	O
those	O
that	O
pagerank	B
and	O
the	O
hits	B
also	O
very	O
general	O
algorithms	O
so	O
they	O
have	O
many	O
applications	O
in	O
analyzing	O
other	O
graphs	O
or	O
networks	O
.	O
	O
.	O
there	O
are	O
many	O
more	O
advanced	O
learning	B
algorithms	I
than	O
the	O
regression	O
based	O
reproaches	O
.	O
and	O
they	O
generally	O
account	O
to	O
theoretically	O
optimize	O
or	O
retrieval	B
method	I
.	O
like	O
map	B
or	O
ndcg	B
.	O
note	O
that	O
the	O
optimization	O
objecting	O
function	O
that	O
we	O
have	O
seen	O
on	O
the	O
previous	O
slide	O
is	O
not	O
directly	O
related	O
to	O
retrieval	O
measure	O
.	O
right	O
by	O
maximizing	O
the	O
prediction	O
of	O
one	O
or	O
zero	O
.	O
or	O
we	O
don	O
t	O
necessarily	O
optimize	O
the	O
ranking	O
of	O
those	O
documents	O
.	O
one	O
can	O
imagine	O
that	O
why	O
our	O
prediction	O
may	O
not	O
be	O
too	O
bad	O
and	O
let	O
s	O
say	O
both	O
are	O
around	O
	O
.	O
	O
.	O
so	O
it	O
s	O
kind	O
of	O
in	O
the	O
middle	O
of	O
zero	O
and	O
one	O
for	O
the	O
two	O
documents	O
but	O
the	O
ranking	O
can	O
be	O
wrong	O
.	O
so	O
we	O
might	O
have	O
the	O
a	O
larger	O
value	O
for	O
.	O
d	O
and	O
then	O
e	O
	O
.	O
so	O
that	O
won	O
t	O
be	O
good	O
from	O
retrieval	O
perspective	O
even	O
though	O
by	O
likelihood	O
function	O
it	O
s	O
not	O
bad	O
.	O
in	O
contrast	O
we	O
might	O
have	O
another	O
case	O
where	O
we	O
predicted	O
values	O
.	O
or	O
around	O
	O
.	O
let	O
s	O
say	O
and	O
by	O
the	O
objective	O
function	O
the	O
error	O
will	O
be	O
larger	O
but	O
if	O
we	O
can	O
get	O
the	O
order	O
of	O
the	O
two	O
documents	O
correct	O
that	O
s	O
actually	O
a	O
better	O
result	O
.	O
so	O
these	O
new	O
more	O
advanced	O
approaches	O
will	O
try	O
to	O
correct	O
that	O
problem	O
.	O
of	O
course	O
then	O
the	O
challenge	O
is	O
that	O
.	O
that	O
the	O
optimization	O
problem	O
will	O
be	O
harder	O
to	O
solve	O
.	O
and	O
then	O
researchers	O
have	O
proposed	O
many	O
solutions	O
to	O
the	O
problem	O
.	O
and	O
you	O
can	O
read	O
more	O
of	O
the	O
references	O
at	O
the	O
end	O
.	O
know	O
more	O
about	O
the	O
these	O
approaches	O
.	O
now	O
these	O
learning	O
to	O
random	O
approaches	O
.	O
are	O
actually	O
general	O
so	O
they	O
can	O
also	O
be	O
applied	O
to	O
many	O
other	O
ranking	O
problems	O
not	O
just	O
retrieval	O
problem	O
.	O
so	O
here	O
i	O
list	O
some	O
for	O
example	O
recommender	B
systems	I
computational	O
adv	O
advertising	O
or	O
summarization	O
and	O
there	O
are	O
many	O
others	O
that	O
you	O
can	O
probably	O
encounter	O
in	O
your	O
applications	O
.	O
to	O
summarize	O
this	O
lecture	O
we	O
have	O
talked	O
about	O
using	O
machine	B
learning	I
to	O
combine	O
much	O
more	O
features	O
to	O
incorporate	O
a	O
ranking	O
without	O
.	O
actually	O
the	O
use	O
of	O
machine	B
learning	I
in	O
information	B
retrieval	I
has	O
started	O
since	O
many	O
decades	O
ago	O
.	O
so	O
for	O
example	O
on	O
the	O
rocchio	B
feedback	O
approach	O
that	O
we	O
talked	O
about	O
earlier	O
was	O
a	O
machine	B
learning	I
approach	O
applied	O
to	O
to	O
learn	O
this	O
feedback	O
but	O
the	O
most	O
reasonable	O
use	O
of	O
machine	B
learning	I
has	O
been	O
driven	O
by	O
some	O
changes	O
.	O
in	O
the	O
environment	O
of	O
applications	O
of	O
retrieval	B
systems	I
.	O
and	O
first	O
it	O
s	O
mostly	O
driven	O
by	O
the	O
availability	O
of	O
a	O
lot	O
of	O
training	O
data	O
in	O
the	O
form	O
of	O
clicks	O
rules	O
.	O
such	O
data	O
weren	O
t	O
available	O
before	O
.	O
so	O
the	O
data	O
can	O
provide	O
a	O
lot	O
of	O
useful	O
knowledge	O
about	O
relevance	O
and	O
machine	B
learning	I
methods	O
can	O
be	O
applied	O
to	O
leverage	O
this	O
.	O
secondly	O
it	O
s	O
also	O
due	O
by	O
the	O
need	O
of	O
combining	O
them	O
.	O
in	O
the	O
features	O
.	O
and	O
this	O
is	O
not	O
only	O
just	O
because	O
there	O
are	O
more	O
features	O
available	O
on	O
the	O
web	O
that	O
can	O
be	O
naturally	O
re-used	O
with	O
improved	O
scoring	O
.	O
it	O
s	O
also	O
because	O
by	O
combining	O
them	O
we	O
can	O
improve	O
the	O
robustness	O
of	O
ranking	O
.	O
so	O
this	O
is	O
designed	O
for	O
combating	O
spams	O
.	O
modern	O
search	B
engines	I
all	O
use	O
some	O
kind	O
of	O
machine	B
learning	I
techniques	O
to	O
combine	O
many	O
features	O
to	O
optimize	O
ranking	O
and	O
this	O
is	O
a	O
major	O
feature	O
of	O
these	O
current	O
engines	O
such	O
as	O
google	O
bing	O
.	O
the	O
topic	O
of	O
learning	B
to	I
rank	I
is	O
still	O
active	O
research	O
.	O
topic	O
in	O
the	O
community	O
and	O
so	O
you	O
can	O
expect	O
to	O
see	O
new	O
results	O
being	O
developed	O
in	O
the	O
next	O
few	O
years	O
.	O
perhaps	O
.	O
here	O
are	O
some	O
additional	O
readings	O
that	O
can	O
give	O
you	O
more	O
information	O
about	O
.	O
about	O
how	O
learning	B
to	I
rank	I
books	O
and	O
also	O
some	O
advanced	O
methods	O
.	O
	O
.	O
this	O
lecture	O
is	O
about	O
the	O
future	O
of	O
web	B
search	I
.	O
in	O
this	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
some	O
possible	O
future	O
trends	O
of	O
web	B
search	I
and	O
intelligent	B
information	B
retrieval	B
systems	I
in	O
general	O
.	O
in	O
order	O
to	O
further	O
improve	O
the	O
accuracy	O
of	O
a	O
search	B
engine	I
it	O
s	O
important	O
that	O
to	O
consider	O
special	O
cases	O
of	O
information	O
need	O
.	O
so	O
one	O
particular	O
trend	O
could	O
be	O
to	O
have	O
more	O
and	O
more	O
specialized	O
than	O
customized	O
search	B
engines	I
and	O
they	O
can	O
be	O
called	O
vertical	O
search	B
engines	I
.	O
these	O
vertical	O
search	B
engines	I
can	O
be	O
expected	O
to	O
be	O
more	O
effective	O
than	O
the	O
current	O
general	O
search	B
engines	I
because	O
they	O
could	O
assume	O
that	O
users	O
are	O
a	O
special	O
group	O
of	O
users	O
that	O
might	O
have	O
a	O
common	O
information	O
need	O
and	O
then	O
the	O
search	B
engine	I
can	O
be	O
customized	O
with	O
this	O
ser	O
so	O
such	O
users	O
.	O
and	O
because	O
of	O
the	O
customization	O
it	O
s	O
also	O
possible	O
to	O
do	O
personalization	O
.	O
so	O
the	O
search	O
can	O
be	O
personalized	O
because	O
we	O
have	O
a	O
better	O
understanding	O
of	O
the	O
users	O
.	O
because	O
of	O
the	O
restrictions	O
with	O
domain	O
we	O
also	O
have	O
some	O
advantages	O
in	O
handling	O
the	O
documents	O
because	O
we	O
can	O
have	O
better	O
understanding	O
of	O
documents	O
.	O
for	O
example	O
particular	O
words	O
may	O
not	O
be	O
ambiguous	O
in	O
such	O
a	O
domain	O
.	O
so	O
we	O
can	O
bypass	O
the	O
problem	O
of	O
ambiguity	O
.	O
another	O
trend	O
we	O
can	O
expect	O
to	O
see	O
is	O
the	O
search	B
engine	I
will	O
be	O
able	O
to	O
learn	O
over	O
time	O
.	O
it	O
s	O
like	O
a	O
lifetime	O
learning	O
or	O
lifelong	O
learning	O
and	O
this	O
is	O
of	O
course	O
very	O
attractive	O
because	O
that	O
means	O
the	O
search	B
engine	I
will	O
self-improve	O
itself	O
.	O
as	O
more	O
people	O
are	O
using	O
it	O
the	O
search	B
engine	I
will	O
become	O
better	O
and	O
better	O
and	O
this	O
is	O
already	O
happening	O
because	O
the	O
search	B
engines	I
can	O
learn	O
from	O
the	O
of	O
feedback	O
.	O
more	O
users	O
use	O
it	O
and	O
the	O
quality	O
of	O
the	O
search	B
engine	I
allows	O
for	O
the	O
popular	O
queries	O
that	O
are	O
typed	O
in	O
by	O
many	O
users	O
allow	O
it	O
to	O
become	O
better	O
so	O
this	O
is	O
sort	O
of	O
another	O
feature	O
that	O
we	O
will	O
see	O
.	O
the	O
third	O
trend	O
might	O
be	O
to	O
the	O
integration	O
of	O
bottles	O
of	O
information	B
access	I
.	O
so	O
search	O
navigation	O
and	O
recommendation	O
or	O
filtering	O
might	O
be	O
combined	O
to	O
form	O
a	O
full-fledged	O
information	O
management	O
system	O
.	O
and	O
in	O
the	O
beginning	O
of	O
this	O
course	O
we	O
talked	O
about	O
push	B
versus	O
pull	O
.	O
these	O
are	O
different	O
modes	O
of	O
information	B
access	I
but	O
these	O
modes	O
can	O
be	O
combined	O
.	O
and	O
similarly	O
in	O
the	O
pull	O
mode	O
querying	B
and	O
the	O
browsing	O
could	O
also	O
be	O
combined	O
.	O
and	O
in	O
fact	O
we	O
re	O
doing	O
that	O
basically	O
today	O
is	O
the	O
search	O
endings	O
.	O
we	O
are	O
querying	B
sometimes	O
browsing	O
clicking	O
on	O
links	O
.	O
sometimes	O
we	O
ve	O
got	O
some	O
information	O
recommended	O
.	O
although	O
most	O
of	O
the	O
cases	O
the	O
information	O
recommended	O
is	O
because	O
of	O
advertising	O
.	O
but	O
in	O
the	O
future	O
you	O
can	O
imagine	O
seamlessly	O
integrate	O
the	O
system	O
with	O
multi-mode	O
for	O
information	B
access	I
and	O
that	O
would	O
be	O
convenient	O
for	O
people	O
.	O
another	O
trend	O
is	O
that	O
we	O
might	O
see	O
systems	O
that	O
try	O
to	O
go	O
beyond	O
the	O
searches	O
to	O
support	O
the	O
user	O
tasks	O
.	O
after	O
all	O
the	O
reason	O
why	O
people	O
want	O
to	O
search	O
is	O
to	O
solve	O
a	O
problem	O
or	O
to	O
make	O
a	O
decision	O
or	O
perform	O
a	O
task	O
.	O
for	O
example	O
consumers	O
might	O
search	O
for	O
opinions	O
about	O
products	O
in	O
order	O
to	O
purchase	O
a	O
product	O
choose	O
a	O
good	O
product	O
by	O
so	O
in	O
this	O
case	O
it	O
would	O
be	O
beneficial	O
to	O
support	O
the	O
whole	O
workflow	O
of	O
purchasing	O
a	O
product	O
or	O
choosing	O
a	O
product	O
.	O
in	O
this	O
era	O
after	O
the	O
common	O
search	B
engines	I
already	O
provide	O
a	O
good	O
support	O
.	O
for	O
example	O
you	O
can	O
sometimes	O
look	O
at	O
the	O
reviews	O
and	O
then	O
if	O
you	O
want	O
to	O
buy	O
it	O
you	O
can	O
just	O
click	O
on	O
the	O
button	O
to	O
go	O
the	O
shopping	O
site	O
and	O
directly	O
get	O
it	O
done	O
.	O
but	O
it	O
does	O
not	O
provide	O
a	O
a	O
good	O
task	O
support	O
for	O
many	O
other	O
tasks	O
.	O
for	O
example	O
for	O
researchers	O
you	O
might	O
want	O
to	O
find	O
the	O
realm	O
in	O
the	O
literature	O
or	O
site	O
of	O
the	O
literature	O
.	O
and	O
then	O
there	O
s	O
no	O
not	O
much	O
support	O
for	O
finishing	O
a	O
task	O
such	O
as	O
writing	O
a	O
paper	O
.	O
so	O
in	O
general	O
i	O
think	O
there	O
are	O
many	O
opportunities	O
in	O
the	O
wait	O
.	O
so	O
in	O
the	O
following	O
few	O
slides	O
i	O
ll	O
be	O
talking	O
a	O
little	O
bit	O
more	O
about	O
some	O
specific	O
ideas	O
or	O
thoughts	O
that	O
hopefully	O
can	O
help	O
you	O
in	O
imagining	O
new	O
application	O
possibilities	O
.	O
some	O
of	O
them	O
might	O
be	O
already	O
relevant	O
to	O
what	O
you	O
are	O
currently	O
working	O
on	O
.	O
in	O
general	O
we	O
can	O
think	O
about	O
any	O
intelligent	O
system	O
especially	O
intelligent	O
information	O
system	O
as	O
we	O
specified	O
by	O
these	O
these	O
three	O
nodes	O
.	O
and	O
so	O
if	O
we	O
connect	O
these	O
three	O
into	O
a	O
triangle	O
then	O
we	O
ll	O
able	O
to	O
specify	O
an	O
information	O
system	O
.	O
and	O
i	O
call	O
this	O
data-user-service	O
triangle	O
.	O
so	O
basically	O
the	O
three	O
questions	O
you	O
ask	O
would	O
be	O
who	O
are	O
you	O
serving	O
and	O
what	O
kind	O
of	O
data	O
are	O
you	O
are	O
managing	O
and	O
what	O
kind	O
of	O
service	O
you	O
provide	O
.	O
right	O
there	O
this	O
would	O
help	O
us	O
basically	O
specify	O
in	O
your	O
system	O
.	O
and	O
there	O
are	O
many	O
different	O
ways	O
to	O
connect	O
them	O
depending	O
on	O
how	O
you	O
connect	O
them	O
you	O
will	O
have	O
a	O
different	O
kind	O
of	O
systems	O
.	O
so	O
let	O
me	O
give	O
you	O
some	O
examples	O
.	O
on	O
the	O
top	O
you	O
can	O
see	O
different	O
kinds	O
of	O
users	O
.	O
on	O
the	O
left	O
side	O
you	O
can	O
see	O
different	O
types	O
of	O
data	O
or	O
information	O
and	O
on	O
the	O
bottom	O
you	O
can	O
see	O
different	O
service	O
functions	O
.	O
now	O
imagine	O
you	O
can	O
connect	O
all	O
these	O
in	O
different	O
ways	O
.	O
so	O
for	O
example	O
you	O
can	O
connect	O
everyone	O
with	O
web	O
pages	O
and	O
the	O
support	O
search	O
and	O
browsing	O
what	O
do	O
you	O
get	O
well	O
that	O
s	O
web	B
search	I
right	O
what	O
if	O
we	O
connect	O
uiuc	O
employees	O
with	O
organization	O
documents	O
or	O
enterprise	O
documents	O
to	O
support	O
the	O
search	O
and	O
browsing	O
but	O
that	O
s	O
enterprise	O
search	O
.	O
if	O
you	O
connect	O
the	O
scientist	O
with	O
literature	O
information	O
to	O
provide	O
all	O
kinds	O
of	O
service	O
including	O
search	O
browsing	O
or	O
alert	O
of	O
new	O
random	O
documents	O
or	O
mining	O
analyzing	O
research	O
trends	O
or	O
provide	O
the	O
task	O
with	O
support	O
or	O
decision	O
support	O
.	O
for	O
example	O
we	O
might	O
be	O
might	O
be	O
able	O
to	O
provide	O
a	O
support	O
for	O
automatically	O
generating	O
related	O
work	O
section	O
for	O
a	O
research	O
paper	O
and	O
this	O
would	O
be	O
closer	O
to	O
task	O
support	O
.	O
right	O
so	O
then	O
we	O
can	O
imagine	O
this	O
would	O
be	O
a	O
literature	O
assistant	O
.	O
if	O
we	O
connect	O
the	O
online	O
shoppers	O
with	O
blog	O
articles	O
or	O
product	O
reviews	O
then	O
we	O
can	O
help	O
these	O
people	O
to	O
improve	O
shopping	O
experience	O
.	O
so	O
we	O
can	O
provide	O
for	O
example	O
data	O
mining	O
capabilities	O
to	O
analyze	O
the	O
reviews	O
to	O
compare	O
products	O
compare	O
sentiment	O
of	O
products	O
and	O
to	O
provide	O
task	O
support	O
or	O
decision	O
support	O
to	O
have	O
them	O
choose	O
what	O
product	O
to	O
buy	O
.	O
or	O
we	O
can	O
connect	O
customer	O
service	O
people	O
with	O
emails	O
from	O
the	O
customers	O
and	O
and	O
we	O
can	O
imagine	O
a	O
system	O
that	O
can	O
provide	O
a	O
analysis	O
of	O
these	O
emails	O
to	O
find	O
that	O
the	O
major	O
complaints	O
of	O
the	O
customers	O
.	O
we	O
can	O
imagine	O
a	O
system	O
we	O
could	O
provide	O
task	O
support	O
by	O
automatically	O
generating	O
a	O
response	O
to	O
a	O
customer	O
email	O
.	O
maybe	O
intelligently	O
attach	O
also	O
a	O
promotion	O
message	O
if	O
appropriate	O
if	O
they	O
detect	O
that	O
that	O
s	O
a	O
positive	O
message	O
not	O
a	O
complaint	O
and	O
then	O
you	O
might	O
take	O
this	O
opportunity	O
to	O
attach	O
some	O
promotion	O
information	O
.	O
whereas	O
if	O
it	O
s	O
a	O
complaint	O
then	O
you	O
might	O
be	O
able	O
to	O
automatically	O
generate	O
some	O
generic	O
response	O
first	O
and	O
tell	O
the	O
customer	O
that	O
he	O
or	O
she	O
can	O
expect	O
a	O
detailed	O
response	O
later	O
etc	O
.	O
all	O
of	O
these	O
are	O
trying	O
to	O
help	O
people	O
to	O
improve	O
the	O
productivity	O
.	O
so	O
this	O
shows	O
that	O
the	O
opportunities	O
are	O
really	O
a	O
lot	O
.	O
it	O
s	O
just	O
only	O
restricted	O
by	O
our	O
imagination	O
.	O
so	O
this	O
picture	O
shows	O
the	O
trend	O
of	O
the	O
technology	O
and	O
also	O
it	O
characterizes	O
the	O
intelligent	O
information	O
system	O
in	O
three	O
angles	O
.	O
you	O
can	O
see	O
in	O
the	O
center	O
there	O
s	O
a	O
triangle	O
that	O
connects	O
keyword	B
queries	I
to	O
search	O
a	O
bag	B
of	I
words	I
representation	I
.	O
that	O
means	O
the	O
current	O
search	B
engines	I
basically	O
provides	O
search	O
support	O
to	O
users	O
and	O
mostly	O
model	O
users	O
based	O
on	O
keyword	B
queries	I
and	O
sees	O
the	O
data	O
through	O
bag	B
of	I
words	I
representation	I
.	O
so	O
it	O
s	O
a	O
very	O
simple	O
approximation	O
of	O
the	O
actual	O
information	O
in	O
the	O
documents	O
.	O
but	O
that	O
s	O
what	O
the	O
current	O
system	O
does	O
.	O
it	O
connects	O
these	O
three	O
nodes	O
in	O
such	O
a	O
simple	O
way	O
or	O
it	O
only	O
provides	O
a	O
basic	O
search	O
function	O
and	O
doesn	O
t	O
really	O
understand	O
the	O
user	O
and	O
it	O
doesn	O
t	O
really	O
understand	O
that	O
much	O
information	O
in	O
the	O
documents	O
.	O
now	O
i	O
showed	O
some	O
trends	O
to	O
push	B
each	O
node	O
toward	O
a	O
more	O
advanced	O
function	O
.	O
so	O
think	O
about	O
the	O
user	O
node	O
here	O
right	O
so	O
we	O
can	O
go	O
beyond	O
the	O
keyword	B
queries	I
look	O
at	O
the	O
user	O
search	O
history	O
and	O
then	O
further	O
model	O
the	O
user	O
completely	O
to	O
understand	O
the	O
the	O
user	O
s	O
task	O
environment	O
task	O
need	O
context	O
or	O
other	O
information	O
.	O
okay	O
so	O
this	O
is	O
pushing	O
for	O
personalization	O
and	O
complete	O
user	O
model	O
.	O
and	O
this	O
is	O
a	O
major	O
direction	O
in	O
research	O
in	O
in	O
order	O
to	O
build	O
intelligent	O
information	O
systems	O
.	O
on	O
the	O
document	O
side	O
we	O
can	O
also	O
see	O
we	O
can	O
go	O
beyond	O
bag	B
of	I
words	I
implementation	O
to	O
have	O
entity	O
relation	O
representation	O
.	O
this	O
means	O
we	O
ll	O
recognize	O
people	O
s	O
names	O
their	O
relations	O
locations	O
etc	O
.	O
and	O
this	O
is	O
already	O
feasible	O
with	O
today	O
s	O
natural	O
processing	O
technique	O
.	O
and	O
google	O
is	O
the	O
reason	O
the	O
initiative	O
on	O
the	O
knowledge	O
graph	O
.	O
if	O
you	O
haven	O
t	O
heard	O
of	O
it	O
it	O
is	O
a	O
good	O
step	O
toward	O
this	O
direction	O
.	O
and	O
once	O
we	O
can	O
get	O
to	O
that	O
level	O
without	O
initiating	O
robust	O
manner	O
at	O
larger	O
scale	O
it	O
can	O
enable	O
the	O
search	B
engine	I
to	O
provide	O
a	O
much	O
better	O
service	O
.	O
in	O
the	O
future	O
we	O
would	O
like	O
to	O
have	O
knowledge	O
representation	O
where	O
we	O
can	O
add	O
perhaps	O
inference	B
rules	I
and	O
then	O
the	O
search	B
engine	I
would	O
become	O
more	O
intelligent	O
.	O
so	O
this	O
calls	O
for	O
large-scale	O
semantic	B
analysis	I
and	O
perhaps	O
this	O
is	O
more	O
feasible	O
for	O
vertical	O
search	B
engines	I
.	O
it	O
s	O
easier	O
to	O
make	O
progress	O
in	O
the	O
particular	O
domain	O
.	O
now	O
on	O
the	O
service	O
side	O
we	O
see	O
we	O
need	O
to	O
go	O
beyond	O
the	O
search	O
of	O
support	O
information	B
access	I
in	O
general	O
.	O
so	O
search	O
is	O
only	O
one	O
way	O
to	O
get	O
access	O
to	O
information	O
as	O
well	O
recommender	B
systems	I
and	O
push	B
and	I
pull	I
so	O
different	O
ways	O
to	O
get	O
access	O
to	O
random	O
information	O
.	O
but	O
going	O
beyond	O
access	O
we	O
also	O
need	O
to	O
help	O
people	O
digest	O
the	O
information	O
once	O
the	O
information	O
is	O
found	O
and	O
this	O
step	O
has	O
to	O
do	O
with	O
analysis	O
of	O
information	O
or	O
data	O
mining	O
.	O
we	O
have	O
to	O
find	O
patterns	O
or	O
convert	O
the	O
text	O
information	O
into	O
real	O
knowledge	O
that	O
can	O
be	O
used	O
in	O
application	O
or	O
actionable	O
knowledge	O
that	O
can	O
be	O
used	O
for	O
decision	O
making	O
.	O
and	O
furthermore	O
the	O
knowledge	O
will	O
be	O
used	O
to	O
help	O
a	O
user	O
to	O
improve	O
productivity	O
in	O
finishing	O
a	O
task	O
for	O
example	O
a	O
decision-making	O
task	O
.	O
right	O
so	O
this	O
is	O
a	O
trend	O
.	O
and	O
and	O
and	O
so	O
basically	O
in	O
this	O
dimension	O
we	O
anticipate	O
in	O
the	O
future	O
intelligent	O
information	O
systems	O
will	O
provide	O
intelligent	O
and	O
interactive	O
task	O
support	O
.	O
now	O
i	O
should	O
also	O
emphasize	O
interactive	O
here	O
because	O
it	O
s	O
important	O
to	O
optimize	O
the	O
combined	O
intelligence	O
of	O
the	O
users	O
and	O
the	O
system	O
.	O
so	O
we	O
we	O
can	O
get	O
some	O
help	O
from	O
users	O
in	O
some	O
natural	O
way	O
.	O
and	O
we	O
don	O
t	O
have	O
to	O
assume	O
the	O
system	O
has	O
to	O
do	O
everything	O
when	O
the	O
human	O
user	O
and	O
the	O
machine	O
can	O
collaborate	O
in	O
an	O
intelligent	O
way	O
an	O
efficient	O
way	O
then	O
the	O
combined	O
intelligence	O
will	O
be	O
high	O
and	O
in	O
general	O
we	O
can	O
minimize	O
the	O
user	O
s	O
overall	O
effort	O
in	O
solving	O
problem	O
.	O
so	O
this	O
is	O
the	O
big	O
picture	O
of	O
future	O
intelligent	O
information	O
systems	O
and	O
this	O
hopefully	O
can	O
provide	O
us	O
with	O
some	O
insights	O
about	O
how	O
to	O
make	O
further	O
innovations	O
on	O
top	O
of	O
what	O
we	O
handled	O
today	O
.	O
	O
.	O
there	O
are	O
some	O
interesting	O
challenges	O
in	O
threshold	O
.	O
would	O
have	O
known	O
in	O
the	O
filtering	O
problem	O
.	O
so	O
here	O
i	O
show	O
the	O
sort	O
of	O
the	O
data	O
that	O
you	O
can	O
collect	O
in	O
in	O
the	O
filtering	O
system	O
.	O
so	O
you	O
can	O
see	O
the	O
scores	O
and	O
the	O
status	O
of	O
relevance	O
.	O
so	O
the	O
first	O
one	O
has	O
a	O
score	O
	O
.	O
and	O
it	O
s	O
relevant	O
.	O
the	O
second	O
one	O
is	O
not	O
relevant	O
.	O
of	O
course	O
we	O
have	O
a	O
lot	O
of	O
documents	O
for	O
which	O
we	O
don	O
t	O
know	O
the	O
status	O
because	O
we	O
will	O
have	O
to	O
the	O
user	O
.	O
so	O
as	O
you	O
can	O
see	O
here	O
we	O
only	O
see	O
the	O
judgements	O
of	O
documents	O
delivered	O
to	O
the	O
user	O
.	O
so	O
this	O
is	O
not	O
a	O
random	O
sample	O
.	O
so	O
it	O
s	O
a	O
censored	O
data	O
.	O
it	O
s	O
kind	O
of	O
biased	O
so	O
that	O
creates	O
some	O
difficulty	O
for	O
learning	O
.	O
and	O
secondly	O
there	O
are	O
in	O
general	O
very	O
little	O
labeled	O
data	O
and	O
very	O
few	O
relevant	O
data	O
so	O
it	O
s	O
it	O
s	O
also	O
challenging	O
for	O
machine	B
learning	I
approaches	O
.	O
typically	O
they	O
require	O
require	O
more	O
training	O
data	O
.	O
and	O
in	O
the	O
extreme	O
case	O
at	O
the	O
beginning	O
we	O
don	O
t	O
even	O
have	O
any	O
label	O
there	O
as	O
well	O
.	O
the	O
system	O
still	O
has	O
to	O
make	O
a	O
decision	O
so	O
that	O
s	O
a	O
very	O
difficult	O
problem	O
at	O
the	O
beginning	O
.	O
finally	O
the	O
results	O
of	O
this	O
issue	O
of	O
exploration	O
versus	O
exploitation	O
tradeoff	O
.	O
now	O
this	O
means	O
we	O
also	O
want	O
to	O
explore	O
the	O
document	O
space	O
a	O
little	O
bit	O
and	O
to	O
to	O
see	O
if	O
the	O
user	O
might	O
be	O
interested	O
in	O
the	O
documents	O
that	O
we	O
have	O
not	O
yet	O
labeled	O
.	O
so	O
in	O
other	O
words	O
we	O
re	O
going	O
to	O
explore	O
the	O
space	O
of	O
user	O
interests	O
by	O
testing	O
whether	O
the	O
user	O
might	O
be	O
interested	O
in	O
some	O
other	O
documents	O
that	O
currently	O
are	O
not	O
matching	O
the	O
user	O
s	O
interest	O
.	O
this	O
so	O
well	O
.	O
so	O
how	O
do	O
we	O
do	O
that	O
well	O
we	O
could	O
lower	O
the	O
threshold	O
a	O
little	O
bit	O
and	O
do	O
we	O
just	O
deliver	O
some	O
near	O
misses	O
to	O
the	O
user	O
to	O
see	O
what	O
the	O
user	O
would	O
respond	O
so	O
see	O
how	O
the	O
user	O
will	O
would	O
respond	O
to	O
this	O
extra	O
document	O
.	O
and	O
and	O
this	O
is	O
a	O
trade	O
off	O
because	O
on	O
the	O
one	O
hand	O
you	O
want	O
to	O
explore	O
but	O
on	O
the	O
other	O
hand	O
you	O
don	O
t	O
want	O
to	O
really	O
explore	O
too	O
much	O
because	O
then	O
you	O
would	O
over-deliver	O
non-relevant	O
information	O
.	O
so	O
exploitation	O
means	O
you	O
would	O
exploit	O
what	O
you	O
learn	O
about	O
the	O
user	O
.	O
and	O
let	O
s	O
say	O
you	O
know	O
the	O
user	O
is	O
interested	O
in	O
this	O
particular	O
topic	O
so	O
you	O
don	O
t	O
want	O
to	O
deviate	O
that	O
much	O
.	O
and	O
but	O
if	O
you	O
don	O
t	O
deviate	O
at	O
all	O
then	O
you	O
don	O
t	O
explore	O
at	O
all	O
.	O
that	O
s	O
also	O
not	O
good	O
.	O
you	O
might	O
miss	O
opportunity	O
to	O
learn	O
another	O
interest	O
of	O
the	O
user	O
.	O
so	O
this	O
is	O
a	O
dilemma	O
.	O
and	O
that	O
s	O
also	O
a	O
difficult	O
problem	O
to	O
solve	O
.	O
now	O
how	O
do	O
we	O
solve	O
these	O
problems	O
in	O
general	O
i	O
think	O
why	O
can	O
t	O
i	O
used	O
the	O
empirical	O
utility	O
optimization	O
strategy	O
and	O
this	O
strategy	O
is	O
basically	O
to	O
optimize	O
the	O
threshold	O
based	O
on	O
historical	O
data	O
just	O
as	O
you	O
have	O
seen	O
on	O
the	O
previous	O
slide	O
right	O
so	O
you	O
can	O
just	O
compute	O
the	O
utility	O
on	O
the	O
training	O
data	O
for	O
each	O
candidate	O
score	O
threshold	O
.	O
pretend	O
that	O
cut	O
at	O
this	O
point	O
.	O
what	O
if	O
i	O
cut	O
out	O
the	O
threshold	O
what	O
would	O
happen	O
what	O
s	O
utility	O
compute	O
the	O
utility	O
right	O
we	O
know	O
the	O
status	O
what	O
s	O
it	O
based	O
on	O
approximation	O
of	O
click-throughs	O
right	O
so	O
then	O
we	O
can	O
just	O
choose	O
this	O
threshold	O
that	O
gives	O
the	O
maximum	O
utility	O
on	O
the	O
training	O
data	O
.	O
now	O
but	O
this	O
of	O
course	O
doesn	O
t	O
account	O
for	O
exploration	O
that	O
we	O
just	O
talked	O
about	O
.	O
and	O
there	O
is	O
also	O
the	O
difficulty	O
of	O
bias	O
.	O
training	O
sample	O
as	O
we	O
mentioned	O
.	O
so	O
in	O
general	O
we	O
can	O
only	O
get	O
an	O
upper	O
bound	O
or	O
for	O
the	O
true	O
optimal	O
threshold	O
because	O
the	O
the	O
al	O
the	O
threshold	O
might	O
be	O
actually	O
lower	O
than	O
this	O
.	O
so	O
it	O
s	O
possible	O
that	O
the	O
discarded	O
item	O
might	O
be	O
actually	O
interesting	O
to	O
the	O
user	O
.	O
so	O
how	O
do	O
we	O
solve	O
this	O
problem	O
well	O
we	O
generally	O
as	O
i	O
said	O
we	O
can	O
lower	O
the	O
threshold	O
to	O
explore	O
a	O
little	O
bit	O
.	O
so	O
here	O
s	O
one	O
particular	O
approach	O
called	O
the	O
beta-gamma	B
threshold	I
learning	I
.	O
so	O
the	O
the	O
idea	O
is	O
foreign	O
.	O
so	O
here	O
i	O
show	O
a	O
ranked	O
list	O
of	O
all	O
the	O
training	O
documents	O
that	O
we	O
have	O
seen	O
so	O
far	O
.	O
and	O
they	O
are	O
ranked	O
by	O
their	O
positions	O
.	O
and	O
on	O
the	O
y-axis	O
we	O
show	O
the	O
utility	O
.	O
of	O
course	O
this	O
function	O
depends	O
on	O
how	O
you	O
specify	O
the	O
coefficients	O
in	O
the	O
utility	B
function	I
.	O
but	O
we	O
can	O
not	O
imagine	O
depending	O
on	O
the	O
cut	O
off	O
position	O
we	O
will	O
have	O
a	O
utility	O
.	O
that	O
means	O
suppose	O
i	O
cut	O
at	O
this	O
position	O
and	O
that	O
will	O
be	O
the	O
utility	O
.	O
so	O
we	O
can	O
for	O
example	O
i	O
then	O
find	O
some	O
cut	O
off	O
point	O
.	O
the	O
optimal	O
point	O
theta	O
optimal	O
is	O
the	O
point	O
when	O
we	O
would	O
achieve	O
the	O
maximum	O
utility	O
if	O
we	O
had	O
chosen	O
this	O
threshold	O
.	O
and	O
there	O
is	O
also	O
threshold	O
utility	O
threshold	O
.	O
as	O
you	O
can	O
see	O
at	O
this	O
cut	O
off	O
.	O
the	O
utility	O
is	O
	O
.	O
now	O
what	O
does	O
that	O
mean	O
that	O
means	O
if	O
i	O
lower	O
the	O
threshold	O
and	O
then	O
get	O
the	O
and	O
now	O
i	O
m	O
i	O
reach	O
this	O
threshold	O
the	O
utility	O
would	O
be	O
lower	O
but	O
it	O
s	O
still	O
positive	O
.	O
still	O
non-elective	O
at	O
least	O
.	O
so	O
it	O
s	O
not	O
as	O
high	O
as	O
the	O
optimal	O
utility	O
but	O
it	O
gives	O
us	O
a	O
a	O
safe	O
point	O
to	O
explore	O
the	O
threshold	O
.	O
as	O
i	O
just	O
explained	O
it	O
s	O
desirable	O
to	O
explore	O
the	O
interest	O
space	O
.	O
so	O
it	O
s	O
desirable	O
to	O
lower	O
the	O
threshold	O
based	O
on	O
your	O
training	O
data	O
.	O
so	O
that	O
means	O
in	O
general	O
we	O
want	O
to	O
set	O
the	O
threshold	O
somewhere	O
in	O
this	O
range	O
.	O
it	O
s	O
the	O
when	O
user	O
off	O
fault	O
to	O
control	O
the	O
the	O
deviation	O
from	O
the	O
optimal	O
utility	O
point	O
.	O
so	O
you	O
can	O
see	O
the	O
formula	O
of	O
the	O
threshold	O
will	O
be	O
just	O
the	O
incorporation	O
of	O
the	O
zero	O
utility	O
threshold	O
and	O
the	O
optimal	O
between	O
the	O
threshold	O
.	O
now	O
the	O
question	O
is	O
how	O
how	O
should	O
we	O
set	O
r	O
form	O
you	O
know	O
and	O
when	O
should	O
we	O
deviate	O
more	O
from	O
the	O
optimal	O
utility	O
point	O
.	O
well	O
this	O
can	O
depend	O
on	O
multiple	O
factors	O
and	O
the	O
one	O
way	O
to	O
solve	O
the	O
problem	O
is	O
to	O
encourage	O
this	O
threshold	O
mechanism	O
to	O
explore	O
up	O
the	O
point	O
and	O
that	O
s	O
a	O
safe	O
point	O
but	O
we	O
re	O
not	O
going	O
to	O
necessarily	O
reach	O
all	O
the	O
way	O
to	O
the	O
point	O
.	O
but	O
rather	O
we	O
re	O
going	O
to	O
use	O
other	O
parameters	O
to	O
further	O
define	O
alpha	O
.	O
and	O
this	O
specifically	O
is	O
as	O
follows	O
.	O
so	O
there	O
will	O
be	O
a	O
beta	B
parameter	O
to	O
control	O
.	O
the	O
deviation	O
from	O
the	O
optimal	O
threshold	O
.	O
and	O
this	O
can	O
be	O
based	O
on	O
for	O
example	O
can	O
be	O
accounting	O
for	O
the	O
over	O
throughout	O
the	O
training	O
data	O
let	O
s	O
say	O
.	O
and	O
so	O
this	O
can	O
be	O
just	O
the	O
adjustment	O
factor	O
.	O
but	O
what	O
s	O
more	O
interesting	O
is	O
this	O
gamma	B
parameter	O
here	O
and	O
you	O
can	O
see	O
in	O
this	O
formula	O
gamma	B
is	O
controlling	O
the	O
the	O
influence	O
of	O
the	O
number	O
of	O
examples	O
in	O
training	O
data	O
set	O
.	O
so	O
you	O
can	O
see	O
in	O
this	O
formula	O
as	O
n	O
which	O
denotes	O
the	O
number	O
of	O
training	O
examples	O
.	O
becomes	O
bigger	O
than	O
it	O
would	O
actually	O
encourage	O
less	O
exploration	O
.	O
in	O
other	O
words	O
when	O
n	O
is	O
very	O
small	O
it	O
will	O
try	O
to	O
explore	O
more	O
.	O
and	O
that	O
just	O
means	O
if	O
we	O
have	O
seen	O
few	O
examples	O
we	O
re	O
not	O
sure	O
whether	O
we	O
have	O
exhausted	O
the	O
space	O
of	O
interests	O
.	O
so	O
	O
.	O
but	O
as	O
we	O
have	O
seen	O
many	O
examples	O
from	O
the	O
user	O
many	O
data	O
points	O
then	O
we	O
feel	O
that	O
we	O
probably	O
dont	O
have	O
to	O
explore	O
more	O
.	O
so	O
this	O
gives	O
us	O
a	O
dynamic	O
of	O
strategy	O
for	O
exploration	O
right	O
the	O
more	O
examples	O
we	O
have	O
seen	O
the	O
less	O
exploration	O
we	O
are	O
going	O
to	O
do	O
.	O
so	O
the	O
threshold	O
will	O
be	O
closer	O
to	O
the	O
optimal	O
threshold	O
.	O
so	O
that	O
s	O
the	O
basic	O
idea	O
of	O
this	O
approach	O
.	O
now	O
this	O
approach	O
actually	O
has	O
been	O
working	O
well	O
in	O
some	O
evaluation	O
studies	O
.	O
and	O
particularly	O
effective	O
.	O
and	O
also	O
can	O
welcome	O
arbitrary	O
utility	O
with	O
a	O
appropriate	O
lower	O
bound	O
.	O
and	O
explicitly	O
addresses	O
exploration-exploration	O
tradeoff	O
.	O
and	O
it	O
kind	O
of	O
uses	O
a	O
zero	O
in	O
this	O
threshold	O
point	O
as	O
a	O
a	O
safeguard	O
.	O
for	O
exploration	O
and	O
exploiting	O
tradeoff	O
.	O
we	O
re	O
not	O
never	O
going	O
to	O
explore	O
further	O
than	O
the	O
zero	O
utility	O
point	O
.	O
so	O
if	O
you	O
take	O
the	O
analogy	O
of	O
gambling	O
and	O
you	O
you	O
don	O
t	O
want	O
to	O
risk	O
losing	O
money	O
.	O
you	O
know	O
so	O
it	O
s	O
a	O
safe	O
strategy	O
a	O
conservative	O
strategy	O
for	O
exploration	O
.	O
and	O
the	O
problem	O
is	O
of	O
course	O
this	O
approach	O
is	O
purely	O
heuristic	O
.	O
and	O
the	O
zero	O
utility	O
lower	O
bound	O
is	O
also	O
often	O
too	O
conservative	O
.	O
and	O
there	O
are	O
of	O
course	O
calls	O
are	O
more	O
advanced	O
than	O
machine	B
learning	I
projects	O
that	O
have	O
been	O
proposed	O
for	O
solving	O
these	O
problems	O
.	O
and	O
this	O
is	O
a	O
very	O
active	O
research	O
area	O
.	O
so	O
to	O
summarize	O
there	O
are	O
two	O
strategies	O
for	O
recommending	O
systems	O
or	O
filtering	O
systems	O
.	O
one	O
is	O
content	O
based	O
which	O
is	O
looking	O
at	O
the	O
item	O
similarity	O
.	O
and	O
the	O
other	O
is	O
collaborative	B
filtering	I
which	O
is	O
looking	O
at	O
the	O
user	O
similarity	O
.	O
in	O
this	O
lecture	O
we	O
have	O
covered	O
content-based	B
filtering	I
approach	O
.	O
in	O
the	O
next	O
lecture	O
we	O
re	O
going	O
to	O
talk	O
about	O
collaborative	B
filtering	I
.	O
the	O
content-based	B
filtering	I
system	O
we	O
generally	O
have	O
to	O
solve	O
several	O
problems	O
related	O
to	O
filtering	O
decision	O
and	O
learning	O
etc	O
.	O
and	O
such	O
a	O
system	O
can	O
actually	O
be	O
based	O
on	O
a	O
search	B
engine	I
system	O
by	O
adding	O
a	O
threshold	O
mechanism	O
and	O
adding	O
adaptive	O
learning	B
algorithm	I
to	O
allow	O
the	O
system	O
to	O
learn	O
from	O
long	O
term	O
feedback	O
from	O
the	O
user	O
.	O
so	O
to	O
summarize	O
our	O
discussion	O
of	O
recommender	B
systems	I
in	O
some	O
sense	O
the	O
filtering	O
task	O
of	O
recommended	O
is	O
easy	O
and	O
in	O
some	O
other	O
sense	O
and	O
the	O
task	O
is	O
actually	O
difficult	O
.	O
so	O
its	O
easy	O
because	O
the	O
user	O
dexpectations	O
though	O
in	O
this	O
case	O
the	O
system	O
takes	O
initiative	O
to	O
push	B
the	O
information	O
to	O
the	O
user	O
.	O
so	O
the	O
user	O
doesn	O
t	O
really	O
make	O
an	O
effort	O
.	O
so	O
any	O
recommendation	O
is	O
better	O
than	O
nothing	O
right	O
so	O
unless	O
you	O
recommend	O
that	O
all	O
the	O
you	O
know	O
noisy	O
items	O
or	O
useless	O
documents	O
if	O
you	O
can	O
recommend	O
that	O
some	O
useful	O
information	O
uses	O
general	O
would	O
appreciate	O
it	O
all	O
right	O
.	O
so	O
that	O
s	O
in	O
that	O
sense	O
that	O
s	O
easy	O
.	O
however	O
filtering	O
is	O
actually	O
a	O
much	O
harder	O
task	O
.	O
because	O
you	O
have	O
to	O
make	O
a	O
binary	O
decision	O
and	O
you	O
can	O
t	O
afford	O
waiting	O
for	O
a	O
lot	O
of	O
items	O
and	O
then	O
you	O
will	O
whether	O
one	O
item	O
is	O
better	O
than	O
others	O
.	O
you	O
have	O
to	O
make	O
a	O
decision	O
when	O
you	O
see	O
this	O
item	O
.	O
let	O
s	O
think	O
about	O
news	O
filtering	O
as	O
well	O
as	O
you	O
see	O
the	O
news	O
.	O
and	O
you	O
have	O
to	O
decide	O
whether	O
the	O
news	O
would	O
be	O
interesting	O
to	O
a	O
user	O
.	O
if	O
you	O
wait	O
for	O
a	O
few	O
days	O
well	O
even	O
if	O
you	O
can	O
make	O
accurate	O
recommendation	O
of	O
the	O
most	O
relevant	O
news	O
only	O
two	O
days	O
wouldn	O
t	O
be	O
significantly	O
decreased	O
.	O
another	O
reason	O
why	O
it	O
s	O
hard	O
it	O
s	O
because	O
of	O
data	O
sparseness	O
.	O
if	O
you	O
think	O
of	O
this	O
as	O
a	O
learning	O
problem	O
in	O
collaborative	B
filtering	I
for	O
example	O
it	O
s	O
purely	O
based	O
on	O
learning	O
from	O
the	O
past	O
ratings	O
.	O
so	O
if	O
you	O
don	O
t	O
have	O
many	O
ratings	O
there	O
s	O
really	O
not	O
much	O
you	O
can	O
do	O
right	O
and	O
may	O
i	O
just	O
mention	O
this	O
problem	O
.	O
this	O
is	O
actually	O
a	O
very	O
serious	O
problem	O
.	O
but	O
of	O
course	O
there	O
are	O
strategies	O
that	O
have	O
been	O
proposed	O
to	O
solve	O
the	O
problem	O
.	O
and	O
there	O
are	O
there	O
are	O
different	O
strategies	O
that	O
we	O
will	O
use	O
to	O
alleviate	O
the	O
problem	O
.	O
we	O
can	O
use	O
for	O
example	O
more	O
user	O
information	O
to	O
assess	O
their	O
similarity	O
instead	O
of	O
using	O
the	O
preferences	O
.	O
of	O
these	O
users	O
on	O
these	O
items	O
the	O
immediate	O
additional	O
information	O
or	O
better	O
for	O
about	O
the	O
user	O
etcetera	O
and	O
and	O
we	O
also	O
talked	O
about	O
the	O
two	O
strategies	O
for	O
filtering	O
task	O
.	O
one	O
is	O
content	O
based	O
where	O
we	O
look	O
at	O
items	O
in	O
clarity	O
you	O
know	O
there	O
s	O
a	O
clarity	O
of	O
filtering	O
where	O
we	O
look	O
at	O
the	O
user	O
similarity	O
.	O
and	O
they	O
obviously	O
can	O
be	O
combined	O
.	O
in	O
a	O
practical	O
system	O
you	O
can	O
imagine	O
they	O
generally	O
would	O
have	O
to	O
be	O
combined	O
.	O
so	O
that	O
will	O
give	O
us	O
a	O
hybrid	O
strategy	O
for	O
filtering	O
.	O
a	O
and	O
we	O
also	O
could	O
recall	B
that	O
we	O
talked	O
about	O
push	B
versus	O
pull	O
as	O
two	O
strategies	O
for	O
getting	O
access	O
to	O
the	O
text	O
data	O
.	O
and	O
recommend	O
the	O
system	O
is	O
it	O
will	O
help	O
users	O
in	O
the	O
push	B
mode	O
.	O
and	O
search	B
engines	I
are	O
certain	O
users	O
in	O
the	O
pull	O
mode	O
.	O
of	O
using	O
the	O
tool	O
should	O
be	O
combined	O
and	O
they	O
can	O
be	O
combined	O
into	O
have	O
a	O
system	O
that	O
can	O
support	O
user	O
with	O
multiple	O
mode	O
and	O
formation	O
access	O
.	O
so	O
in	O
the	O
future	O
we	O
could	O
anticipate	O
for	O
such	O
a	O
system	O
to	O
be	O
more	O
usable	O
to	O
a	O
user	O
.	O
and	O
also	O
this	O
is	O
a	O
active	O
research	O
area	O
so	O
there	O
are	O
a	O
lot	O
of	O
new	O
algorithms	O
being	O
being	O
proposed	O
over	O
time	O
.	O
in	O
particular	O
those	O
new	O
algorithms	O
tend	O
to	O
use	O
a	O
lot	O
of	O
context	O
information	O
.	O
now	O
the	O
context	O
here	O
could	O
be	O
the	O
context	O
of	O
the	O
user	O
you	O
know	O
it	O
could	O
also	O
be	O
context	O
of	O
documents	O
or	O
items	O
.	O
the	O
items	O
are	O
not	O
isolated	O
.	O
they	O
are	O
connected	O
in	O
many	O
ways	O
.	O
the	O
users	O
might	O
form	O
social	O
network	O
as	O
well	O
so	O
there	O
s	O
a	O
rich	O
context	O
there	O
that	O
we	O
can	O
leverage	O
in	O
order	O
to	O
really	O
solve	O
the	O
problem	O
well	O
and	O
then	O
that	O
s	O
a	O
active	O
research	O
area	O
where	O
also	O
machine	B
learning	B
algorithms	I
have	O
been	O
applied	O
.	O
here	O
are	O
some	O
additional	O
readings	O
in	O
the	O
handbook	O
called	O
recommender	B
systems	I
.	O
and	O
has	O
a	O
collection	O
of	O
a	O
lot	O
of	O
good	O
articles	O
that	O
can	O
give	O
you	O
an	O
overview	O
of	O
a	O
number	O
of	O
specific	O
approaches	O
to	O
recommender	B
systems	I
.	O
this	O
lecture	O
is	O
a	O
summary	O
of	O
this	O
course	O
.	O
this	O
map	B
shows	O
the	O
major	O
topics	O
we	O
have	O
covered	O
in	O
this	O
course	O
.	O
and	O
here	O
are	O
some	O
key	O
high-level	O
take-away	O
messages	O
.	O
first	O
we	O
talk	O
about	O
natural	B
language	I
content	O
analysis	O
.	O
here	O
the	O
main	O
take-away	O
message	O
is	O
natural	B
language	I
processing	I
is	O
the	O
foundation	O
for	O
textual	O
retrieval	O
but	O
current	O
nlp	B
isn	O
t	O
robust	O
enough	O
.	O
so	O
the	O
back	O
of	O
words	O
replenishing	O
is	O
generally	O
the	O
main	O
method	O
used	O
in	O
modern	O
search	B
engines	I
and	O
it	O
s	O
often	O
sufficient	O
for	O
most	O
of	O
the	O
search	B
tasks	I
.	O
but	O
obviously	O
for	O
more	O
compass	O
search	B
tasks	I
then	O
we	O
need	O
a	O
deeper	O
measurement	O
processing	O
techniques	O
.	O
and	O
we	O
then	O
talked	O
about	O
a	O
high-level	O
strategies	O
for	O
text	O
access	O
and	O
we	O
talked	O
about	O
push	B
versus	O
pull	O
in	O
plural	O
.	O
we	O
talked	O
about	O
a	O
query	O
which	O
is	O
browsing	O
.	O
now	O
in	O
general	O
in	O
future	O
search	B
engines	I
we	O
should	O
integrate	O
all	O
these	O
techniques	O
to	O
provide	O
a	O
multiple	O
information	B
access	I
and	O
then	O
we	O
talked	O
about	O
a	O
number	O
of	O
issues	O
related	O
to	O
search	B
engines	I
.	O
we	O
talked	O
about	O
the	O
search	O
problem	O
and	O
we	O
framed	O
that	O
as	O
a	O
ranking	B
problem	I
and	O
we	O
talked	O
about	O
the	O
a	O
number	O
of	O
retrieval	B
methods	I
.	O
we	O
start	O
with	O
an	O
overview	O
of	O
the	O
vector	B
space	I
model	I
and	O
probabilistic	B
model	I
and	O
then	O
we	O
talked	O
about	O
the	O
vector	B
space	I
model	I
in	O
that	O
.	O
we	O
also	O
later	O
talked	O
about	O
leverageable	O
learning	O
approach	O
and	O
that	O
s	O
probabilistic	B
model	I
.	O
and	O
here	O
the	O
main	O
take-away	O
message	O
is	O
that	O
model	O
retrieval	O
functions	O
tend	O
to	O
look	O
similar	O
and	O
they	O
generally	O
use	O
various	O
heuristics	O
.	O
most	O
important	O
ones	O
are	O
tf-idf	B
waiting	O
document	O
length	B
normalization	I
and	O
that	O
tf	B
is	O
often	O
transformed	O
through	O
a	O
sub-linear	O
transformation	O
function	O
and	O
then	O
we	O
talked	O
about	O
how	O
to	O
implement	O
a	O
retrieval	O
system	O
.	O
and	O
here	O
the	O
main	O
technique	O
that	O
we	O
talked	O
about	O
how	O
to	O
construct	O
an	O
inverted	B
index	I
.	O
so	O
that	O
we	O
can	O
prepare	O
the	O
system	O
to	O
answer	O
a	O
query	O
quickly	O
and	O
we	O
talked	O
about	O
how	O
to	O
to	O
fast	O
research	O
by	O
using	O
the	O
inverted	B
index	I
and	O
we	O
then	O
talked	O
about	O
how	O
to	O
evaluate	O
the	O
text	B
retrieval	I
system	O
mainly	O
introduced	O
the	O
cranfield	B
evaluation	B
methodology	I
.	O
this	O
was	O
a	O
very	O
important	O
the	O
various	O
methodology	O
of	O
that	O
can	O
be	O
applied	O
to	O
many	O
tasks	O
.	O
we	O
talked	O
about	O
the	O
major	O
evaluation	O
measures	O
.	O
so	O
the	O
most	O
important	O
measures	O
for	O
a	O
search	B
engine	I
are	O
map	B
mean	B
average	B
precision	B
and	O
ndcg	B
.	O
normalized	O
discounted	O
accumulative	O
gain	B
and	O
also	O
precision	B
and	O
record	O
the	O
two	O
basic	O
measures	O
.	O
and	O
we	O
then	O
talked	O
about	O
feedback	O
techniques	O
.	O
and	O
we	O
talked	O
about	O
the	O
rock	O
you	O
in	O
the	O
vector	B
space	I
model	I
and	O
the	O
mixture	O
model	O
in	O
the	O
language	B
modeling	I
approach	I
.	O
feedback	O
is	O
very	O
important	O
technique	O
especially	O
considering	O
the	O
opportunity	O
of	O
learning	O
from	O
a	O
lot	O
of	O
pixels	O
on	O
the	O
web	O
.	O
we	O
then	O
talked	O
about	O
the	O
web	B
search	I
.	O
and	O
here	O
we	O
talk	O
about	O
the	O
how	O
to	O
use	O
parallel	O
indexing	O
to	O
resolve	O
the	O
scalability	O
issue	O
in	O
indexing	O
we	O
introduce	O
a	O
mapreduce	B
and	O
then	O
we	O
talked	O
about	O
the	O
how	O
to	O
using	O
information	O
interacting	O
pull	O
search	O
.	O
we	O
talked	O
about	O
page	O
random	O
hits	B
as	O
the	O
major	O
algorithms	O
to	O
analyze	O
links	O
on	O
the	O
web	O
.	O
we	O
then	O
talked	O
about	O
learning	B
to	I
rank	I
.	O
this	O
is	O
a	O
use	O
of	O
machine	B
learning	I
to	O
combine	O
multiple	O
features	O
for	O
improving	O
scoring	O
.	O
not	O
only	O
the	O
effectiveness	O
can	O
be	O
improved	O
using	O
this	O
approach	O
but	O
we	O
can	O
also	O
improve	O
the	O
robustness	O
of	O
the	O
ranking	B
function	I
so	O
that	O
it	O
s	O
not	O
easy	O
to	O
spam	O
a	O
search	B
engine	I
with	O
just	O
a	O
a	O
some	O
features	O
to	O
promote	O
a	O
page	O
.	O
and	O
finally	O
we	O
talked	O
about	O
the	O
future	O
of	O
web	B
search	I
.	O
we	O
talked	O
about	O
some	O
major	O
interactions	O
that	O
we	O
might	O
assume	O
in	O
the	O
future	O
in	O
improving	O
the	O
current	O
generation	O
of	O
search	B
engines	I
.	O
and	O
then	O
finally	O
we	O
talked	O
about	O
the	O
recommender	B
system	I
and	O
these	O
are	O
systems	O
to	O
implement	O
the	O
push	B
mode	O
and	O
we	O
ll	O
talk	O
about	O
the	O
two	O
approaches	O
.	O
one	O
is	O
content	O
based	O
one	O
is	O
collaborative	B
filtering	I
and	O
they	O
can	O
be	O
combined	O
together	O
.	O
now	O
an	O
obvious	O
missing	O
piece	O
in	O
this	O
picture	O
is	O
the	O
user	O
you	O
can	O
see	O
.	O
so	O
user	O
interface	O
is	O
also	O
a	O
important	O
component	O
in	O
any	O
search	B
engine	I
even	O
though	O
the	O
current	O
search	O
interface	O
is	O
relatively	O
simple	O
.	O
there	O
actually	O
have	O
been	O
a	O
lot	O
of	O
studies	O
of	O
user	O
interfaces	O
related	O
to	O
visualization	O
for	O
example	O
and	O
this	O
is	O
topic	O
to	O
that	O
you	O
can	O
learn	O
more	O
by	O
reading	O
this	O
book	O
.	O
it	O
s	O
a	O
excellent	O
book	O
about	O
all	O
kind	O
of	O
studies	O
of	O
search	O
user	O
interface	O
.	O
if	O
you	O
want	O
to	O
know	O
more	O
about	O
the	O
the	O
topics	O
that	O
we	O
talked	O
about	O
you	O
can	O
also	O
read	O
some	O
additional	O
readings	O
that	O
are	O
listed	O
here	O
.	O
in	O
this	O
short	O
course	O
we	O
are	O
only	O
managing	O
to	O
cover	O
some	O
basic	O
topics	O
in	O
text	B
retrieval	I
in	O
search	B
engines	I
.	O
and	O
these	O
resources	O
provide	O
additional	O
information	O
about	O
more	O
advanced	O
topics	O
and	O
they	O
give	O
more	O
thorough	O
treatment	O
of	O
some	O
of	O
the	O
topics	O
that	O
we	O
talked	O
about	O
.	O
and	O
a	O
main	O
source	O
is	O
synthesis	O
digital	O
library	O
where	O
you	O
can	O
see	O
a	O
lot	O
of	O
short	O
textbook	O
or	O
textbooks	O
or	O
long	O
tutorials	O
.	O
they	O
tend	O
to	O
provide	O
us	O
with	O
a	O
lot	O
of	O
information	O
to	O
explain	O
a	O
topic	O
and	O
there	O
are	O
multiple	O
series	O
that	O
are	O
related	O
to	O
this	O
course	O
.	O
one	O
is	O
information	O
concepts	O
retrieval	O
and	O
services	O
.	O
another	O
is	O
human	O
language	O
technology	O
and	O
yet	O
another	O
is	O
artificial	O
intelligence	O
and	O
machine	B
learning	I
.	O
there	O
are	O
also	O
some	O
major	O
journals	O
and	O
conferences	O
listed	O
over	O
here	O
that	O
tend	O
to	O
have	O
a	O
lot	O
of	O
research	O
papers	O
related	O
to	O
the	O
topic	O
of	O
this	O
course	O
.	O
and	O
finally	O
for	O
more	O
information	O
about	O
resources	O
including	O
readings	O
and	O
tool	O
kits	O
etc	O
.	O
you	O
can	O
check	O
out	O
this	O
url	O
.	O
so	O
if	O
you	O
have	O
not	O
taken	O
the	O
text	O
mining	O
course	O
in	O
this	O
in	O
this	O
data	O
mining	O
specialization	O
series	O
then	O
naturally	O
the	O
next	O
step	O
is	O
to	O
take	O
that	O
calls	O
.	O
as	O
this	O
picture	O
shows	O
to	O
mine	O
the	O
text	O
data	O
we	O
generally	O
need	O
two	O
kinds	O
of	O
techniques	O
.	O
one	O
is	O
text	B
retrieval	I
which	O
is	O
covered	O
in	O
this	O
course	O
.	O
and	O
these	O
techniques	O
will	O
help	O
us	O
convert	O
raw	O
big	O
text	O
data	O
into	O
small	O
relevant	O
text	O
data	O
which	O
are	O
actually	O
needed	O
in	O
the	O
specific	O
application	O
.	O
and	O
human	O
plays	O
important	O
role	O
in	O
mining	O
any	O
text	O
data	O
because	O
text	O
data	O
is	O
written	O
for	O
humans	O
to	O
consume	O
.	O
so	O
involving	O
humans	O
in	O
the	O
process	O
of	O
data	O
mining	O
is	O
very	O
important	O
.	O
and	O
in	O
this	O
course	O
we	O
have	O
covered	O
various	O
strategies	O
to	O
help	O
users	O
get	O
access	O
to	O
the	O
most	O
relevant	O
data	O
.	O
these	O
techniques	O
are	O
also	O
essential	O
in	O
any	O
text	O
mining	O
system	O
to	O
help	O
provide	O
providence	O
and	O
to	O
help	O
users	O
interpret	O
the	O
inner	O
patterns	O
that	O
the	O
user	O
would	O
find	O
through	O
text	O
data	O
mining	O
.	O
so	O
in	O
general	O
the	O
user	O
would	O
have	O
to	O
go	O
back	O
to	O
the	O
original	O
data	O
to	O
better	O
understand	O
the	O
patterns	O
.	O
so	O
the	O
text	O
mining	O
course	O
or	O
rather	O
text	O
mining	O
and	O
ana	O
analytics	O
course	O
will	O
be	O
deal	O
dealing	O
with	O
what	O
to	O
do	O
once	O
the	O
user	O
has	O
found	O
the	O
information	O
.	O
so	O
this	O
is	O
a	O
in	O
this	O
picture	O
where	O
we	O
would	O
convert	O
the	O
text	O
data	O
into	O
action	O
or	O
knowledge	O
.	O
and	O
this	O
has	O
to	O
do	O
with	O
helping	O
users	O
to	O
go	O
further	O
digest	O
with	O
a	O
found	O
information	O
or	O
to	O
find	O
the	O
patterns	O
and	O
to	O
reveal	O
knowledge	O
buried	O
in	O
text	O
and	O
such	O
knowledge	O
can	O
be	O
used	O
in	O
application	O
system	O
to	O
help	O
decision-making	O
or	O
to	O
help	O
user	O
finish	O
a	O
task	O
.	O
so	O
if	O
you	O
have	O
not	O
taken	O
that	O
course	O
the	O
natural	O
step	O
and	O
the	O
natural	O
next	O
step	O
would	O
be	O
to	O
take	O
that	O
course	O
.	O
thank	O
you	O
for	O
taking	O
this	O
course	O
.	O
i	O
hope	O
you	O
have	O
found	O
this	O
course	O
to	O
be	O
useful	O
to	O
you	O
and	O
i	O
look	O
forward	O
to	O
interacting	O
with	O
you	O
at	O
a	O
future	O
activity	O
.	O