<!DOCTYPE html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="theme-color" content="#2D2D2D" />
<title>NLTK :: Sample usage for corpus</title>
<link rel="stylesheet" href="../_static/css/nltk_theme.css"/>
<link rel="stylesheet" href="../_static/css/custom.css"/>
<script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
<script type="text/javascript" src="../_static/documentation_options.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/sphinx_highlight.js"></script>
<script src="https://email.tl.fortawesome.com/c/eJxNjUEOgyAQAF8jR7Kw6wIHDh7sP1Cw2mgxgmn6-3JsMqc5zEQfE8dkxOY1KKMUOI3ACFKRJpSW2AAp7ontYIaxI6i7XPJVwyeVfCQ550Os3jLrGSNOLgbdAy6s0PBk2TFNjEbsfq31LB0OnX407pJa5v2faRadwSW63mn5KuLyR9j2tgx3zecanl-55R_-jjPs"></script>
</head>
<body>
<div id="nltk-theme-container">
<header>
<div id="logo-container">
<h1>
<a href="../index.html">NLTK</a>
</h1>
</div>
<div id="project-container">
<h1>Documentation</h1>
</div>
<a id="menu-toggle" class="fa fa-bars" aria-hidden="true"></a>
<script type="text/javascript">
$("#menu-toggle").click(function() {
$("#menu-toggle").toggleClass("toggled");
$("#side-menu-container").slideToggle(300);
});
</script>
</header>
<div id="content-container">
<div id="side-menu-container">
<div id="search" role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<div id="side-menu" role="navigation">
<p class="caption" role="heading"><span class="caption-text">NLTK Documentation</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../api/nltk.html">API Reference</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../howto.html">Example Usage</a></li>
<li class="toctree-l1"><a class="reference internal" href="../py-modindex.html">Module Index</a></li>
<li class="toctree-l1"><a class="reference external" href="https://github.com/nltk/nltk/wiki">Wiki</a></li>
<li class="toctree-l1"><a class="reference external" href="https://github.com/nltk/nltk/wiki/FAQ">FAQ</a></li>
<li class="toctree-l1"><a class="reference external" href="https://github.com/nltk/nltk/issues">Open Issues</a></li>
<li class="toctree-l1"><a class="reference external" href="https://github.com/nltk/nltk">NLTK on GitHub</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Installation</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../install.html">Installing NLTK</a></li>
<li class="toctree-l1"><a class="reference internal" href="../data.html">Installing NLTK Data</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">More</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../news.html">Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contribute.html">Contributing to NLTK</a></li>
<li class="toctree-l1"><a class="reference internal" href="../team.html">NLTK Team</a></li>
</ul>
</div>
</div>
<div id="main-content-container">
<div id="main-content" role="main">
<section id="sample-usage-for-corpus">
<h1>Sample usage for corpus<a class="headerlink" href="#sample-usage-for-corpus" title="Link to this heading">¶</a></h1>
<section id="corpus-readers">
<h2>Corpus Readers<a class="headerlink" href="#corpus-readers" title="Link to this heading">¶</a></h2>
<p>The <cite>nltk.corpus</cite> package defines a collection of <em>corpus reader</em>
classes, which can be used to access the contents of a diverse set of
corpora. The list of available corpora is given at:</p>
<p><a class="reference external" href="https://www.nltk.org/nltk_data/">https://www.nltk.org/nltk_data/</a></p>
<p>Each corpus reader class is specialized to handle a specific
corpus format. In addition, the <cite>nltk.corpus</cite> package automatically
creates a set of corpus reader instances that can be used to access
the corpora in the NLTK data package.
Section <a class="reference internal" href="#corpus-reader-objects">Corpus Reader Objects</a> (“Corpus Reader Objects”) describes
the corpus reader instances that can be used to read the corpora in
the NLTK data package. Section <a class="reference internal" href="#corpus-reader-classes">Corpus Reader Classes</a> (“Corpus
Reader Classes”) describes the corpus reader classes themselves, and
discusses the issues involved in creating new corpus reader objects
and new corpus reader classes. Section <a class="reference internal" href="#regression-tests">Regression Tests</a>
(“Regression Tests”) contains regression tests for the corpus readers
and associated functions and classes.</p>
<nav class="contents" id="table-of-contents">
<p class="topic-title"><strong>Table of Contents</strong></p>
<ul class="simple">
<li><p><a class="reference internal" href="#sample-usage-for-corpus" id="id2">Sample usage for corpus</a></p>
<ul>
<li><p><a class="reference internal" href="#corpus-readers" id="id3">Corpus Readers</a></p>
<ul>
<li><p><a class="reference internal" href="#corpus-reader-objects" id="id4">Corpus Reader Objects</a></p>
<ul>
<li><p><a class="reference internal" href="#overview" id="id5">Overview</a></p></li>
<li><p><a class="reference internal" href="#plaintext-corpora" id="id6">Plaintext Corpora</a></p></li>
<li><p><a class="reference internal" href="#tagged-corpora" id="id7">Tagged Corpora</a></p></li>
<li><p><a class="reference internal" href="#chunked-corpora" id="id8">Chunked Corpora</a></p></li>
<li><p><a class="reference internal" href="#parsed-corpora" id="id9">Parsed Corpora</a></p></li>
<li><p><a class="reference internal" href="#word-lists-and-lexicons" id="id10">Word Lists and Lexicons</a></p></li>
<li><p><a class="reference internal" href="#wordnet" id="id11">WordNet</a></p></li>
<li><p><a class="reference internal" href="#framenet" id="id12">FrameNet</a></p></li>
<li><p><a class="reference internal" href="#propbank" id="id13">PropBank</a></p></li>
<li><p><a class="reference internal" href="#sentiwordnet" id="id14">SentiWordNet</a></p></li>
<li><p><a class="reference internal" href="#categorized-corpora" id="id15">Categorized Corpora</a></p></li>
<li><p><a class="reference internal" href="#other-corpora" id="id16">Other Corpora</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#corpus-reader-classes" id="id17">Corpus Reader Classes</a></p>
<ul>
<li><p><a class="reference internal" href="#automatically-created-corpus-reader-instances" id="id18">Automatically Created Corpus Reader Instances</a></p></li>
<li><p><a class="reference internal" href="#creating-new-corpus-reader-instances" id="id19">Creating New Corpus Reader Instances</a></p></li>
<li><p><a class="reference internal" href="#corpus-types" id="id20">Corpus Types</a></p></li>
<li><p><a class="reference internal" href="#common-corpus-reader-methods" id="id21">Common Corpus Reader Methods</a></p></li>
<li><p><a class="reference internal" href="#data-access-methods" id="id22">Data Access Methods</a></p></li>
<li><p><a class="reference internal" href="#stream-backed-corpus-views" id="id23">Stream Backed Corpus Views</a></p></li>
<li><p><a class="reference internal" href="#writing-new-corpus-readers" id="id24">Writing New Corpus Readers</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#regression-tests" id="id25">Regression Tests</a></p>
<ul>
<li><p><a class="reference internal" href="#plaintext-corpus-reader" id="id26">Plaintext Corpus Reader</a></p></li>
<li><p><a class="reference internal" href="#tagged-corpus-reader" id="id27">Tagged Corpus Reader</a></p></li>
<li><p><a class="reference internal" href="#categorized-markdown-corpus-reader" id="id28">Categorized Markdown Corpus Reader</a></p></li>
<li><p><a class="reference internal" href="#verbnet-corpus-reader" id="id29">Verbnet Corpus Reader</a></p></li>
<li><p><a class="reference internal" href="#corpus-view-regression-tests" id="id30">Corpus View Regression Tests</a></p></li>
<li><p><a class="reference internal" href="#seekableunicodestreamreader" id="id31">SeekableUnicodeStreamReader</a></p></li>
<li><p><a class="reference internal" href="#squashed-bugs" id="id32">Squashed Bugs</a></p></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</nav>
<section id="corpus-reader-objects">
<h3>Corpus Reader Objects<a class="headerlink" href="#corpus-reader-objects" title="Link to this heading">¶</a></h3>
<section id="overview">
<h4>Overview<a class="headerlink" href="#overview" title="Link to this heading">¶</a></h4>
<p>NLTK includes a diverse set of corpora which can be
read using the <code class="docutils literal notranslate"><span class="pre">nltk.corpus</span></code> package. Each corpus is accessed by
means of a “corpus reader” object from <code class="docutils literal notranslate"><span class="pre">nltk.corpus</span></code>:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">nltk.corpus</span>
<span class="gp">>>> </span><span class="c1"># The Brown corpus:</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="go"><CategorizedTaggedCorpusReader in '.../corpora/brown'...></span>
<span class="gp">>>> </span><span class="c1"># The Penn Treebank Corpus:</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">treebank</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="go"><BracketParseCorpusReader in '.../corpora/treebank/combined'...></span>
<span class="gp">>>> </span><span class="c1"># The Name Genders Corpus:</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">names</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="go"><WordListCorpusReader in '.../corpora/names'...></span>
<span class="gp">>>> </span><span class="c1"># The Inaugural Address Corpus:</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">inaugural</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="go"><PlaintextCorpusReader in '.../corpora/inaugural'...></span>
</pre></div>
</div>
<p>Most corpora consist of a set of files, each containing a document (or
other pieces of text). A list of identifiers for these files is
accessed via the <code class="docutils literal notranslate"><span class="pre">fileids()</span></code> method of the corpus reader:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">treebank</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['wsj_0001.mrg', 'wsj_0002.mrg', 'wsj_0003.mrg', 'wsj_0004.mrg', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">inaugural</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['1789-Washington.txt', '1793-Washington.txt', '1797-Adams.txt', ...]</span>
</pre></div>
</div>
<p>Each corpus reader provides a variety of methods to read data from the
corpus, depending on the format of the corpus. For example, plaintext
corpora support methods to read the corpus as raw text, a list of
words, a list of sentences, or a list of paragraphs.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">inaugural</span>
<span class="gp">>>> </span><span class="n">inaugural</span><span class="o">.</span><span class="n">raw</span><span class="p">(</span><span class="s1">'1789-Washington.txt'</span><span class="p">)</span>
<span class="go">'Fellow-Citizens of the Senate ...'</span>
<span class="gp">>>> </span><span class="n">inaugural</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'1789-Washington.txt'</span><span class="p">)</span>
<span class="go">['Fellow', '-', 'Citizens', 'of', 'the', ...]</span>
<span class="gp">>>> </span><span class="n">inaugural</span><span class="o">.</span><span class="n">sents</span><span class="p">(</span><span class="s1">'1789-Washington.txt'</span><span class="p">)</span>
<span class="go">[['Fellow', '-', 'Citizens'...], ['Among', 'the', 'vicissitudes'...]...]</span>
<span class="gp">>>> </span><span class="n">inaugural</span><span class="o">.</span><span class="n">paras</span><span class="p">(</span><span class="s1">'1789-Washington.txt'</span><span class="p">)</span>
<span class="go">[[['Fellow', '-', 'Citizens'...]],</span>
<span class="go"> [['Among', 'the', 'vicissitudes'...],</span>
<span class="go"> ['On', 'the', 'one', 'hand', ',', 'I'...]...]...]</span>
</pre></div>
</div>
<p>Each of these reader methods may be given a single document’s item
name or a list of document item names. When given a list of document
item names, the reader methods will concatenate together the contents
of the individual documents.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">l1</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">inaugural</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'1789-Washington.txt'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">l2</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">inaugural</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'1793-Washington.txt'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">l3</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">inaugural</span><span class="o">.</span><span class="n">words</span><span class="p">([</span><span class="s1">'1789-Washington.txt'</span><span class="p">,</span> <span class="s1">'1793-Washington.txt'</span><span class="p">]))</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1">+</span><span class="si">%s</span><span class="s1"> == </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">l1</span><span class="p">,</span> <span class="n">l2</span><span class="p">,</span> <span class="n">l3</span><span class="p">))</span>
<span class="go">1538+147 == 1685</span>
</pre></div>
</div>
<p>If the reader methods are called without any arguments, they will
typically load all documents in the corpus.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">inaugural</span><span class="o">.</span><span class="n">words</span><span class="p">())</span>
<span class="go">152901</span>
</pre></div>
</div>
<p>If a corpus contains a README file, it can be accessed with a <code class="docutils literal notranslate"><span class="pre">readme()</span></code> method:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">inaugural</span><span class="o">.</span><span class="n">readme</span><span class="p">()[:</span><span class="mi">32</span><span class="p">]</span>
<span class="go">'C-Span Inaugural Address Corpus\n'</span>
</pre></div>
</div>
</section>
<section id="plaintext-corpora">
<h4>Plaintext Corpora<a class="headerlink" href="#plaintext-corpora" title="Link to this heading">¶</a></h4>
<p>Here are the first few words from each of NLTK’s plaintext corpora:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">abc</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['PM', 'denies', 'knowledge', 'of', 'AWB', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">genesis</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['In', 'the', 'beginning', 'God', 'created', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">gutenberg</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="n">fileids</span><span class="o">=</span><span class="s1">'austen-emma.txt'</span><span class="p">)</span>
<span class="go">['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">inaugural</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['Fellow', '-', 'Citizens', 'of', 'the', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">state_union</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['PRESIDENT', 'HARRY', 'S', '.', 'TRUMAN', "'", ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">webtext</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['Cookie', 'Manager', ':', '"', 'Don', "'", 't', ...]</span>
</pre></div>
</div>
</section>
<section id="tagged-corpora">
<h4>Tagged Corpora<a class="headerlink" href="#tagged-corpora" title="Link to this heading">¶</a></h4>
<p>In addition to the plaintext corpora, NLTK’s data package also
contains a wide variety of annotated corpora. For example, the Brown
Corpus is annotated with part-of-speech tags, and defines additional
methods <code class="docutils literal notranslate"><span class="pre">tagged_*()</span></code> which words as <cite>(word,tag)</cite> tuples, rather
than just bare word strings.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">brown</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">brown</span><span class="o">.</span><span class="n">words</span><span class="p">())</span>
<span class="go">['The', 'Fulton', 'County', 'Grand', 'Jury', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">())</span>
<span class="go">[('The', 'AT'), ('Fulton', 'NP-TL'), ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">brown</span><span class="o">.</span><span class="n">sents</span><span class="p">())</span>
<span class="go">[['The', 'Fulton', 'County'...], ['The', 'jury', 'further'...], ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_sents</span><span class="p">())</span>
<span class="go">[[('The', 'AT'), ('Fulton', 'NP-TL')...],</span>
<span class="go"> [('The', 'AT'), ('jury', 'NN'), ('further', 'RBR')...]...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">brown</span><span class="o">.</span><span class="n">paras</span><span class="p">(</span><span class="n">categories</span><span class="o">=</span><span class="s1">'reviews'</span><span class="p">))</span>
<span class="go">[[['It', 'is', 'not', 'news', 'that', 'Nathan', 'Milstein'...],</span>
<span class="go"> ['Certainly', 'not', 'in', 'Orchestra', 'Hall', 'where'...]],</span>
<span class="go"> [['There', 'was', 'about', 'that', 'song', 'something', ...],</span>
<span class="go"> ['Not', 'the', 'noblest', 'performance', 'we', 'have', ...], ...], ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_paras</span><span class="p">(</span><span class="n">categories</span><span class="o">=</span><span class="s1">'reviews'</span><span class="p">))</span>
<span class="go">[[[('It', 'PPS'), ('is', 'BEZ'), ('not', '*'), ...],</span>
<span class="go"> [('Certainly', 'RB'), ('not', '*'), ('in', 'IN'), ...]],</span>
<span class="go"> [[('There', 'EX'), ('was', 'BEDZ'), ('about', 'IN'), ...],</span>
<span class="go"> [('Not', '*'), ('the', 'AT'), ('noblest', 'JJT'), ...], ...], ...]</span>
</pre></div>
</div>
<p>Similarly, the Indian Language POS-Tagged Corpus includes samples of
Indian text annotated with part-of-speech tags:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">indian</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">indian</span><span class="o">.</span><span class="n">words</span><span class="p">())</span>
<span class="go">['\xe0\xa6\xae\xe0\xa6\xb9\xe0\xa6\xbf\...',</span>
<span class="go"> '\xe0\xa6\xb8\xe0\xa6\xa8\xe0\xa7\x8d\xe0...', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">indian</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">())</span>
<span class="go">[('\xe0\xa6\xae\xe0\xa6\xb9\xe0\xa6\xbf...', 'NN'),</span>
<span class="go"> ('\xe0\xa6\xb8\xe0\xa6\xa8\xe0\xa7\x8d\xe0...', 'NN'), ...]</span>
</pre></div>
</div>
<p>Several tagged corpora support access to a simplified, universal tagset, e.g. where all nouns
tags are collapsed to a single category <code class="docutils literal notranslate"><span class="pre">NOUN</span></code>:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_sents</span><span class="p">(</span><span class="n">tagset</span><span class="o">=</span><span class="s1">'universal'</span><span class="p">))</span>
<span class="go">[[('The', 'DET'), ('Fulton', 'NOUN'), ('County', 'NOUN'), ('Grand', 'ADJ'), ('Jury', 'NOUN'), ...],</span>
<span class="go"> [('The', 'DET'), ('jury', 'NOUN'), ('further', 'ADV'), ('said', 'VERB'), ('in', 'ADP'), ...]...]</span>
<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">conll2000</span><span class="p">,</span> <span class="n">switchboard</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">conll2000</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">(</span><span class="n">tagset</span><span class="o">=</span><span class="s1">'universal'</span><span class="p">))</span>
<span class="go">[('Confidence', 'NOUN'), ('in', 'ADP'), ...]</span>
</pre></div>
</div>
<p>Use <code class="docutils literal notranslate"><span class="pre">nltk.app.pos_concordance()</span></code> to access a GUI for searching tagged corpora.</p>
</section>
<section id="chunked-corpora">
<h4>Chunked Corpora<a class="headerlink" href="#chunked-corpora" title="Link to this heading">¶</a></h4>
<p>The CoNLL corpora also provide chunk structures, which are encoded as
flat trees. The CoNLL 2000 Corpus includes phrasal chunks; and the
CoNLL 2002 Corpus includes named entity chunks.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">conll2000</span><span class="p">,</span> <span class="n">conll2002</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">conll2000</span><span class="o">.</span><span class="n">sents</span><span class="p">())</span>
<span class="go">[['Confidence', 'in', 'the', 'pound', 'is', 'widely', ...],</span>
<span class="go"> ['Chancellor', 'of', 'the', 'Exchequer', ...], ...]</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">tree</span> <span class="ow">in</span> <span class="n">conll2000</span><span class="o">.</span><span class="n">chunked_sents</span><span class="p">()[:</span><span class="mi">2</span><span class="p">]:</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>
<span class="go">(S</span>
<span class="go"> (NP Confidence/NN)</span>
<span class="go"> (PP in/IN)</span>
<span class="go"> (NP the/DT pound/NN)</span>
<span class="go"> (VP is/VBZ widely/RB expected/VBN to/TO take/VB)</span>
<span class="go"> (NP another/DT sharp/JJ dive/NN)</span>
<span class="go"> if/IN</span>
<span class="go"> ...)</span>
<span class="go">(S</span>
<span class="go"> Chancellor/NNP</span>
<span class="go"> (PP of/IN)</span>
<span class="go"> (NP the/DT Exchequer/NNP)</span>
<span class="go"> ...)</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">conll2002</span><span class="o">.</span><span class="n">sents</span><span class="p">())</span>
<span class="go">[['Sao', 'Paulo', '(', 'Brasil', ')', ',', ...], ['-'], ...]</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">tree</span> <span class="ow">in</span> <span class="n">conll2002</span><span class="o">.</span><span class="n">chunked_sents</span><span class="p">()[:</span><span class="mi">2</span><span class="p">]:</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>
<span class="go">(S</span>
<span class="go"> (LOC Sao/NC Paulo/VMI)</span>
<span class="go"> (/Fpa</span>
<span class="go"> (LOC Brasil/NC)</span>
<span class="go"> )/Fpt</span>
<span class="go"> ...)</span>
<span class="go">(S -/Fg)</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Since the CONLL corpora do not contain paragraph break
information, these readers do not support the <code class="docutils literal notranslate"><span class="pre">para()</span></code> method.)</p>
</div>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>if you call the conll corpora reader methods without any
arguments, they will return the contents of the entire corpus,
<em>including</em> the ‘test’ portions of the corpus.)</p>
</div>
<p>SemCor is a subset of the Brown corpus tagged with WordNet senses and
named entities. Both kinds of lexical items include multiword units,
which are encoded as chunks (senses and part-of-speech tags pertain
to the entire chunk).</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">semcor</span>
<span class="gp">>>> </span><span class="n">semcor</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['The', 'Fulton', 'County', 'Grand', 'Jury', ...]</span>
<span class="gp">>>> </span><span class="n">semcor</span><span class="o">.</span><span class="n">chunks</span><span class="p">()</span>
<span class="go">[['The'], ['Fulton', 'County', 'Grand', 'Jury'], ...]</span>
<span class="gp">>>> </span><span class="n">semcor</span><span class="o">.</span><span class="n">sents</span><span class="p">()</span>
<span class="go">[['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...],</span>
<span class="go">['The', 'jury', 'further', 'said', ...], ...]</span>
<span class="gp">>>> </span><span class="n">semcor</span><span class="o">.</span><span class="n">chunk_sents</span><span class="p">()</span>
<span class="go">[[['The'], ['Fulton', 'County', 'Grand', 'Jury'], ['said'], ...</span>
<span class="go">['.']], [['The'], ['jury'], ['further'], ['said'], ... ['.']], ...]</span>
<span class="gp">>>> </span><span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">semcor</span><span class="o">.</span><span class="n">tagged_chunks</span><span class="p">(</span><span class="n">tag</span><span class="o">=</span><span class="s1">'both'</span><span class="p">)[:</span><span class="mi">3</span><span class="p">]))</span>
<span class="go">['(DT The)', "(Lemma('group.n.01.group') (NE (NNP Fulton County Grand Jury)))", "(Lemma('state.v.01.say') (VB said))"]</span>
<span class="gp">>>> </span><span class="p">[[</span><span class="nb">str</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s</span><span class="p">]</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">semcor</span><span class="o">.</span><span class="n">tagged_sents</span><span class="p">(</span><span class="n">tag</span><span class="o">=</span><span class="s1">'both'</span><span class="p">)[:</span><span class="mi">2</span><span class="p">]]</span>
<span class="go">[['(DT The)', "(Lemma('group.n.01.group') (NE (NNP Fulton County Grand Jury)))", ...</span>
<span class="go"> '(None .)'], ['(DT The)', ... '(None .)']]</span>
</pre></div>
</div>
<p>The IEER corpus is another chunked corpus. This corpus is unusual in
that each corpus item contains multiple documents. (This reflects the
fact that each corpus file contains multiple documents.) The IEER
corpus defines the <cite>parsed_docs</cite> method, which returns the documents
in a given item as <cite>IEERDocument</cite> objects:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">ieer</span>
<span class="gp">>>> </span><span class="n">ieer</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['APW_19980314', 'APW_19980424', 'APW_19980429',</span>
<span class="go"> 'NYT_19980315', 'NYT_19980403', 'NYT_19980407']</span>
<span class="gp">>>> </span><span class="n">docs</span> <span class="o">=</span> <span class="n">ieer</span><span class="o">.</span><span class="n">parsed_docs</span><span class="p">(</span><span class="s1">'APW_19980314'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">docs</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="go"><IEERDocument APW19980314.0391: 'Kenyans protest tax hikes'></span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">docs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">docno</span><span class="p">)</span>
<span class="go">APW19980314.0391</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">docs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">doctype</span><span class="p">)</span>
<span class="go">NEWS STORY</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">docs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">date_time</span><span class="p">)</span>
<span class="go">03/14/1998 10:36:00</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">docs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">headline</span><span class="p">)</span>
<span class="go">(DOCUMENT Kenyans protest tax hikes)</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">docs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
<span class="go">(DOCUMENT</span>
<span class="go"> (LOCATION NAIROBI)</span>
<span class="go"> ,</span>
<span class="go"> (LOCATION Kenya)</span>
<span class="go"> (</span>
<span class="go"> (ORGANIZATION AP)</span>
<span class="go"> )</span>
<span class="go"> _</span>
<span class="go"> (CARDINAL Thousands)</span>
<span class="go"> of</span>
<span class="go"> laborers,</span>
<span class="go"> ...</span>
<span class="go"> on</span>
<span class="go"> (DATE Saturday)</span>
<span class="go"> ...)</span>
</pre></div>
</div>
</section>
<section id="parsed-corpora">
<h4>Parsed Corpora<a class="headerlink" href="#parsed-corpora" title="Link to this heading">¶</a></h4>
<p>The Treebank corpora provide a syntactic parse for each sentence. The
NLTK data package includes a 10% sample of the Penn Treebank (in
<code class="docutils literal notranslate"><span class="pre">treebank</span></code>), as well as the Sinica Treebank (in <code class="docutils literal notranslate"><span class="pre">sinica_treebank</span></code>).</p>
<p>Reading the Penn Treebank (Wall Street Journal sample):</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">treebank</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">treebank</span><span class="o">.</span><span class="n">fileids</span><span class="p">())</span>
<span class="go">['wsj_0001.mrg', 'wsj_0002.mrg', 'wsj_0003.mrg', 'wsj_0004.mrg', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">treebank</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'wsj_0003.mrg'</span><span class="p">))</span>
<span class="go">['A', 'form', 'of', 'asbestos', 'once', 'used', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">treebank</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">(</span><span class="s1">'wsj_0003.mrg'</span><span class="p">))</span>
<span class="go">[('A', 'DT'), ('form', 'NN'), ('of', 'IN'), ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">treebank</span><span class="o">.</span><span class="n">parsed_sents</span><span class="p">(</span><span class="s1">'wsj_0003.mrg'</span><span class="p">)[</span><span class="mi">0</span><span class="p">])</span>
<span class="go">(S</span>
<span class="go"> (S-TPC-1</span>
<span class="go"> (NP-SBJ</span>
<span class="go"> (NP (NP (DT A) (NN form)) (PP (IN of) (NP (NN asbestos))))</span>
<span class="go"> (RRC ...)...)...)</span>
<span class="go"> ...</span>
<span class="go"> (VP (VBD reported) (SBAR (-NONE- 0) (S (-NONE- *T*-1))))</span>
<span class="go"> (. .))</span>
</pre></div>
</div>
<p>If you have access to a full installation of the Penn Treebank, NLTK
can be configured to load it as well. Download the <code class="docutils literal notranslate"><span class="pre">ptb</span></code> package,
and in the directory <code class="docutils literal notranslate"><span class="pre">nltk_data/corpora/ptb</span></code> place the <code class="docutils literal notranslate"><span class="pre">BROWN</span></code>
and <code class="docutils literal notranslate"><span class="pre">WSJ</span></code> directories of the Treebank installation (symlinks work
as well). Then use the <code class="docutils literal notranslate"><span class="pre">ptb</span></code> module instead of <code class="docutils literal notranslate"><span class="pre">treebank</span></code>:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">ptb</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">ptb</span><span class="o">.</span><span class="n">fileids</span><span class="p">())</span>
<span class="go">['BROWN/CF/CF01.MRG', 'BROWN/CF/CF02.MRG', 'BROWN/CF/CF03.MRG', 'BROWN/CF/CF04.MRG', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">ptb</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'WSJ/00/WSJ_0003.MRG'</span><span class="p">))</span>
<span class="go">['A', 'form', 'of', 'asbestos', 'once', 'used', '*', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">ptb</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">(</span><span class="s1">'WSJ/00/WSJ_0003.MRG'</span><span class="p">))</span>
<span class="go">[('A', 'DT'), ('form', 'NN'), ('of', 'IN'), ...]</span>
</pre></div>
</div>
<p>…and so forth, like <code class="docutils literal notranslate"><span class="pre">treebank</span></code> but with extended fileids. Categories
specified in <code class="docutils literal notranslate"><span class="pre">allcats.txt</span></code> can be used to filter by genre; they consist
of <code class="docutils literal notranslate"><span class="pre">news</span></code> (for WSJ articles) and names of the Brown subcategories
(<code class="docutils literal notranslate"><span class="pre">fiction</span></code>, <code class="docutils literal notranslate"><span class="pre">humor</span></code>, <code class="docutils literal notranslate"><span class="pre">romance</span></code>, etc.):</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ptb</span><span class="o">.</span><span class="n">categories</span><span class="p">()</span>
<span class="go">['adventure', 'belles_lettres', 'fiction', 'humor', 'lore', 'mystery', 'news', 'romance', 'science_fiction']</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">ptb</span><span class="o">.</span><span class="n">fileids</span><span class="p">(</span><span class="s1">'news'</span><span class="p">))</span>
<span class="go">['WSJ/00/WSJ_0001.MRG', 'WSJ/00/WSJ_0002.MRG', 'WSJ/00/WSJ_0003.MRG', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">ptb</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="n">categories</span><span class="o">=</span><span class="p">[</span><span class="s1">'humor'</span><span class="p">,</span><span class="s1">'fiction'</span><span class="p">]))</span>
<span class="go">['Thirty-three', 'Scotty', 'did', 'not', 'go', 'back', ...]</span>
</pre></div>
</div>
<p>As PropBank and NomBank depend on the (WSJ portion of the) Penn Treebank,
the modules <code class="docutils literal notranslate"><span class="pre">propbank_ptb</span></code> and <code class="docutils literal notranslate"><span class="pre">nombank_ptb</span></code> are provided for access
to a full PTB installation.</p>
<p>Reading the Sinica Treebank:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">sinica_treebank</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">sinica_treebank</span><span class="o">.</span><span class="n">sents</span><span class="p">())</span>
<span class="go">[['\xe4\xb8\x80'], ['\xe5\x8f\x8b\xe6\x83\x85'], ...]</span>
<span class="gp">>>> </span><span class="n">sinica_treebank</span><span class="o">.</span><span class="n">parsed_sents</span><span class="p">()[</span><span class="mi">25</span><span class="p">]</span>
<span class="go">Tree('S',</span>
<span class="go"> [Tree('NP',</span>
<span class="go"> [Tree('Nba', ['\xe5\x98\x89\xe7\x8f\x8d'])]),</span>
<span class="go"> Tree('V\xe2\x80\xa7\xe5\x9c\xb0',</span>
<span class="go"> [Tree('VA11', ['\xe4\xb8\x8d\xe5\x81\x9c']),</span>
<span class="go"> Tree('DE', ['\xe7\x9a\x84'])]),</span>
<span class="go"> Tree('VA4', ['\xe5\x93\xad\xe6\xb3\xa3'])])</span>
</pre></div>
</div>
<p>Reading the CoNLL 2007 Dependency Treebanks:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">conll2007</span>
<span class="gp">>>> </span><span class="n">conll2007</span><span class="o">.</span><span class="n">sents</span><span class="p">(</span><span class="s1">'esp.train'</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="go">['El', 'aumento', 'del', 'índice', 'de', 'desempleo', ...]</span>
<span class="gp">>>> </span><span class="n">conll2007</span><span class="o">.</span><span class="n">parsed_sents</span><span class="p">(</span><span class="s1">'esp.train'</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="go"><DependencyGraph with 38 nodes></span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">conll2007</span><span class="o">.</span><span class="n">parsed_sents</span><span class="p">(</span><span class="s1">'esp.train'</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">tree</span><span class="p">())</span>
<span class="go">(fortaleció</span>
<span class="go"> (aumento El (del (índice (de (desempleo estadounidense)))))</span>
<span class="go"> hoy</span>
<span class="go"> considerablemente</span>
<span class="go"> (al</span>
<span class="go"> (euro</span>
<span class="go"> (cotizaba</span>
<span class="go"> ,</span>
<span class="go"> que</span>
<span class="go"> (a (15.35 las GMT))</span>
<span class="go"> se</span>
<span class="go"> (en (mercado el (de divisas) (de Fráncfort)))</span>
<span class="go"> (a 0,9452_dólares)</span>
<span class="go"> (frente_a , (0,9349_dólares los (de (mañana esta)))))))</span>
<span class="go"> .)</span>
</pre></div>
</div>
</section>
<section id="word-lists-and-lexicons">
<h4>Word Lists and Lexicons<a class="headerlink" href="#word-lists-and-lexicons" title="Link to this heading">¶</a></h4>
<p>The NLTK data package also includes a number of lexicons and word
lists. These are accessed just like text corpora. The following
examples illustrate the use of the wordlist corpora:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">names</span><span class="p">,</span> <span class="n">stopwords</span><span class="p">,</span> <span class="n">words</span>
<span class="gp">>>> </span><span class="n">words</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['en', 'en-basic']</span>
<span class="gp">>>> </span><span class="n">words</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'en'</span><span class="p">)</span>
<span class="go">['A', 'a', 'aa', 'aal', 'aalii', 'aam', 'Aani', 'aardvark', 'aardwolf', ...]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">stopwords</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['arabic', 'azerbaijani', 'bengali', 'danish', 'dutch', 'english', 'finnish', 'french', ...]</span>
<span class="gp">>>> </span><span class="nb">sorted</span><span class="p">(</span><span class="n">stopwords</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'portuguese'</span><span class="p">))</span>
<span class="go">['a', 'ao', 'aos', 'aquela', 'aquelas', 'aquele', 'aqueles', ...]</span>
<span class="gp">>>> </span><span class="n">names</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['female.txt', 'male.txt']</span>
<span class="gp">>>> </span><span class="n">names</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'male.txt'</span><span class="p">)</span>
<span class="go">['Aamir', 'Aaron', 'Abbey', 'Abbie', 'Abbot', 'Abbott', ...]</span>
<span class="gp">>>> </span><span class="n">names</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'female.txt'</span><span class="p">)</span>
<span class="go">['Abagael', 'Abagail', 'Abbe', 'Abbey', 'Abbi', 'Abbie', ...]</span>
</pre></div>
</div>
<p>The CMU Pronunciation Dictionary corpus contains pronunciation
transcriptions for over 100,000 words. It can be accessed as a list
of entries (where each entry consists of a word, an identifier, and a
transcription) or as a dictionary from words to lists of
transcriptions. Transcriptions are encoded as tuples of phoneme
strings.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">cmudict</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">cmudict</span><span class="o">.</span><span class="n">entries</span><span class="p">()[</span><span class="mi">653</span><span class="p">:</span><span class="mi">659</span><span class="p">])</span>
<span class="go">[('acetate', ['AE1', 'S', 'AH0', 'T', 'EY2', 'T']),</span>
<span class="go">('acetic', ['AH0', 'S', 'EH1', 'T', 'IH0', 'K']),</span>
<span class="go">('acetic', ['AH0', 'S', 'IY1', 'T', 'IH0', 'K']),</span>
<span class="go">('aceto', ['AA0', 'S', 'EH1', 'T', 'OW0']),</span>
<span class="go">('acetochlor', ['AA0', 'S', 'EH1', 'T', 'OW0', 'K', 'L', 'AO2', 'R']),</span>
<span class="go">('acetone', ['AE1', 'S', 'AH0', 'T', 'OW2', 'N'])]</span>
<span class="gp">>>> </span><span class="c1"># Load the entire cmudict corpus into a Python dictionary:</span>
<span class="gp">>>> </span><span class="n">transcr</span> <span class="o">=</span> <span class="n">cmudict</span><span class="o">.</span><span class="n">dict</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">([</span><span class="n">transcr</span><span class="p">[</span><span class="n">w</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="s1">'Natural Language Tool Kit'</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()])</span>
<span class="go">[['N', 'AE1', 'CH', 'ER0', 'AH0', 'L'],</span>
<span class="go"> ['L', 'AE1', 'NG', 'G', 'W', 'AH0', 'JH'],</span>
<span class="go"> ['T', 'UW1', 'L'],</span>
<span class="go"> ['K', 'IH1', 'T']]</span>
</pre></div>
</div>
</section>
<section id="wordnet">
<h4>WordNet<a class="headerlink" href="#wordnet" title="Link to this heading">¶</a></h4>
<p>Please see the separate WordNet howto.</p>
</section>
<section id="framenet">
<h4>FrameNet<a class="headerlink" href="#framenet" title="Link to this heading">¶</a></h4>
<p>Please see the separate FrameNet howto.</p>
</section>
<section id="propbank">
<h4>PropBank<a class="headerlink" href="#propbank" title="Link to this heading">¶</a></h4>
<p>Please see the separate PropBank howto.</p>
</section>
<section id="sentiwordnet">
<h4>SentiWordNet<a class="headerlink" href="#sentiwordnet" title="Link to this heading">¶</a></h4>
<p>Please see the separate SentiWordNet howto.</p>
</section>
<section id="categorized-corpora">
<h4>Categorized Corpora<a class="headerlink" href="#categorized-corpora" title="Link to this heading">¶</a></h4>
<p>Several corpora included with NLTK contain documents that have been categorized for
topic, genre, polarity, etc. In addition to the standard corpus interface, these
corpora provide access to the list of categories and the mapping between the documents
and their categories (in both directions). Access the categories using the <code class="docutils literal notranslate"><span class="pre">categories()</span></code>
method, e.g.:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">brown</span><span class="p">,</span> <span class="n">movie_reviews</span><span class="p">,</span> <span class="n">reuters</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">categories</span><span class="p">()</span>
<span class="go">['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor',</span>
<span class="go">'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']</span>
<span class="gp">>>> </span><span class="n">movie_reviews</span><span class="o">.</span><span class="n">categories</span><span class="p">()</span>
<span class="go">['neg', 'pos']</span>
<span class="gp">>>> </span><span class="n">reuters</span><span class="o">.</span><span class="n">categories</span><span class="p">()</span>
<span class="go">['acq', 'alum', 'barley', 'bop', 'carcass', 'castor-oil', 'cocoa',</span>
<span class="go">'coconut', 'coconut-oil', 'coffee', 'copper', 'copra-cake', 'corn',</span>
<span class="go">'cotton', 'cotton-oil', 'cpi', 'cpu', 'crude', 'dfl', 'dlr', ...]</span>
</pre></div>
</div>
<p>This method has an optional argument that specifies a document or a list
of documents, allowing us to map from (one or more) documents to (one or more) categories:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">categories</span><span class="p">(</span><span class="s1">'ca01'</span><span class="p">)</span>
<span class="go">['news']</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">categories</span><span class="p">([</span><span class="s1">'ca01'</span><span class="p">,</span><span class="s1">'cb01'</span><span class="p">])</span>
<span class="go">['editorial', 'news']</span>
<span class="gp">>>> </span><span class="n">reuters</span><span class="o">.</span><span class="n">categories</span><span class="p">(</span><span class="s1">'training/9865'</span><span class="p">)</span>
<span class="go">['barley', 'corn', 'grain', 'wheat']</span>
<span class="gp">>>> </span><span class="n">reuters</span><span class="o">.</span><span class="n">categories</span><span class="p">([</span><span class="s1">'training/9865'</span><span class="p">,</span> <span class="s1">'training/9880'</span><span class="p">])</span>
<span class="go">['barley', 'corn', 'grain', 'money-fx', 'wheat']</span>
</pre></div>
</div>
<p>We can go back the other way using the optional argument of the <code class="docutils literal notranslate"><span class="pre">fileids()</span></code> method:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">reuters</span><span class="o">.</span><span class="n">fileids</span><span class="p">(</span><span class="s1">'barley'</span><span class="p">)</span>
<span class="go">['test/15618', 'test/15649', 'test/15676', 'test/15728', 'test/15871', ...]</span>
</pre></div>
</div>
<p>Both the <code class="docutils literal notranslate"><span class="pre">categories()</span></code> and <code class="docutils literal notranslate"><span class="pre">fileids()</span></code> methods return a sorted list containing
no duplicates.</p>
<p>In addition to mapping between categories and documents, these corpora permit
direct access to their contents via the categories. Instead of accessing a subset
of a corpus by specifying one or more fileids, we can identify one or more categories, e.g.:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">(</span><span class="n">categories</span><span class="o">=</span><span class="s1">'news'</span><span class="p">)</span>
<span class="go">[('The', 'AT'), ('Fulton', 'NP-TL'), ...]</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">sents</span><span class="p">(</span><span class="n">categories</span><span class="o">=</span><span class="p">[</span><span class="s1">'editorial'</span><span class="p">,</span><span class="s1">'reviews'</span><span class="p">])</span>
<span class="go">[['Assembly', 'session', 'brought', 'much', 'good'], ['The', 'General',</span>
<span class="go">'Assembly', ',', 'which', 'adjourns', 'today', ',', 'has', 'performed',</span>
<span class="go">'in', 'an', 'atmosphere', 'of', 'crisis', 'and', 'struggle', 'from',</span>
<span class="go">'the', 'day', 'it', 'convened', '.'], ...]</span>
</pre></div>
</div>
<p>Note that it is an error to specify both documents and categories.</p>
<p>In the context of a text categorization system, we can easily test if the
category assigned to a document is correct as follows:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span><span class="w"> </span><span class="nf">classify</span><span class="p">(</span><span class="n">doc</span><span class="p">):</span> <span class="k">return</span> <span class="s1">'news'</span> <span class="c1"># Trivial classifier</span>
<span class="gp">>>> </span><span class="n">doc</span> <span class="o">=</span> <span class="s1">'ca01'</span>
<span class="gp">>>> </span><span class="n">classify</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span> <span class="ow">in</span> <span class="n">brown</span><span class="o">.</span><span class="n">categories</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span>
<span class="go">True</span>
</pre></div>
</div>
</section>
<section id="other-corpora">
<h4>Other Corpora<a class="headerlink" href="#other-corpora" title="Link to this heading">¶</a></h4>
<section id="comparative-sentences">
<h5>comparative_sentences<a class="headerlink" href="#comparative-sentences" title="Link to this heading">¶</a></h5>
<p>A list of sentences from various sources, especially reviews and articles. Each
line contains one sentence; sentences were separated by using a sentence tokenizer.
Comparative sentences have been annotated with their type, entities, features and
keywords.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">comparative_sentences</span>
<span class="gp">>>> </span><span class="n">comparison</span> <span class="o">=</span> <span class="n">comparative_sentences</span><span class="o">.</span><span class="n">comparisons</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">comparison</span><span class="o">.</span><span class="n">text</span>
<span class="go">['its', 'fast-forward', 'and', 'rewind', 'work', 'much', 'more', 'smoothly',</span>
<span class="go">'and', 'consistently', 'than', 'those', 'of', 'other', 'models', 'i', "'ve",</span>
<span class="go">'had', '.']</span>
<span class="gp">>>> </span><span class="n">comparison</span><span class="o">.</span><span class="n">entity_2</span>
<span class="go">'models'</span>
<span class="gp">>>> </span><span class="p">(</span><span class="n">comparison</span><span class="o">.</span><span class="n">feature</span><span class="p">,</span> <span class="n">comparison</span><span class="o">.</span><span class="n">keyword</span><span class="p">)</span>
<span class="go">('rewind', 'more')</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">comparative_sentences</span><span class="o">.</span><span class="n">comparisons</span><span class="p">())</span>
<span class="go">853</span>
</pre></div>
</div>
</section>
<section id="opinion-lexicon">
<h5>opinion_lexicon<a class="headerlink" href="#opinion-lexicon" title="Link to this heading">¶</a></h5>
<p>A list of positive and negative opinion words or sentiment words for English.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">opinion_lexicon</span>
<span class="gp">>>> </span><span class="n">opinion_lexicon</span><span class="o">.</span><span class="n">words</span><span class="p">()[:</span><span class="mi">4</span><span class="p">]</span>
<span class="go"> ['2-faced', '2-faces', 'abnormal', 'abolish']</span>
</pre></div>
</div>
<p>The OpinionLexiconCorpusReader also provides shortcuts to retrieve positive/negative
words:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">opinion_lexicon</span><span class="o">.</span><span class="n">negative</span><span class="p">()[:</span><span class="mi">4</span><span class="p">]</span>
<span class="go">['2-faced', '2-faces', 'abnormal', 'abolish']</span>
</pre></div>
</div>
<p>Note that words from <cite>words()</cite> method in opinion_lexicon are sorted by file id,
not alphabetically:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">opinion_lexicon</span><span class="o">.</span><span class="n">words</span><span class="p">()[</span><span class="mi">0</span><span class="p">:</span><span class="mi">10</span><span class="p">]</span>
<span class="go">['2-faced', '2-faces', 'abnormal', 'abolish', 'abominable', 'abominably',</span>
<span class="go">'abominate', 'abomination', 'abort', 'aborted']</span>
<span class="gp">>>> </span><span class="nb">sorted</span><span class="p">(</span><span class="n">opinion_lexicon</span><span class="o">.</span><span class="n">words</span><span class="p">())[</span><span class="mi">0</span><span class="p">:</span><span class="mi">10</span><span class="p">]</span>
<span class="go">['2-faced', '2-faces', 'a+', 'abnormal', 'abolish', 'abominable', 'abominably',</span>
<span class="go">'abominate', 'abomination', 'abort']</span>
</pre></div>
</div>
</section>
<section id="ppattach">
<h5>ppattach<a class="headerlink" href="#ppattach" title="Link to this heading">¶</a></h5>
<p>The Prepositional Phrase Attachment corpus is a corpus of
prepositional phrase attachment decisions. Each instance in the
corpus is encoded as a <code class="docutils literal notranslate"><span class="pre">PPAttachment</span></code> object:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">ppattach</span>
<span class="gp">>>> </span><span class="n">ppattach</span><span class="o">.</span><span class="n">attachments</span><span class="p">(</span><span class="s1">'training'</span><span class="p">)</span>
<span class="go">[PPAttachment(sent='0', verb='join', noun1='board',</span>
<span class="go"> prep='as', noun2='director', attachment='V'),</span>
<span class="go"> PPAttachment(sent='1', verb='is', noun1='chairman',</span>
<span class="go"> prep='of', noun2='N.V.', attachment='N'),</span>
<span class="go"> ...]</span>
<span class="gp">>>> </span><span class="n">inst</span> <span class="o">=</span> <span class="n">ppattach</span><span class="o">.</span><span class="n">attachments</span><span class="p">(</span><span class="s1">'training'</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="gp">>>> </span><span class="p">(</span><span class="n">inst</span><span class="o">.</span><span class="n">sent</span><span class="p">,</span> <span class="n">inst</span><span class="o">.</span><span class="n">verb</span><span class="p">,</span> <span class="n">inst</span><span class="o">.</span><span class="n">noun1</span><span class="p">,</span> <span class="n">inst</span><span class="o">.</span><span class="n">prep</span><span class="p">,</span> <span class="n">inst</span><span class="o">.</span><span class="n">noun2</span><span class="p">)</span>
<span class="go">('0', 'join', 'board', 'as', 'director')</span>
<span class="gp">>>> </span><span class="n">inst</span><span class="o">.</span><span class="n">attachment</span>
<span class="go">'V'</span>
</pre></div>
</div>
</section>
<section id="product-reviews-1-and-product-reviews-2">
<h5>product_reviews_1 and product_reviews_2<a class="headerlink" href="#product-reviews-1-and-product-reviews-2" title="Link to this heading">¶</a></h5>
<p>These two datasets respectively contain annotated customer reviews of 5 and 9
products from amazon.com.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">product_reviews_1</span>
<span class="gp">>>> </span><span class="n">camera_reviews</span> <span class="o">=</span> <span class="n">product_reviews_1</span><span class="o">.</span><span class="n">reviews</span><span class="p">(</span><span class="s1">'Canon_G3.txt'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">review</span> <span class="o">=</span> <span class="n">camera_reviews</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">review</span><span class="o">.</span><span class="n">sents</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="go">['i', 'recently', 'purchased', 'the', 'canon', 'powershot', 'g3', 'and', 'am',</span>
<span class="go">'extremely', 'satisfied', 'with', 'the', 'purchase', '.']</span>
<span class="gp">>>> </span><span class="n">review</span><span class="o">.</span><span class="n">features</span><span class="p">()</span>
<span class="go">[('canon powershot g3', '+3'), ('use', '+2'), ('picture', '+2'),</span>
<span class="go">('picture quality', '+1'), ('picture quality', '+1'), ('camera', '+2'),</span>
<span class="go">('use', '+2'), ('feature', '+1'), ('picture quality', '+3'), ('use', '+1'),</span>
<span class="go">('option', '+1')]</span>
</pre></div>
</div>
<p>It is also possible to reach the same information directly from the stream:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">product_reviews_1</span><span class="o">.</span><span class="n">features</span><span class="p">(</span><span class="s1">'Canon_G3.txt'</span><span class="p">)</span>
<span class="go">[('canon powershot g3', '+3'), ('use', '+2'), ...]</span>
</pre></div>
</div>
<p>We can compute stats for specific product features:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">n_reviews</span> <span class="o">=</span> <span class="nb">len</span><span class="p">([(</span><span class="n">feat</span><span class="p">,</span><span class="n">score</span><span class="p">)</span> <span class="k">for</span> <span class="p">(</span><span class="n">feat</span><span class="p">,</span><span class="n">score</span><span class="p">)</span> <span class="ow">in</span> <span class="n">product_reviews_1</span><span class="o">.</span><span class="n">features</span><span class="p">(</span><span class="s1">'Canon_G3.txt'</span><span class="p">)</span> <span class="k">if</span> <span class="n">feat</span><span class="o">==</span><span class="s1">'picture'</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">tot</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">([</span><span class="nb">int</span><span class="p">(</span><span class="n">score</span><span class="p">)</span> <span class="k">for</span> <span class="p">(</span><span class="n">feat</span><span class="p">,</span><span class="n">score</span><span class="p">)</span> <span class="ow">in</span> <span class="n">product_reviews_1</span><span class="o">.</span><span class="n">features</span><span class="p">(</span><span class="s1">'Canon_G3.txt'</span><span class="p">)</span> <span class="k">if</span> <span class="n">feat</span><span class="o">==</span><span class="s1">'picture'</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">mean</span> <span class="o">=</span> <span class="n">tot</span> <span class="o">/</span> <span class="n">n_reviews</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">n_reviews</span><span class="p">,</span> <span class="n">tot</span><span class="p">,</span> <span class="n">mean</span><span class="p">)</span>
<span class="go">15 24 1.6</span>
</pre></div>
</div>
</section>
<section id="pros-cons">
<h5>pros_cons<a class="headerlink" href="#pros-cons" title="Link to this heading">¶</a></h5>
<p>A list of pros/cons sentences for determining context (aspect) dependent
sentiment words, which are then applied to sentiment analysis of comparative
sentences.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">pros_cons</span>
<span class="gp">>>> </span><span class="n">pros_cons</span><span class="o">.</span><span class="n">sents</span><span class="p">(</span><span class="n">categories</span><span class="o">=</span><span class="s1">'Cons'</span><span class="p">)</span>
<span class="go">[['East', 'batteries', '!', 'On', '-', 'off', 'switch', 'too', 'easy',</span>
<span class="go">'to', 'maneuver', '.'], ['Eats', '...', 'no', ',', 'GULPS', 'batteries'],</span>
<span class="go">...]</span>
<span class="gp">>>> </span><span class="n">pros_cons</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'IntegratedPros.txt'</span><span class="p">)</span>
<span class="go">['Easy', 'to', 'use', ',', 'economical', '!', ...]</span>
</pre></div>
</div>
</section>
<section id="semcor">
<h5>semcor<a class="headerlink" href="#semcor" title="Link to this heading">¶</a></h5>
<p>The Brown Corpus, annotated with WordNet senses.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">semcor</span>
<span class="gp">>>> </span><span class="n">semcor</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'brown2/tagfiles/br-n12.xml'</span><span class="p">)</span>
<span class="go">['When', 'several', 'minutes', 'had', 'passed', ...]</span>
</pre></div>
</div>
</section>
<section id="senseval">
<h5>senseval<a class="headerlink" href="#senseval" title="Link to this heading">¶</a></h5>
<p>The Senseval 2 corpus is a word sense disambiguation corpus. Each
item in the corpus corresponds to a single ambiguous word. For each
of these words, the corpus contains a list of instances, corresponding
to occurrences of that word. Each instance provides the word; a list
of word senses that apply to the word occurrence; and the word’s
context.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">senseval</span>
<span class="gp">>>> </span><span class="n">senseval</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['hard.pos', 'interest.pos', 'line.pos', 'serve.pos']</span>
<span class="gp">>>> </span><span class="n">senseval</span><span class="o">.</span><span class="n">instances</span><span class="p">(</span><span class="s1">'hard.pos'</span><span class="p">)</span>
<span class="gp">...</span>
<span class="go">[SensevalInstance(word='hard-a',</span>
<span class="go"> position=20,</span>
<span class="go"> context=[('``', '``'), ('he', 'PRP'), ...('hard', 'JJ'), ...],</span>
<span class="go"> senses=('HARD1',)),</span>
<span class="go"> SensevalInstance(word='hard-a',</span>
<span class="go"> position=10,</span>
<span class="go"> context=[('clever', 'NNP'), ...('hard', 'JJ'), ('time', 'NN'), ...],</span>
<span class="go"> senses=('HARD1',)), ...]</span>
</pre></div>
</div>
<p>The following code looks at instances of the word ‘interest’, and
displays their local context (2 words on each side) and word sense(s):</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">inst</span> <span class="ow">in</span> <span class="n">senseval</span><span class="o">.</span><span class="n">instances</span><span class="p">(</span><span class="s1">'interest.pos'</span><span class="p">)[:</span><span class="mi">10</span><span class="p">]:</span>
<span class="gp">... </span> <span class="n">p</span> <span class="o">=</span> <span class="n">inst</span><span class="o">.</span><span class="n">position</span>
<span class="gp">... </span> <span class="n">left</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">w</span> <span class="k">for</span> <span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="n">t</span><span class="p">)</span> <span class="ow">in</span> <span class="n">inst</span><span class="o">.</span><span class="n">context</span><span class="p">[</span><span class="n">p</span><span class="o">-</span><span class="mi">2</span><span class="p">:</span><span class="n">p</span><span class="p">])</span>
<span class="gp">... </span> <span class="n">word</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">w</span> <span class="k">for</span> <span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="n">t</span><span class="p">)</span> <span class="ow">in</span> <span class="n">inst</span><span class="o">.</span><span class="n">context</span><span class="p">[</span><span class="n">p</span><span class="p">:</span><span class="n">p</span><span class="o">+</span><span class="mi">1</span><span class="p">])</span>
<span class="gp">... </span> <span class="n">right</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">w</span> <span class="k">for</span> <span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="n">t</span><span class="p">)</span> <span class="ow">in</span> <span class="n">inst</span><span class="o">.</span><span class="n">context</span><span class="p">[</span><span class="n">p</span><span class="o">+</span><span class="mi">1</span><span class="p">:</span><span class="n">p</span><span class="o">+</span><span class="mi">3</span><span class="p">])</span>
<span class="gp">... </span> <span class="n">senses</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">inst</span><span class="o">.</span><span class="n">senses</span><span class="p">)</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">%20s</span><span class="s1"> |</span><span class="si">%10s</span><span class="s1"> | </span><span class="si">%-15s</span><span class="s1"> -> </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">left</span><span class="p">,</span> <span class="n">word</span><span class="p">,</span> <span class="n">right</span><span class="p">,</span> <span class="n">senses</span><span class="p">))</span>
<span class="go"> declines in | interest | rates . -> interest_6</span>
<span class="go"> indicate declining | interest | rates because -> interest_6</span>
<span class="go"> in short-term | interest | rates . -> interest_6</span>
<span class="go"> 4 % | interest | in this -> interest_5</span>
<span class="go"> company with | interests | in the -> interest_5</span>
<span class="go"> , plus | interest | . -> interest_6</span>
<span class="go"> set the | interest | rate on -> interest_6</span>
<span class="go"> 's own | interest | , prompted -> interest_4</span>
<span class="go"> principal and | interest | is the -> interest_6</span>
<span class="go"> increase its | interest | to 70 -> interest_5</span>
</pre></div>
</div>
</section>
<section id="sentence-polarity">
<h5>sentence_polarity<a class="headerlink" href="#sentence-polarity" title="Link to this heading">¶</a></h5>
<p>The Sentence Polarity dataset contains 5331 positive and 5331 negative processed
sentences.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">sentence_polarity</span>
<span class="gp">>>> </span><span class="n">sentence_polarity</span><span class="o">.</span><span class="n">sents</span><span class="p">()</span>
<span class="go">[['simplistic', ',', 'silly', 'and', 'tedious', '.'], ["it's", 'so', 'laddish',</span>
<span class="go">'and', 'juvenile', ',', 'only', 'teenage', 'boys', 'could', 'possibly', 'find',</span>
<span class="go">'it', 'funny', '.'], ...]</span>
<span class="gp">>>> </span><span class="n">sentence_polarity</span><span class="o">.</span><span class="n">categories</span><span class="p">()</span>
<span class="go">['neg', 'pos']</span>
<span class="gp">>>> </span><span class="n">sentence_polarity</span><span class="o">.</span><span class="n">sents</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span>
<span class="go">["it's", 'so', 'laddish', 'and', 'juvenile', ',', 'only', 'teenage', 'boys',</span>
<span class="go">'could', 'possibly', 'find', 'it', 'funny', '.']</span>
</pre></div>
</div>
</section>
<section id="shakespeare">
<h5>shakespeare<a class="headerlink" href="#shakespeare" title="Link to this heading">¶</a></h5>
<p>The Shakespeare corpus contains a set of Shakespeare plays, formatted
as XML files. These corpora are returned as ElementTree objects:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">shakespeare</span>
<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">xml.etree</span><span class="w"> </span><span class="kn">import</span> <span class="n">ElementTree</span>
<span class="gp">>>> </span><span class="n">shakespeare</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['a_and_c.xml', 'dream.xml', 'hamlet.xml', 'j_caesar.xml', ...]</span>
<span class="gp">>>> </span><span class="n">play</span> <span class="o">=</span> <span class="n">shakespeare</span><span class="o">.</span><span class="n">xml</span><span class="p">(</span><span class="s1">'dream.xml'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">play</span><span class="p">)</span>
<span class="go"><Element 'PLAY' at ...></span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1">: </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">play</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">tag</span><span class="p">,</span> <span class="n">play</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="p">))</span>
<span class="go">TITLE: A Midsummer Night's Dream</span>
<span class="gp">>>> </span><span class="n">personae</span> <span class="o">=</span> <span class="p">[</span><span class="n">persona</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">persona</span> <span class="ow">in</span>
<span class="gp">... </span> <span class="n">play</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s1">'PERSONAE/PERSONA'</span><span class="p">)]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">personae</span><span class="p">)</span>
<span class="go">['THESEUS, Duke of Athens.', 'EGEUS, father to Hermia.', ...]</span>
<span class="gp">>>> </span><span class="c1"># Find and print speakers not listed as personae</span>
<span class="gp">>>> </span><span class="n">names</span> <span class="o">=</span> <span class="p">[</span><span class="n">persona</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">','</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">persona</span> <span class="ow">in</span> <span class="n">personae</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">speakers</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">speaker</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">speaker</span> <span class="ow">in</span>
<span class="gp">... </span> <span class="n">play</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s1">'*/*/*/SPEAKER'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">speakers</span><span class="o">.</span><span class="n">difference</span><span class="p">(</span><span class="n">names</span><span class="p">)))</span>
<span class="go">['ALL', 'COBWEB', 'DEMETRIUS', 'Fairy', 'HERNIA', 'LYSANDER',</span>
<span class="go"> 'Lion', 'MOTH', 'MUSTARDSEED', 'Moonshine', 'PEASEBLOSSOM',</span>
<span class="go"> 'Prologue', 'Pyramus', 'Thisbe', 'Wall']</span>
</pre></div>
</div>
</section>
<section id="subjectivity">
<h5>subjectivity<a class="headerlink" href="#subjectivity" title="Link to this heading">¶</a></h5>
<p>The Subjectivity Dataset contains 5000 subjective and 5000 objective processed
sentences.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">subjectivity</span>
<span class="gp">>>> </span><span class="n">subjectivity</span><span class="o">.</span><span class="n">categories</span><span class="p">()</span>
<span class="go">['obj', 'subj']</span>
<span class="gp">>>> </span><span class="n">subjectivity</span><span class="o">.</span><span class="n">sents</span><span class="p">()[</span><span class="mi">23</span><span class="p">]</span>
<span class="go">['television', 'made', 'him', 'famous', ',', 'but', 'his', 'biggest', 'hits',</span>
<span class="go">'happened', 'off', 'screen', '.']</span>
<span class="gp">>>> </span><span class="n">subjectivity</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="n">categories</span><span class="o">=</span><span class="s1">'subj'</span><span class="p">)</span>
<span class="go">['smart', 'and', 'alert', ',', 'thirteen', ...]</span>
</pre></div>
</div>
</section>
<section id="toolbox">
<h5>toolbox<a class="headerlink" href="#toolbox" title="Link to this heading">¶</a></h5>
<p>The Toolbox corpus distributed with NLTK contains a sample lexicon and
several sample texts from the Rotokas language. The Toolbox corpus
reader returns Toolbox files as XML ElementTree objects. The
following example loads the Rotokas dictionary, and figures out the
distribution of part-of-speech tags for reduplicated words.</p>
<p>This example displays some records from a Rotokas text:</p>
</section>
<section id="timit">
<h5>timit<a class="headerlink" href="#timit" title="Link to this heading">¶</a></h5>
<p>The NLTK data package includes a fragment of the TIMIT
Acoustic-Phonetic Continuous Speech Corpus. This corpus is broken
down into small speech samples, each of which is available as a wave
file, a phonetic transcription, and a tokenized word list.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">timit</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">utteranceids</span><span class="p">())</span>
<span class="go">['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466',</span>
<span class="go">'dr1-fvmh0/si2096', 'dr1-fvmh0/si836', 'dr1-fvmh0/sx116',</span>
<span class="go">'dr1-fvmh0/sx206', 'dr1-fvmh0/sx26', 'dr1-fvmh0/sx296', ...]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">item</span> <span class="o">=</span> <span class="n">timit</span><span class="o">.</span><span class="n">utteranceids</span><span class="p">()[</span><span class="mi">5</span><span class="p">]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">phones</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="go">['h#', 'k', 'l', 'ae', 's', 'pcl', 'p', 'dh', 'ax',</span>
<span class="go"> 's', 'kcl', 'k', 'r', 'ux', 'ix', 'nx', 'y', 'ax',</span>
<span class="go"> 'l', 'eh', 'f', 'tcl', 't', 'hh', 'ae', 'n', 'dcl',</span>
<span class="go"> 'd', 'h#']</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="go">['clasp', 'the', 'screw', 'in', 'your', 'left', 'hand']</span>
<span class="gp">>>> </span><span class="n">timit</span><span class="o">.</span><span class="n">play</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
</pre></div>
</div>
<p>The corpus reader can combine the word segmentation information with
the phonemes to produce a single tree structure:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">tree</span> <span class="ow">in</span> <span class="n">timit</span><span class="o">.</span><span class="n">phone_trees</span><span class="p">(</span><span class="n">item</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>
<span class="go">(S</span>
<span class="go"> h#</span>
<span class="go"> (clasp k l ae s pcl p)</span>
<span class="go"> (the dh ax)</span>
<span class="go"> (screw s kcl k r ux)</span>
<span class="go"> (in ix nx)</span>
<span class="go"> (your y ax)</span>
<span class="go"> (left l eh f tcl t)</span>
<span class="go"> (hand hh ae n dcl d)</span>
<span class="go"> h#)</span>
</pre></div>
</div>
<p>The start time and stop time of each phoneme, word, and sentence are
also available:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">phone_times</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="go">[('h#', 0, 2190), ('k', 2190, 3430), ('l', 3430, 4326), ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">word_times</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="go">[('clasp', 2190, 8804), ('the', 8804, 9734), ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">sent_times</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="go">[('Clasp the screw in your left hand.', 0, 32154)]</span>
</pre></div>
</div>
<p>We can use these times to play selected pieces of a speech sample:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">timit</span><span class="o">.</span><span class="n">play</span><span class="p">(</span><span class="n">item</span><span class="p">,</span> <span class="mi">2190</span><span class="p">,</span> <span class="mi">8804</span><span class="p">)</span> <span class="c1"># 'clasp'</span>
</pre></div>
</div>
<p>The corpus reader can also be queried for information about the
speaker and sentence identifier for a given speech sample:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">spkrid</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="go">dr1-fvmh0</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">sentid</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="go">sx116</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">spkrinfo</span><span class="p">(</span><span class="n">timit</span><span class="o">.</span><span class="n">spkrid</span><span class="p">(</span><span class="n">item</span><span class="p">)))</span>
<span class="go">SpeakerInfo(id='VMH0',</span>
<span class="go"> sex='F',</span>
<span class="go"> dr='1',</span>
<span class="go"> use='TRN',</span>
<span class="go"> recdate='03/11/86',</span>
<span class="go"> birthdate='01/08/60',</span>
<span class="go"> ht='5\'05"',</span>
<span class="go"> race='WHT',</span>
<span class="go"> edu='BS',</span>
<span class="go"> comments='BEST NEW ENGLAND ACCENT SO FAR')</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># List the speech samples from the same speaker:</span>
<span class="gp">>>> </span><span class="n">timit</span><span class="o">.</span><span class="n">utteranceids</span><span class="p">(</span><span class="n">spkrid</span><span class="o">=</span><span class="n">timit</span><span class="o">.</span><span class="n">spkrid</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="go">['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466', ...]</span>
</pre></div>
</div>
</section>
<section id="twitter-samples">
<h5>twitter_samples<a class="headerlink" href="#twitter-samples" title="Link to this heading">¶</a></h5>
<p>Twitter is well-known microblog service that allows public data to be
collected via APIs. NLTK’s twitter corpus currently contains a sample of 20k Tweets
retrieved from the Twitter Streaming API.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">twitter_samples</span>
<span class="gp">>>> </span><span class="n">twitter_samples</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['negative_tweets.json', 'positive_tweets.json', 'tweets.20150430-223406.json']</span>
</pre></div>
</div>
<p>We follow standard practice in storing full Tweets as line-separated
JSON. These data structures can be accessed via <cite>tweets.docs()</cite>. However, in general it
is more practical to focus just on the text field of the Tweets, which
are accessed via the <cite>strings()</cite> method.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">twitter_samples</span><span class="o">.</span><span class="n">strings</span><span class="p">(</span><span class="s1">'tweets.20150430-223406.json'</span><span class="p">)[:</span><span class="mi">5</span><span class="p">]</span>
<span class="go">['RT @KirkKus: Indirect cost of the UK being in the EU is estimated to be costing Britain \xa3170 billion per year! #BetterOffOut #UKIP', ...]</span>
</pre></div>
</div>
<p>The default tokenizer for Tweets is specialised for ‘casual’ text, and
the <cite>tokenized()</cite> method returns a list of lists of tokens.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">twitter_samples</span><span class="o">.</span><span class="n">tokenized</span><span class="p">(</span><span class="s1">'tweets.20150430-223406.json'</span><span class="p">)[:</span><span class="mi">5</span><span class="p">]</span>
<span class="go">[['RT', '@KirkKus', ':', 'Indirect', 'cost', 'of', 'the', 'UK', 'being', 'in', ...],</span>
<span class="go"> ['VIDEO', ':', 'Sturgeon', 'on', 'post-election', 'deals', 'http://t.co/BTJwrpbmOY'], ...]</span>
</pre></div>
</div>
</section>
<section id="rte">
<h5>rte<a class="headerlink" href="#rte" title="Link to this heading">¶</a></h5>
<p>The RTE (Recognizing Textual Entailment) corpus was derived from the
RTE1, RTE2 and RTE3 datasets (dev and test data), and consists of a
list of XML-formatted ‘text’/’hypothesis’ pairs.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">rte</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">rte</span><span class="o">.</span><span class="n">fileids</span><span class="p">())</span>
<span class="go">['rte1_dev.xml', 'rte1_test.xml', 'rte2_dev.xml', ..., 'rte3_test.xml']</span>
<span class="gp">>>> </span><span class="n">rtepairs</span> <span class="o">=</span> <span class="n">rte</span><span class="o">.</span><span class="n">pairs</span><span class="p">([</span><span class="s1">'rte2_test.xml'</span><span class="p">,</span> <span class="s1">'rte3_test.xml'</span><span class="p">])</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">rtepairs</span><span class="p">)</span>
<span class="go">[<RTEPair: gid=2-8>, <RTEPair: gid=2-9>, <RTEPair: gid=2-15>, ...]</span>
</pre></div>
</div>
<p>In the gold standard test sets, each pair is labeled according to
whether or not the text ‘entails’ the hypothesis; the
entailment value is mapped to an integer 1 (True) or 0 (False).</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">rtepairs</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
<span class="go"><RTEPair: gid=2-23></span>
<span class="gp">>>> </span><span class="n">rtepairs</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">text</span>
<span class="go">'His wife Strida won a seat in parliament after forging an alliance</span>
<span class="go">with the main anti-Syrian coalition in the recent election.'</span>
<span class="gp">>>> </span><span class="n">rtepairs</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">hyp</span>
<span class="go">'Strida elected to parliament.'</span>
<span class="gp">>>> </span><span class="n">rtepairs</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">value</span>
<span class="go">1</span>
</pre></div>
</div>
<p>The RTE corpus also supports an <code class="docutils literal notranslate"><span class="pre">xml()</span></code> method which produces ElementTrees.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">xmltree</span> <span class="o">=</span> <span class="n">rte</span><span class="o">.</span><span class="n">xml</span><span class="p">(</span><span class="s1">'rte3_dev.xml'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">xmltree</span>
<span class="go"><Element entailment-corpus at ...></span>
<span class="gp">>>> </span><span class="n">xmltree</span><span class="p">[</span><span class="mi">7</span><span class="p">]</span><span class="o">.</span><span class="n">findtext</span><span class="p">(</span><span class="s1">'t'</span><span class="p">)</span>
<span class="go">"Mrs. Bush's approval ratings have remained very high, above 80%,</span>
<span class="go">even as her husband's have recently dropped below 50%."</span>
</pre></div>
</div>
</section>
<section id="verbnet">
<h5>verbnet<a class="headerlink" href="#verbnet" title="Link to this heading">¶</a></h5>
<p>The VerbNet corpus is a lexicon that divides verbs into classes, based
on their syntax-semantics linking behavior. The basic elements in the
lexicon are verb lemmas, such as ‘abandon’ and ‘accept’, and verb
classes, which have identifiers such as ‘remove-10.1’ and
‘admire-31.2-1’. These class identifiers consist of a representative
verb selected from the class, followed by a numerical identifier. The
list of verb lemmas, and the list of class identifiers, can be
retrieved with the following methods:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">verbnet</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">lemmas</span><span class="p">()[</span><span class="mi">20</span><span class="p">:</span><span class="mi">25</span><span class="p">]</span>
<span class="go">['accelerate', 'accept', 'acclaim', 'accompany', 'accrue']</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">classids</span><span class="p">()[:</span><span class="mi">5</span><span class="p">]</span>
<span class="go">['accompany-51.7', 'admire-31.2', 'admire-31.2-1', 'admit-65', 'adopt-93']</span>
</pre></div>
</div>
<p>The <cite>classids()</cite> method may also be used to retrieve the classes that
a given lemma belongs to:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">classids</span><span class="p">(</span><span class="s1">'accept'</span><span class="p">)</span>
<span class="go">['approve-77', 'characterize-29.2-1-1', 'obtain-13.5.2']</span>
</pre></div>
</div>
<p>The <cite>classids()</cite> method may additionally be used to retrieve all classes
within verbnet if nothing is passed:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">classids</span><span class="p">()</span>
<span class="go">['accompany-51.7', 'admire-31.2', 'admire-31.2-1', 'admit-65', 'adopt-93', 'advise-37.9', 'advise-37.9-1', 'allow-64', 'amalgamate-22.2', 'amalgamate-22.2-1', 'amalgamate-22.2-1-1', 'amalgamate-22.2-2', 'amalgamate-22.2-2-1', 'amalgamate-22.2-3', 'amalgamate-22.2-3-1', 'amalgamate-22.2-3-1-1', 'amalgamate-22.2-3-2', 'amuse-31.1', 'animal_sounds-38', 'appeal-31.4', 'appeal-31.4-1', 'appeal-31.4-2', 'appeal-31.4-3', 'appear-48.1.1', 'appoint-29.1', 'approve-77', 'assessment-34', 'assuming_position-50', 'avoid-52', 'banish-10.2', 'battle-36.4', 'battle-36.4-1', 'begin-55.1', 'begin-55.1-1', 'being_dressed-41.3.3', 'bend-45.2', 'berry-13.7', 'bill-54.5', 'body_internal_motion-49', 'body_internal_states-40.6', 'braid-41.2.2', 'break-45.1', 'breathe-40.1.2', 'breathe-40.1.2-1', 'bring-11.3', 'bring-11.3-1', 'build-26.1', 'build-26.1-1', 'bulge-47.5.3', 'bump-18.4', 'bump-18.4-1', 'butter-9.9', 'calibratable_cos-45.6', 'calibratable_cos-45.6-1', 'calve-28', 'captain-29.8', 'captain-29.8-1', 'captain-29.8-1-1', 'care-88', 'care-88-1', 'carry-11.4', 'carry-11.4-1', 'carry-11.4-1-1', 'carve-21.2', 'carve-21.2-1', 'carve-21.2-2', 'change_bodily_state-40.8.4', 'characterize-29.2', 'characterize-29.2-1', 'characterize-29.2-1-1', 'characterize-29.2-1-2', 'chase-51.6', 'cheat-10.6', 'cheat-10.6-1', 'cheat-10.6-1-1', 'chew-39.2', 'chew-39.2-1', 'chew-39.2-2', 'chit_chat-37.6', 'clear-10.3', 'clear-10.3-1', 'cling-22.5', 'coil-9.6', 'coil-9.6-1', 'coloring-24', 'complain-37.8', 'complete-55.2', 'concealment-16', 'concealment-16-1', 'confess-37.10', 'confine-92', 'confine-92-1', 'conjecture-29.5', 'conjecture-29.5-1', 'conjecture-29.5-2', 'consider-29.9', 'consider-29.9-1', 'consider-29.9-1-1', 'consider-29.9-1-1-1', 'consider-29.9-2', 'conspire-71', 'consume-66', 'consume-66-1', 'contiguous_location-47.8', 'contiguous_location-47.8-1', 'contiguous_location-47.8-2', 'continue-55.3', 'contribute-13.2', 'contribute-13.2-1', 'contribute-13.2-1-1', 'contribute-13.2-1-1-1', 'contribute-13.2-2', 'contribute-13.2-2-1', 'convert-26.6.2', 'convert-26.6.2-1', 'cooking-45.3', 'cooperate-73', 'cooperate-73-1', 'cooperate-73-2', 'cooperate-73-3', 'cope-83', 'cope-83-1', 'cope-83-1-1', 'correlate-86', 'correspond-36.1', 'correspond-36.1-1', 'correspond-36.1-1-1', 'cost-54.2', 'crane-40.3.2', 'create-26.4', 'create-26.4-1', 'curtsey-40.3.3', 'cut-21.1', 'cut-21.1-1', 'debone-10.8', 'declare-29.4', 'declare-29.4-1', 'declare-29.4-1-1', 'declare-29.4-1-1-1', 'declare-29.4-1-1-2', 'declare-29.4-1-1-3', 'declare-29.4-2', 'dedicate-79', 'defend-85', 'destroy-44', 'devour-39.4', 'devour-39.4-1', 'devour-39.4-2', 'differ-23.4', 'dine-39.5', 'disappearance-48.2', 'disassemble-23.3', 'discover-84', 'discover-84-1', 'discover-84-1-1', 'dress-41.1.1', 'dressing_well-41.3.2', 'drive-11.5', 'drive-11.5-1', 'dub-29.3', 'dub-29.3-1', 'eat-39.1', 'eat-39.1-1', 'eat-39.1-2', 'enforce-63', 'engender-27', 'entity_specific_cos-45.5', 'entity_specific_modes_being-47.2', 'equip-13.4.2', 'equip-13.4.2-1', 'equip-13.4.2-1-1', 'escape-51.1', 'escape-51.1-1', 'escape-51.1-2', 'escape-51.1-2-1', 'exceed-90', 'exchange-13.6', 'exchange-13.6-1', 'exchange-13.6-1-1', 'exhale-40.1.3', 'exhale-40.1.3-1', 'exhale-40.1.3-2', 'exist-47.1', 'exist-47.1-1', 'exist-47.1-1-1', 'feeding-39.7', 'ferret-35.6', 'fill-9.8', 'fill-9.8-1', 'fit-54.3', 'flinch-40.5', 'floss-41.2.1', 'focus-87', 'forbid-67', 'force-59', 'force-59-1', 'free-80', 'free-80-1', 'fulfilling-13.4.1', 'fulfilling-13.4.1-1', 'fulfilling-13.4.1-2', 'funnel-9.3', 'funnel-9.3-1', 'funnel-9.3-2', 'funnel-9.3-2-1', 'future_having-13.3', 'get-13.5.1', 'get-13.5.1-1', 'give-13.1', 'give-13.1-1', 'gobble-39.3', 'gobble-39.3-1', 'gobble-39.3-2', 'gorge-39.6', 'groom-41.1.2', 'grow-26.2', 'help-72', 'help-72-1', 'herd-47.5.2', 'hiccup-40.1.1', 'hit-18.1', 'hit-18.1-1', 'hold-15.1', 'hold-15.1-1', 'hunt-35.1', 'hurt-40.8.3', 'hurt-40.8.3-1', 'hurt-40.8.3-1-1', 'hurt-40.8.3-2', 'illustrate-25.3', 'image_impression-25.1', 'indicate-78', 'indicate-78-1', 'indicate-78-1-1', 'inquire-37.1.2', 'instr_communication-37.4', 'investigate-35.4', 'judgement-33', 'keep-15.2', 'knead-26.5', 'learn-14', 'learn-14-1', 'learn-14-2', 'learn-14-2-1', 'leave-51.2', 'leave-51.2-1', 'lecture-37.11', 'lecture-37.11-1', 'lecture-37.11-1-1', 'lecture-37.11-2', 'light_emission-43.1', 'limit-76', 'linger-53.1', 'linger-53.1-1', 'lodge-46', 'long-32.2', 'long-32.2-1', 'long-32.2-2', 'manner_speaking-37.3', 'marry-36.2', 'marvel-31.3', 'marvel-31.3-1', 'marvel-31.3-2', 'marvel-31.3-3', 'marvel-31.3-4', 'marvel-31.3-5', 'marvel-31.3-6', 'marvel-31.3-7', 'marvel-31.3-8', 'marvel-31.3-9', 'masquerade-29.6', 'masquerade-29.6-1', 'masquerade-29.6-2', 'matter-91', 'meander-47.7', 'meet-36.3', 'meet-36.3-1', 'meet-36.3-2', 'mine-10.9', 'mix-22.1', 'mix-22.1-1', 'mix-22.1-1-1', 'mix-22.1-2', 'mix-22.1-2-1', 'modes_of_being_with_motion-47.3', 'murder-42.1', 'murder-42.1-1', 'neglect-75', 'neglect-75-1', 'neglect-75-1-1', 'neglect-75-2', 'nonvehicle-51.4.2', 'nonverbal_expression-40.2', 'obtain-13.5.2', 'obtain-13.5.2-1', 'occurrence-48.3', 'order-60', 'order-60-1', 'orphan-29.7', 'other_cos-45.4', 'pain-40.8.1', 'pay-68', 'peer-30.3', 'pelt-17.2', 'performance-26.7', 'performance-26.7-1', 'performance-26.7-1-1', 'performance-26.7-2', 'performance-26.7-2-1', 'pit-10.7', 'pocket-9.10', 'pocket-9.10-1', 'poison-42.2', 'poke-19', 'pour-9.5', 'preparing-26.3', 'preparing-26.3-1', 'preparing-26.3-2', 'price-54.4', 'push-12', 'push-12-1', 'push-12-1-1', 'put-9.1', 'put-9.1-1', 'put-9.1-2', 'put_direction-9.4', 'put_spatial-9.2', 'put_spatial-9.2-1', 'reach-51.8', 'reflexive_appearance-48.1.2', 'refrain-69', 'register-54.1', 'rely-70', 'remove-10.1', 'risk-94', 'risk-94-1', 'roll-51.3.1', 'rummage-35.5', 'run-51.3.2', 'rush-53.2', 'say-37.7', 'say-37.7-1', 'say-37.7-1-1', 'say-37.7-2', 'scribble-25.2', 'search-35.2', 'see-30.1', 'see-30.1-1', 'see-30.1-1-1', 'send-11.1', 'send-11.1-1', 'separate-23.1', 'separate-23.1-1', 'separate-23.1-2', 'settle-89', 'shake-22.3', 'shake-22.3-1', 'shake-22.3-1-1', 'shake-22.3-2', 'shake-22.3-2-1', 'sight-30.2', 'simple_dressing-41.3.1', 'slide-11.2', 'slide-11.2-1-1', 'smell_emission-43.3', 'snooze-40.4', 'sound_emission-43.2', 'sound_existence-47.4', 'spank-18.3', 'spatial_configuration-47.6', 'split-23.2', 'spray-9.7', 'spray-9.7-1', 'spray-9.7-1-1', 'spray-9.7-2', 'stalk-35.3', 'steal-10.5', 'stimulus_subject-30.4', 'stop-55.4', 'stop-55.4-1', 'substance_emission-43.4', 'succeed-74', 'succeed-74-1', 'succeed-74-1-1', 'succeed-74-2', 'suffocate-40.7', 'suspect-81', 'swarm-47.5.1', 'swarm-47.5.1-1', 'swarm-47.5.1-2', 'swarm-47.5.1-2-1', 'swat-18.2', 'talk-37.5', 'tape-22.4', 'tape-22.4-1', 'tell-37.2', 'throw-17.1', 'throw-17.1-1', 'throw-17.1-1-1', 'tingle-40.8.2', 'touch-20', 'touch-20-1', 'transcribe-25.4', 'transfer_mesg-37.1.1', 'transfer_mesg-37.1.1-1', 'transfer_mesg-37.1.1-1-1', 'try-61', 'turn-26.6.1', 'turn-26.6.1-1', 'urge-58', 'vehicle-51.4.1', 'vehicle-51.4.1-1', 'waltz-51.5', 'want-32.1', 'want-32.1-1', 'want-32.1-1-1', 'weather-57', 'weekend-56', 'wink-40.3.1', 'wink-40.3.1-1', 'wipe_instr-10.4.2', 'wipe_instr-10.4.2-1', 'wipe_manner-10.4.1', 'wipe_manner-10.4.1-1', 'wish-62', 'withdraw-82', 'withdraw-82-1', 'withdraw-82-2', 'withdraw-82-3']</span>
</pre></div>
</div>
<p>The primary object in the lexicon is a class record, which is stored
as an ElementTree xml object. The class record for a given class
identifier is returned by the <cite>vnclass()</cite> method:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">vnclass</span><span class="p">(</span><span class="s1">'remove-10.1'</span><span class="p">)</span>
<span class="go"><Element 'VNCLASS' at ...></span>
</pre></div>
</div>
<p>The <cite>vnclass()</cite> method also accepts “short” identifiers, such as ‘10.1’:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">vnclass</span><span class="p">(</span><span class="s1">'10.1'</span><span class="p">)</span>
<span class="go"><Element 'VNCLASS' at ...></span>
</pre></div>
</div>
<p>See the Verbnet documentation, or the Verbnet files, for information
about the structure of this xml. As an example, we can retrieve a
list of thematic roles for a given Verbnet class:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">vn_31_2</span> <span class="o">=</span> <span class="n">verbnet</span><span class="o">.</span><span class="n">vnclass</span><span class="p">(</span><span class="s1">'admire-31.2'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">themrole</span> <span class="ow">in</span> <span class="n">vn_31_2</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s1">'THEMROLES/THEMROLE'</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">themrole</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">'type'</span><span class="p">],</span> <span class="n">end</span><span class="o">=</span><span class="s1">' '</span><span class="p">)</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">selrestr</span> <span class="ow">in</span> <span class="n">themrole</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s1">'SELRESTRS/SELRESTR'</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s1">'[</span><span class="si">%(Value)s%(type)s</span><span class="s1">]'</span> <span class="o">%</span> <span class="n">selrestr</span><span class="o">.</span><span class="n">attrib</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">' '</span><span class="p">)</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">()</span>
<span class="go">Theme</span>
<span class="go">Experiencer [+animate]</span>
<span class="go">Predicate</span>
</pre></div>
</div>
<p>The Verbnet corpus also provides a variety of pretty printing
functions that can be used to display the xml contents in a more
concise form. The simplest such method is <cite>pprint()</cite>:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">verbnet</span><span class="o">.</span><span class="n">pprint</span><span class="p">(</span><span class="s1">'57'</span><span class="p">))</span>
<span class="go">weather-57</span>
<span class="go"> Subclasses: (none)</span>
<span class="go"> Members: blow clear drizzle fog freeze gust hail howl lightning mist</span>
<span class="go"> mizzle pelt pour precipitate rain roar shower sleet snow spit spot</span>
<span class="go"> sprinkle storm swelter teem thaw thunder</span>
<span class="go"> Thematic roles:</span>
<span class="go"> * Theme[+concrete +force]</span>
<span class="go"> Frames:</span>
<span class="go"> Intransitive (Expletive Subject)</span>
<span class="go"> Example: It's raining.</span>
<span class="go"> Syntax: LEX[it] LEX[[+be]] VERB</span>
<span class="go"> Semantics:</span>
<span class="go"> * weather(during(E), Weather_type, ?Theme)</span>
<span class="go"> NP (Expletive Subject, Theme Object)</span>
<span class="go"> Example: It's raining cats and dogs.</span>
<span class="go"> Syntax: LEX[it] LEX[[+be]] VERB NP[Theme]</span>
<span class="go"> Semantics:</span>
<span class="go"> * weather(during(E), Weather_type, Theme)</span>
<span class="go"> PP (Expletive Subject, Theme-PP)</span>
<span class="go"> Example: It was pelting with rain.</span>
<span class="go"> Syntax: LEX[it[+be]] VERB PREP[with] NP[Theme]</span>
<span class="go"> Semantics:</span>
<span class="go"> * weather(during(E), Weather_type, Theme)</span>
</pre></div>
</div>
<p>Verbnet gives us frames that link the syntax and semantics using an example.
These frames are part of the corpus and we can use <cite>frames()</cite> to get a frame
for a given verbnet class.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">frame</span> <span class="o">=</span> <span class="n">verbnet</span><span class="o">.</span><span class="n">frames</span><span class="p">(</span><span class="s1">'57'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">frame</span> <span class="o">==</span> <span class="p">[{</span><span class="s1">'example'</span><span class="p">:</span> <span class="s2">"It's raining."</span><span class="p">,</span> <span class="s1">'description'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'primary'</span><span class="p">:</span> <span class="s1">'Intransitive'</span><span class="p">,</span> <span class="s1">'secondary'</span><span class="p">:</span> <span class="s1">'Expletive Subject'</span><span class="p">},</span> <span class="s1">'syntax'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'LEX'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">'it'</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}},</span> <span class="p">{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'LEX'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">'[+be]'</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}},</span> <span class="p">{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'VERB'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}}],</span> <span class="s1">'semantics'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'predicate_value'</span><span class="p">:</span> <span class="s1">'weather'</span><span class="p">,</span> <span class="s1">'arguments'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'Event'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'during(E)'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'VerbSpecific'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'Weather_type'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'ThemRole'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'?Theme'</span><span class="p">}],</span> <span class="s1">'negated'</span><span class="p">:</span> <span class="kc">False</span><span class="p">}]},</span> <span class="p">{</span><span class="s1">'example'</span><span class="p">:</span> <span class="s2">"It's raining cats and dogs."</span><span class="p">,</span> <span class="s1">'description'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'primary'</span><span class="p">:</span> <span class="s1">'NP'</span><span class="p">,</span> <span class="s1">'secondary'</span><span class="p">:</span> <span class="s1">'Expletive Subject, Theme Object'</span><span class="p">},</span> <span class="s1">'syntax'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'LEX'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">'it'</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}},</span> <span class="p">{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'LEX'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">'[+be]'</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}},</span> <span class="p">{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'VERB'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}},</span> <span class="p">{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'NP'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">'Theme'</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}}],</span> <span class="s1">'semantics'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'predicate_value'</span><span class="p">:</span> <span class="s1">'weather'</span><span class="p">,</span> <span class="s1">'arguments'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'Event'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'during(E)'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'VerbSpecific'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'Weather_type'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'ThemRole'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'Theme'</span><span class="p">}],</span> <span class="s1">'negated'</span><span class="p">:</span> <span class="kc">False</span><span class="p">}]},</span> <span class="p">{</span><span class="s1">'example'</span><span class="p">:</span> <span class="s1">'It was pelting with rain.'</span><span class="p">,</span> <span class="s1">'description'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'primary'</span><span class="p">:</span> <span class="s1">'PP'</span><span class="p">,</span> <span class="s1">'secondary'</span><span class="p">:</span> <span class="s1">'Expletive Subject, Theme-PP'</span><span class="p">},</span> <span class="s1">'syntax'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'LEX'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">'it[+be]'</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}},</span> <span class="p">{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'VERB'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}},</span> <span class="p">{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'PREP'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">'with'</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}},</span> <span class="p">{</span><span class="s1">'pos_tag'</span><span class="p">:</span> <span class="s1">'NP'</span><span class="p">,</span> <span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'value'</span><span class="p">:</span> <span class="s1">'Theme'</span><span class="p">,</span> <span class="s1">'selrestrs'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'synrestrs'</span><span class="p">:</span> <span class="p">[]}}],</span> <span class="s1">'semantics'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'predicate_value'</span><span class="p">:</span> <span class="s1">'weather'</span><span class="p">,</span> <span class="s1">'arguments'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'Event'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'during(E)'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'VerbSpecific'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'Weather_type'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'ThemRole'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'Theme'</span><span class="p">}],</span> <span class="s1">'negated'</span><span class="p">:</span> <span class="kc">False</span><span class="p">}]}]</span>
<span class="go">True</span>
</pre></div>
</div>
<p>Verbnet corpus lets us access thematic roles individually using <cite>themroles()</cite>.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">themroles</span> <span class="o">=</span> <span class="n">verbnet</span><span class="o">.</span><span class="n">themroles</span><span class="p">(</span><span class="s1">'57'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">themroles</span> <span class="o">==</span> <span class="p">[{</span><span class="s1">'modifiers'</span><span class="p">:</span> <span class="p">[{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'concrete'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'+'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'type'</span><span class="p">:</span> <span class="s1">'force'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'+'</span><span class="p">}],</span> <span class="s1">'type'</span><span class="p">:</span> <span class="s1">'Theme'</span><span class="p">}]</span>
<span class="go">True</span>
</pre></div>
</div>
<p>Verbnet classes may also have subclasses sharing similar syntactic and semantic properties
while having differences with the superclass. The Verbnet corpus allows us to access these
subclasses using <cite>subclasses()</cite>.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">verbnet</span><span class="o">.</span><span class="n">subclasses</span><span class="p">(</span><span class="s1">'9.1'</span><span class="p">))</span> <span class="c1">#Testing for 9.1 since '57' does not have subclasses</span>
<span class="go">['put-9.1-1', 'put-9.1-2']</span>
</pre></div>
</div>
</section>
<section id="nps-chat">
<h5>nps_chat<a class="headerlink" href="#nps-chat" title="Link to this heading">¶</a></h5>
<p>The NPS Chat Corpus, Release 1.0 consists of over 10,000 posts in age-specific
chat rooms, which have been anonymized, POS-tagged and dialogue-act tagged.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">nps_chat</span><span class="o">.</span><span class="n">words</span><span class="p">())</span>
<span class="go">['now', 'im', 'left', 'with', 'this', 'gay', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">nps_chat</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">())</span>
<span class="go">[('now', 'RB'), ('im', 'PRP'), ('left', 'VBD'), ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">nps_chat</span><span class="o">.</span><span class="n">tagged_posts</span><span class="p">())</span>
<span class="go">[[('now', 'RB'), ('im', 'PRP'), ('left', 'VBD'), ('with', 'IN'),</span>
<span class="go">('this', 'DT'), ('gay', 'JJ'), ('name', 'NN')], [(':P', 'UH')], ...]</span>
</pre></div>
</div>
<p>We can access the XML elements corresponding to individual posts. These elements
have <code class="docutils literal notranslate"><span class="pre">class</span></code> and <code class="docutils literal notranslate"><span class="pre">user</span></code> attributes that we can access using <code class="docutils literal notranslate"><span class="pre">p.attrib['class']</span></code>
and <code class="docutils literal notranslate"><span class="pre">p.attrib['user']</span></code>. They also have text content, accessed using <code class="docutils literal notranslate"><span class="pre">p.text</span></code>.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">nps_chat</span><span class="o">.</span><span class="n">xml_posts</span><span class="p">())</span>
<span class="go">[<Element 'Post' at 0...>, <Element 'Post' at 0...>, ...]</span>
<span class="gp">>>> </span><span class="n">posts</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">nps_chat</span><span class="o">.</span><span class="n">xml_posts</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">sorted</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">FreqDist</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">'class'</span><span class="p">]</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">posts</span><span class="p">)</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>
<span class="go">['Accept', 'Bye', 'Clarify', 'Continuer', 'Emotion', 'Emphasis',</span>
<span class="go">'Greet', 'Other', 'Reject', 'Statement', 'System', 'nAnswer',</span>
<span class="go">'whQuestion', 'yAnswer', 'ynQuestion']</span>
<span class="gp">>>> </span><span class="n">posts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">text</span>
<span class="go">'now im left with this gay name'</span>
</pre></div>
</div>
<p>In addition to the above methods for accessing tagged text, we can navigate
the XML structure directly, as follows:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">tokens</span> <span class="o">=</span> <span class="n">posts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s1">'terminals/t'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">'pos'</span><span class="p">]</span> <span class="o">+</span> <span class="s2">"/"</span> <span class="o">+</span> <span class="n">t</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">'word'</span><span class="p">]</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">tokens</span><span class="p">]</span>
<span class="go">['RB/now', 'PRP/im', 'VBD/left', 'IN/with', 'DT/this', 'JJ/gay', 'NN/name']</span>
</pre></div>
</div>
</section>
<section id="multext-east">
<h5>multext_east<a class="headerlink" href="#multext-east" title="Link to this heading">¶</a></h5>
<p>The Multext-East Corpus consists of POS-tagged versions of George Orwell’s book
1984 in 12 languages: English, Czech, Hungarian, Macedonian, Slovenian, Serbian,
Slovak, Romanian, Estonian, Farsi, Bulgarian and Polish.
The corpus can be accessed using the usual methods for tagged corpora. The tagset
can be transformed from the Multext-East specific MSD tags to the Universal tagset
using the “tagset” parameter of all functions returning tagged parts of the corpus.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">multext_east</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s2">"oana-en.xml"</span><span class="p">))</span>
<span class="go">['It', 'was', 'a', 'bright', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">multext_east</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">(</span><span class="s2">"oana-en.xml"</span><span class="p">))</span>
<span class="go">[('It', '#Pp3ns'), ('was', '#Vmis3s'), ('a', '#Di'), ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">multext_east</span><span class="o">.</span><span class="n">tagged_sents</span><span class="p">(</span><span class="s2">"oana-en.xml"</span><span class="p">,</span> <span class="s2">"universal"</span><span class="p">))</span>
<span class="go">[[('It', 'PRON'), ('was', 'VERB'), ('a', 'DET'), ...]</span>
</pre></div>
</div>
</section>
</section>
</section>
<section id="corpus-reader-classes">
<h3>Corpus Reader Classes<a class="headerlink" href="#corpus-reader-classes" title="Link to this heading">¶</a></h3>
<p>NLTK’s <em>corpus reader</em> classes are used to access the contents of a
diverse set of corpora. Each corpus reader class is specialized to
handle a specific corpus format. Examples include the
<cite>PlaintextCorpusReader</cite>, which handles corpora that consist of a set
of unannotated text files, and the <cite>BracketParseCorpusReader</cite>, which
handles corpora that consist of files containing
parenthesis-delineated parse trees.</p>
<section id="automatically-created-corpus-reader-instances">
<h4>Automatically Created Corpus Reader Instances<a class="headerlink" href="#automatically-created-corpus-reader-instances" title="Link to this heading">¶</a></h4>
<p>When the <cite>nltk.corpus</cite> module is imported, it automatically creates a
set of corpus reader instances that can be used to access the corpora
in the NLTK data distribution. Here is a small sample of those
corpus reader instances:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">nltk</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span>
<span class="go"><CategorizedTaggedCorpusReader ...></span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">treebank</span>
<span class="go"><BracketParseCorpusReader ...></span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">names</span>
<span class="go"><WordListCorpusReader ...></span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">genesis</span>
<span class="go"><PlaintextCorpusReader ...></span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">inaugural</span>
<span class="go"><PlaintextCorpusReader ...></span>
</pre></div>
</div>
<p>This sample illustrates that different corpus reader classes are used
to read different corpora; but that the same corpus reader class may
be used for more than one corpus (e.g., <code class="docutils literal notranslate"><span class="pre">genesis</span></code> and <code class="docutils literal notranslate"><span class="pre">inaugural</span></code>).</p>
</section>
<section id="creating-new-corpus-reader-instances">
<h4>Creating New Corpus Reader Instances<a class="headerlink" href="#creating-new-corpus-reader-instances" title="Link to this heading">¶</a></h4>
<p>Although the <cite>nltk.corpus</cite> module automatically creates corpus reader
instances for the corpora in the NLTK data distribution, you may
sometimes need to create your own corpus reader. In particular, you
would need to create your own corpus reader if you want…</p>
<ul class="simple">
<li><p>To access a corpus that is not included in the NLTK data
distribution.</p></li>
<li><p>To access a full copy of a corpus for which the NLTK data
distribution only provides a sample.</p></li>
<li><p>To access a corpus using a customized corpus reader (e.g., with
a customized tokenizer).</p></li>
</ul>
<p>To create a new corpus reader, you will first need to look up the
signature for that corpus reader’s constructor. Different corpus
readers have different constructor signatures, but most of the
constructor signatures have the basic form:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">SomeCorpusReader</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">files</span><span class="p">,</span> <span class="o">...</span><span class="n">options</span><span class="o">...</span><span class="p">)</span>
</pre></div>
</div>
<p>Where <code class="docutils literal notranslate"><span class="pre">root</span></code> is an absolute path to the directory containing the
corpus data files; <code class="docutils literal notranslate"><span class="pre">files</span></code> is either a list of file names (relative
to <code class="docutils literal notranslate"><span class="pre">root</span></code>) or a regexp specifying which files should be included;
and <code class="docutils literal notranslate"><span class="pre">options</span></code> are additional reader-specific options. For example,
we can create a customized corpus reader for the genesis corpus that
uses a different sentence tokenizer as follows:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># Find the directory where the corpus lives.</span>
<span class="gp">>>> </span><span class="n">genesis_dir</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'corpora/genesis'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c1"># Create our custom sentence tokenizer.</span>
<span class="gp">>>> </span><span class="n">my_sent_tokenizer</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">RegexpTokenizer</span><span class="p">(</span><span class="s1">'[^.!?]+'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c1"># Create the new corpus reader object.</span>
<span class="gp">>>> </span><span class="n">my_genesis</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">PlaintextCorpusReader</span><span class="p">(</span>
<span class="gp">... </span> <span class="n">genesis_dir</span><span class="p">,</span> <span class="sa">r</span><span class="s1">'.*\.txt'</span><span class="p">,</span> <span class="n">sent_tokenizer</span><span class="o">=</span><span class="n">my_sent_tokenizer</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c1"># Use the new corpus reader object.</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">my_genesis</span><span class="o">.</span><span class="n">sents</span><span class="p">(</span><span class="s1">'english-kjv.txt'</span><span class="p">)[</span><span class="mi">0</span><span class="p">])</span>
<span class="go">['In', 'the', 'beginning', 'God', 'created', 'the', 'heaven',</span>
<span class="go"> 'and', 'the', 'earth']</span>
</pre></div>
</div>
<p>If you wish to read your own plaintext corpus, which is stored in the
directory ‘/usr/share/some-corpus’, then you can create a corpus
reader for it with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">my_corpus</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">PlaintextCorpusReader</span><span class="p">(</span>
<span class="gp">... </span> <span class="s1">'/usr/share/some-corpus'</span><span class="p">,</span> <span class="sa">r</span><span class="s1">'.*\.txt'</span><span class="p">)</span>
</pre></div>
</div>
<p>For a complete list of corpus reader subclasses, see the API
documentation for <cite>nltk.corpus.reader</cite>.</p>
</section>
<section id="corpus-types">
<h4>Corpus Types<a class="headerlink" href="#corpus-types" title="Link to this heading">¶</a></h4>
<p>Corpora vary widely in the types of content they include. This is
reflected in the fact that the base class <cite>CorpusReader</cite> only defines
a few general-purpose methods for listing and accessing the files that
make up a corpus. It is up to the subclasses to define <em>data access
methods</em> that provide access to the information in the corpus.
However, corpus reader subclasses should be consistent in their
definitions of these data access methods wherever possible.</p>
<p>At a high level, corpora can be divided into three basic types:</p>
<ul class="simple">
<li><p>A <em>token corpus</em> contains information about specific occurrences of
language use (or linguistic tokens), such as dialogues or written
texts. Examples of token corpora are collections of written text
and collections of speech.</p></li>
<li><p>A <em>type corpus</em>, or <em>lexicon</em>, contains information about a coherent
set of lexical items (or linguistic types). Examples of lexicons
are dictionaries and word lists.</p></li>
<li><p>A <em>language description corpus</em> contains information about a set of
non-lexical linguistic constructs, such as grammar rules.</p></li>
</ul>
<p>However, many individual corpora blur the distinctions between these
types. For example, corpora that are primarily lexicons may include
token data in the form of example sentences; and corpora that are
primarily token corpora may be accompanied by one or more word lists
or other lexical data sets.</p>
<p>Because corpora vary so widely in their information content, we have
decided that it would not be wise to use separate corpus reader base
classes for different corpus types. Instead, we simply try to make
the corpus readers consistent wherever possible, but let them differ
where the underlying data itself differs.</p>
</section>
<section id="common-corpus-reader-methods">
<h4>Common Corpus Reader Methods<a class="headerlink" href="#common-corpus-reader-methods" title="Link to this heading">¶</a></h4>
<p>As mentioned above, there are only a handful of methods that all
corpus readers are guaranteed to implement. These methods provide
access to the files that contain the corpus data. Every corpus is
assumed to consist of one or more files, all located in a common root
directory (or in subdirectories of that root directory). The absolute
path to the root directory is stored in the <code class="docutils literal notranslate"><span class="pre">root</span></code> property:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">os</span>
<span class="gp">>>> </span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">genesis</span><span class="o">.</span><span class="n">root</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">sep</span><span class="p">,</span><span class="s1">'/'</span><span class="p">)</span>
<span class="go">'.../nltk_data/corpora/genesis'</span>
</pre></div>
</div>
<p>Each file within the corpus is identified by a platform-independent
identifier, which is basically a path string that uses <code class="docutils literal notranslate"><span class="pre">/</span></code> as the
path separator. I.e., this identifier can be converted to a relative
path as follows:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">some_corpus_file_id</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">reuters</span><span class="o">.</span><span class="n">fileids</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">os.path</span>
<span class="gp">>>> </span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">normpath</span><span class="p">(</span><span class="n">some_corpus_file_id</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">sep</span><span class="p">,</span><span class="s1">'/'</span><span class="p">)</span>
<span class="go">'test/14826'</span>
</pre></div>
</div>
<p>To get a list of all data files that make up a corpus, use the
<code class="docutils literal notranslate"><span class="pre">fileids()</span></code> method. In some corpora, these files will not all contain
the same type of data; for example, for the <code class="docutils literal notranslate"><span class="pre">nltk.corpus.timit</span></code>
corpus, <code class="docutils literal notranslate"><span class="pre">fileids()</span></code> will return a list including text files, word
segmentation files, phonetic transcription files, sound files, and
metadata files. For corpora with diverse file types, the <code class="docutils literal notranslate"><span class="pre">fileids()</span></code>
method will often take one or more optional arguments, which can be
used to get a list of the files with a specific file type:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">timit</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa1.txt', 'dr1-fvmh0/sa1.wav', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">timit</span><span class="o">.</span><span class="n">fileids</span><span class="p">(</span><span class="s1">'phn'</span><span class="p">)</span>
<span class="go">['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa2.phn', 'dr1-fvmh0/si1466.phn', ...]</span>
</pre></div>
</div>
<p>In some corpora, the files are divided into distinct categories. For
these corpora, the <code class="docutils literal notranslate"><span class="pre">fileids()</span></code> method takes an optional argument,
which can be used to get a list of the files within a specific category:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="o">.</span><span class="n">fileids</span><span class="p">(</span><span class="s1">'hobbies'</span><span class="p">)</span>
<span class="go">['ce01', 'ce02', 'ce03', 'ce04', 'ce05', 'ce06', 'ce07', ...]</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">abspath()</span></code> method can be used to find the absolute path to a
corpus file, given its file identifier:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="s1">'ce06'</span><span class="p">))</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">sep</span><span class="p">,</span><span class="s1">'/'</span><span class="p">)</span>
<span class="go">'.../corpora/brown/ce06'</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">abspaths()</span></code> method can be used to find the absolute paths for
one corpus file, a list of corpus files, or (if no fileids are specified),
all corpus files.</p>
<p>This method is mainly useful as a helper method when defining corpus
data access methods, since data access methods can usually be called
with a string argument (to get a view for a specific file), with a
list argument (to get a view for a specific list of files), or with no
argument (to get a view for the whole corpus).</p>
</section>
<section id="data-access-methods">
<h4>Data Access Methods<a class="headerlink" href="#data-access-methods" title="Link to this heading">¶</a></h4>
<p>Individual corpus reader subclasses typically extend this basic set of
file-access methods with one or more <em>data access methods</em>, which provide
easy access to the data contained in the corpus. The signatures for
data access methods often have the basic form:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">corpus_reader</span><span class="o">.</span><span class="n">some_data</span> <span class="n">access</span><span class="p">(</span><span class="n">fileids</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">...</span><span class="n">options</span><span class="o">...</span><span class="p">)</span>
</pre></div>
</div>
<p>Where <code class="docutils literal notranslate"><span class="pre">fileids</span></code> can be a single file identifier string (to get a view
for a specific file); a list of file identifier strings (to get a view
for a specific list of files); or None (to get a view for the entire
corpus). Some of the common data access methods, and their return
types, are:</p>
<blockquote>
<div><ul class="simple">
<li><p>I{corpus}.words(): list of str</p></li>
<li><p>I{corpus}.sents(): list of (list of str)</p></li>
<li><p>I{corpus}.paras(): list of (list of (list of str))</p></li>
<li><p>I{corpus}.tagged_words(): list of (str,str) tuple</p></li>
<li><p>I{corpus}.tagged_sents(): list of (list of (str,str))</p></li>
<li><p>I{corpus}.tagged_paras(): list of (list of (list of (str,str)))</p></li>
<li><p>I{corpus}.chunked_sents(): list of (Tree w/ (str,str) leaves)</p></li>
<li><p>I{corpus}.parsed_sents(): list of (Tree with str leaves)</p></li>
<li><p>I{corpus}.parsed_paras(): list of (list of (Tree with str leaves))</p></li>
<li><p>I{corpus}.xml(): A single xml ElementTree</p></li>
<li><p>I{corpus}.raw(): str (unprocessed corpus contents)</p></li>
</ul>
</div></blockquote>
<p>For example, the <cite>words()</cite> method is supported by many different
corpora, and returns a flat list of word strings:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['The', 'Fulton', 'County', 'Grand', 'Jury', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">treebank</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['Pierre', 'Vinken', ',', '61', 'years', 'old', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">conll2002</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['Sao', 'Paulo', '(', 'Brasil', ')', ',', '23', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">genesis</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['In', 'the', 'beginning', 'God', 'created', ...]</span>
</pre></div>
</div>
<p>On the other hand, the <cite>tagged_words()</cite> method is only supported by
corpora that include part-of-speech annotations:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">()</span>
<span class="go">[('The', 'AT'), ('Fulton', 'NP-TL'), ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">treebank</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">()</span>
<span class="go">[('Pierre', 'NNP'), ('Vinken', 'NNP'), ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">conll2002</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">()</span>
<span class="go">[('Sao', 'NC'), ('Paulo', 'VMI'), ('(', 'Fpa'), ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">genesis</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">()</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="w"> </span><span class="c">...</span>
<span class="gr">AttributeError</span>: <span class="n">'PlaintextCorpusReader' object has no attribute 'tagged_words'</span>
</pre></div>
</div>
<p>Although most corpus readers use file identifiers to index their
content, some corpora use different identifiers instead. For example,
the data access methods for the <code class="docutils literal notranslate"><span class="pre">timit</span></code> corpus uses <em>utterance
identifiers</em> to select which corpus items should be returned:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">timit</span><span class="o">.</span><span class="n">utteranceids</span><span class="p">()</span>
<span class="go">['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">timit</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'dr1-fvmh0/sa2'</span><span class="p">)</span>
<span class="go">["don't", 'ask', 'me', 'to', 'carry', 'an', 'oily', 'rag', 'like', 'that']</span>
</pre></div>
</div>
<p>Attempting to call <code class="docutils literal notranslate"><span class="pre">timit</span></code>‘s data access methods with a file
identifier will result in an exception:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">timit</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa1.txt', 'dr1-fvmh0/sa1.wav', ...]</span>
<span class="gp">>>> </span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">timit</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'dr1-fvmh0/sa1.txt'</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="w"> </span><span class="c">...</span>
<span class="gr">IOError</span>: <span class="n">No such file or directory: '.../dr1-fvmh0/sa1.txt.wrd'</span>
</pre></div>
</div>
<p>As another example, the <code class="docutils literal notranslate"><span class="pre">propbank</span></code> corpus defines the <code class="docutils literal notranslate"><span class="pre">roleset()</span></code>
method, which expects a roleset identifier, not a file identifier:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">roleset</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">propbank</span><span class="o">.</span><span class="n">roleset</span><span class="p">(</span><span class="s1">'eat.01'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">xml.etree</span><span class="w"> </span><span class="kn">import</span> <span class="n">ElementTree</span> <span class="k">as</span> <span class="n">ET</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">ET</span><span class="o">.</span><span class="n">tostring</span><span class="p">(</span><span class="n">roleset</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf8'</span><span class="p">))</span>
<span class="go"><roleset id="eat.01" name="consume" vncls="39.1"></span>
<span class="go"> <roles></span>
<span class="go"> <role descr="consumer, eater" n="0">...</role>...</span>
<span class="go"> </roles>...</span>
<span class="go"></roleset>...</span>
</pre></div>
</div>
</section>
<section id="stream-backed-corpus-views">
<h4>Stream Backed Corpus Views<a class="headerlink" href="#stream-backed-corpus-views" title="Link to this heading">¶</a></h4>
<p>An important feature of NLTK’s corpus readers is that many of them
access the underlying data files using “corpus views.” A <em>corpus
view</em> is an object that acts like a simple data structure (such as a
list), but does not store the data elements in memory; instead, data
elements are read from the underlying data files on an as-needed
basis.</p>
<p>By only loading items from the file on an as-needed basis, corpus
views maintain both memory efficiency and responsiveness. The memory
efficiency of corpus readers is important because some corpora contain
very large amounts of data, and storing the entire data set in memory
could overwhelm many machines. The responsiveness is important when
experimenting with corpora in interactive sessions and in in-class
demonstrations.</p>
<p>The most common corpus view is the <cite>StreamBackedCorpusView</cite>, which
acts as a read-only list of tokens. Two additional corpus view
classes, <cite>ConcatenatedCorpusView</cite> and <cite>LazySubsequence</cite>, make it
possible to create concatenations and take slices of
<cite>StreamBackedCorpusView</cite> objects without actually storing the
resulting list-like object’s elements in memory.</p>
<p>In the future, we may add additional corpus views that act like other
basic data structures, such as dictionaries.</p>
</section>
<section id="writing-new-corpus-readers">
<h4>Writing New Corpus Readers<a class="headerlink" href="#writing-new-corpus-readers" title="Link to this heading">¶</a></h4>
<p>In order to add support for new corpus formats, it is necessary to
define new corpus reader classes. For many corpus formats, writing
new corpus readers is relatively straight-forward. In this section,
we’ll describe what’s involved in creating a new corpus reader. If
you do create a new corpus reader, we encourage you to contribute it
back to the NLTK project.</p>
<section id="don-t-reinvent-the-wheel">
<h5>Don’t Reinvent the Wheel<a class="headerlink" href="#don-t-reinvent-the-wheel" title="Link to this heading">¶</a></h5>
<p>Before you start writing a new corpus reader, you should check to be
sure that the desired format can’t be read using an existing corpus
reader with appropriate constructor arguments. For example, although
the <cite>TaggedCorpusReader</cite> assumes that words and tags are separated by
<code class="docutils literal notranslate"><span class="pre">/</span></code> characters by default, an alternative tag-separation character
can be specified via the <code class="docutils literal notranslate"><span class="pre">sep</span></code> constructor argument. You should
also check whether the new corpus format can be handled by subclassing
an existing corpus reader, and tweaking a few methods or variables.</p>
</section>
<section id="design">
<h5>Design<a class="headerlink" href="#design" title="Link to this heading">¶</a></h5>
<p>If you decide to write a new corpus reader from scratch, then you
should first decide which data access methods you want the reader to
provide, and what their signatures should be. You should look at
existing corpus readers that process corpora with similar data
contents, and try to be consistent with those corpus readers whenever
possible.</p>
<p>You should also consider what sets of identifiers are appropriate for
the corpus format. Where it’s practical, file identifiers should be
used. However, for some corpora, it may make sense to use additional
sets of identifiers. Each set of identifiers should have a distinct
name (e.g., fileids, utteranceids, rolesets); and you should be consistent
in using that name to refer to that identifier. Do not use parameter
names like <code class="docutils literal notranslate"><span class="pre">id</span></code>, which leave it unclear what type of identifier is
required.</p>
<p>Once you’ve decided what data access methods and identifiers are
appropriate for your corpus, you should decide if there are any
customizable parameters that you’d like the corpus reader to handle.
These parameters make it possible to use a single corpus reader to
handle a wider variety of corpora. The <code class="docutils literal notranslate"><span class="pre">sep</span></code> argument for
<cite>TaggedCorpusReader</cite>, mentioned above, is an example of a customizable
corpus reader parameter.</p>
</section>
<section id="implementation">
<h5>Implementation<a class="headerlink" href="#implementation" title="Link to this heading">¶</a></h5>
<section id="constructor">
<h6>Constructor<a class="headerlink" href="#constructor" title="Link to this heading">¶</a></h6>
<p>If your corpus reader implements any customizable parameters, then
you’ll need to override the constructor. Typically, the new
constructor will first call its base class’s constructor, and then
store the customizable parameters. For example, the
<cite>ConllChunkCorpusReader</cite>‘s constructor is defined as follows:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="p">,</span> <span class="n">fileids</span><span class="p">,</span> <span class="n">chunk_types</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf8'</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">tagset</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">separator</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="gp">... </span> <span class="n">ConllCorpusReader</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="gp">... </span> <span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="p">,</span> <span class="n">fileids</span><span class="p">,</span> <span class="p">(</span><span class="s1">'words'</span><span class="p">,</span> <span class="s1">'pos'</span><span class="p">,</span> <span class="s1">'chunk'</span><span class="p">),</span>
<span class="gp">... </span> <span class="n">chunk_types</span><span class="o">=</span><span class="n">chunk_types</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="n">encoding</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">tagset</span><span class="o">=</span><span class="n">tagset</span><span class="p">,</span> <span class="n">separator</span><span class="o">=</span><span class="n">separator</span><span class="p">)</span>
</pre></div>
</div>
<p>If your corpus reader does not implement any customization parameters,
then you can often just inherit the base class’s constructor.</p>
</section>
<section id="id1">
<h6>Data Access Methods<a class="headerlink" href="#id1" title="Link to this heading">¶</a></h6>
<p>The most common type of data access method takes an argument
identifying which files to access, and returns a view covering those
files. This argument may be a single file identifier string (to get a
view for a specific file); a list of file identifier strings (to get a
view for a specific list of files); or None (to get a view for the
entire corpus). The method’s implementation converts this argument to
a list of path names using the <cite>abspaths()</cite> method, which handles all
three value types (string, list, and None):</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="o">.</span><span class="n">abspaths</span><span class="p">())</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="go">[FileSystemPathPointer('.../corpora/brown/ca01'),</span>
<span class="go"> FileSystemPathPointer('.../corpora/brown/ca02'), ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="o">.</span><span class="n">abspaths</span><span class="p">(</span><span class="s1">'ce06'</span><span class="p">))</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="go">[FileSystemPathPointer('.../corpora/brown/ce06')]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="o">.</span><span class="n">abspaths</span><span class="p">([</span><span class="s1">'ce06'</span><span class="p">,</span> <span class="s1">'ce07'</span><span class="p">]))</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="go">[FileSystemPathPointer('.../corpora/brown/ce06'),</span>
<span class="go"> FileSystemPathPointer('.../corpora/brown/ce07')]</span>
</pre></div>
</div>
<p>An example of this type of method is the <cite>words()</cite> method, defined by
the <cite>PlaintextCorpusReader</cite> as follows:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span><span class="w"> </span><span class="nf">words</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fileids</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">return</span> <span class="n">concat</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">CorpusView</span><span class="p">(</span><span class="n">fileid</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_read_word_block</span><span class="p">)</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">fileid</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">abspaths</span><span class="p">(</span><span class="n">fileids</span><span class="p">)])</span>
</pre></div>
</div>
<p>This method first uses <cite>abspaths()</cite> to convert <code class="docutils literal notranslate"><span class="pre">fileids</span></code> to a list of
absolute paths. It then creates a corpus view for each file, using
the <cite>PlaintextCorpusReader._read_word_block()</cite> method to read elements
from the data file (see the discussion of corpus views below).
Finally, it combines these corpus views using the
<cite>nltk.corpus.reader.util.concat()</cite> function.</p>
<p>When writing a corpus reader for a corpus that is never expected to be
very large, it can sometimes be appropriate to read the files
directly, rather than using a corpus view. For example, the
<cite>WordListCorpusView</cite> class defines its <cite>words()</cite> method as follows:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span><span class="w"> </span><span class="nf">words</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fileids</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">return</span> <span class="n">concat</span><span class="p">([[</span><span class="n">w</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="nb">open</span><span class="p">(</span><span class="n">fileid</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span> <span class="k">if</span> <span class="n">w</span><span class="p">]</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">fileid</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">abspaths</span><span class="p">(</span><span class="n">fileids</span><span class="p">)])</span>
</pre></div>
</div>
<p>(This is usually more appropriate for lexicons than for token corpora.)</p>
<p>If the type of data returned by a data access method is one for which
NLTK has a conventional representation (e.g., words, tagged words, and
parse trees), then you should use that representation. Otherwise, you
may find it necessary to define your own representation. For data
structures that are relatively corpus-specific, it’s usually best to
define new classes for these elements. For example, the <code class="docutils literal notranslate"><span class="pre">propbank</span></code>
corpus defines the <cite>PropbankInstance</cite> class to store the semantic role
labeling instances described by the corpus; and the <code class="docutils literal notranslate"><span class="pre">ppattach</span></code>
corpus defines the <cite>PPAttachment</cite> class to store the prepositional
attachment instances described by the corpus.</p>
</section>
<section id="corpus-views">
<h6>Corpus Views<a class="headerlink" href="#corpus-views" title="Link to this heading">¶</a></h6>
<p>The heart of a <cite>StreamBackedCorpusView</cite> is its <em>block reader</em>
function, which reads zero or more tokens from a stream, and returns
them as a list. A very simple example of a block reader is:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span><span class="w"> </span><span class="nf">simple_block_reader</span><span class="p">(</span><span class="n">stream</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">return</span> <span class="n">stream</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
</pre></div>
</div>
<p>This simple block reader reads a single line at a time, and returns a
single token (consisting of a string) for each whitespace-separated
substring on the line. A <cite>StreamBackedCorpusView</cite> built from this
block reader will act like a read-only list of all the
whitespace-separated tokens in an underlying file.</p>
<p>When deciding how to define the block reader for a given corpus,
careful consideration should be given to the size of blocks handled by
the block reader. Smaller block sizes will increase the memory
requirements of the corpus view’s internal data structures (by 2
integers per block). On the other hand, larger block sizes may
decrease performance for random access to the corpus. (But note that
larger block sizes will <em>not</em> decrease performance for iteration.)</p>
<p>Internally, the <cite>StreamBackedCorpusView</cite> class maintains a partial
mapping from token index to file position, with one entry per block.
When a token with a given index <em>i</em> is requested, the corpus view
constructs it as follows:</p>
<ol class="arabic simple">
<li><p>First, it searches the toknum/filepos mapping for the token index
closest to (but less than or equal to) <em>i</em>.</p></li>
<li><p>Then, starting at the file position corresponding to that index, it
reads one block at a time using the block reader until it reaches
the requested token.</p></li>
</ol>
<p>The toknum/filepos mapping is created lazily: it is initially empty,
but every time a new block is read, the block’s initial token is added
to the mapping. (Thus, the toknum/filepos map has one entry per
block.)</p>
<p>You can create your own corpus view in one of two ways:</p>
<ol class="arabic simple">
<li><p>Call the <cite>StreamBackedCorpusView</cite> constructor, and provide your
block reader function via the <code class="docutils literal notranslate"><span class="pre">block_reader</span></code> argument.</p></li>
<li><p>Subclass <cite>StreamBackedCorpusView</cite>, and override the
<cite>read_block()</cite> method.</p></li>
</ol>
<p>The first option is usually easier, but the second option can allow
you to write a single <cite>read_block</cite> method whose behavior can be
customized by different parameters to the subclass’s constructor. For
an example of this design pattern, see the <cite>TaggedCorpusView</cite> class,
which is used by <cite>TaggedCorpusView</cite>.</p>
</section>
</section>
</section>
</section>
<section id="regression-tests">
<h3>Regression Tests<a class="headerlink" href="#regression-tests" title="Link to this heading">¶</a></h3>
<p>The following helper functions are used to create and then delete
testing corpora that are stored in temporary directories. These
testing corpora are used to make sure the readers work correctly.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">tempfile</span><span class="o">,</span><span class="w"> </span><span class="nn">os.path</span><span class="o">,</span><span class="w"> </span><span class="nn">textwrap</span>
<span class="gp">>>> </span><span class="k">def</span><span class="w"> </span><span class="nf">make_testcorpus</span><span class="p">(</span><span class="n">ext</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="o">**</span><span class="n">fileids</span><span class="p">):</span>
<span class="gp">... </span> <span class="n">root</span> <span class="o">=</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">mkdtemp</span><span class="p">()</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">fileid</span><span class="p">,</span> <span class="n">contents</span> <span class="ow">in</span> <span class="n">fileids</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="gp">... </span> <span class="n">fileid</span> <span class="o">+=</span> <span class="n">ext</span>
<span class="gp">... </span> <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">fileid</span><span class="p">),</span> <span class="s1">'w'</span><span class="p">)</span>
<span class="gp">... </span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">textwrap</span><span class="o">.</span><span class="n">dedent</span><span class="p">(</span><span class="n">contents</span><span class="p">))</span>
<span class="gp">... </span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="gp">... </span> <span class="k">return</span> <span class="n">root</span>
<span class="gp">>>> </span><span class="k">def</span><span class="w"> </span><span class="nf">del_testcorpus</span><span class="p">(</span><span class="n">root</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">fileid</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="n">root</span><span class="p">):</span>
<span class="gp">... </span> <span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">fileid</span><span class="p">))</span>
<span class="gp">... </span> <span class="n">os</span><span class="o">.</span><span class="n">rmdir</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
</pre></div>
</div>
<section id="plaintext-corpus-reader">
<h4>Plaintext Corpus Reader<a class="headerlink" href="#plaintext-corpus-reader" title="Link to this heading">¶</a></h4>
<p>The plaintext corpus reader is used to access corpora that consist of
unprocessed plaintext data. It assumes that paragraph breaks are
indicated by blank lines. Sentences and words can be tokenized using
the default tokenizers, or by custom tokenizers specified as
parameters to the constructor.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">root</span> <span class="o">=</span> <span class="n">make_testcorpus</span><span class="p">(</span><span class="n">ext</span><span class="o">=</span><span class="s1">'.txt'</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">a</span><span class="o">=</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> This is the first sentence. Here is another</span>
<span class="gp">... </span><span class="s2"> sentence! And here's a third sentence.</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> This is the second paragraph. Tokenization is currently</span>
<span class="gp">... </span><span class="s2"> fairly simple, so the period in Mr. gets tokenized.</span>
<span class="gp">... </span><span class="s2"> """</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">b</span><span class="o">=</span><span class="s2">"""This is the second file."""</span><span class="p">)</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus.reader.plaintext</span><span class="w"> </span><span class="kn">import</span> <span class="n">PlaintextCorpusReader</span>
</pre></div>
</div>
<p>The list of documents can be specified explicitly, or implicitly (using a
regexp). The <code class="docutils literal notranslate"><span class="pre">ext</span></code> argument specifies a file extension.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">corpus</span> <span class="o">=</span> <span class="n">PlaintextCorpusReader</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="p">[</span><span class="s1">'a.txt'</span><span class="p">,</span> <span class="s1">'b.txt'</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['a.txt', 'b.txt']</span>
<span class="gp">>>> </span><span class="n">corpus</span> <span class="o">=</span> <span class="n">PlaintextCorpusReader</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="sa">r</span><span class="s1">'.*\.txt'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['a.txt', 'b.txt']</span>
</pre></div>
</div>
<p>The directory containing the corpus is corpus.root:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">str</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">root</span><span class="p">)</span> <span class="o">==</span> <span class="nb">str</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
<span class="go">True</span>
</pre></div>
</div>
<p>We can get a list of words, or the raw string:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['This', 'is', 'the', 'first', 'sentence', '.', ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">raw</span><span class="p">()[:</span><span class="mi">40</span><span class="p">]</span>
<span class="go">'This is the first sentence. Here is ano'</span>
</pre></div>
</div>
<p>Check that reading individual documents works, and reading all documents at
once works:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">()),</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="n">d</span><span class="p">))</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">()]</span>
<span class="go">(46, [40, 6])</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'a.txt'</span><span class="p">)</span>
<span class="go">['This', 'is', 'the', 'first', 'sentence', '.', ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'b.txt'</span><span class="p">)</span>
<span class="go">['This', 'is', 'the', 'second', 'file', '.']</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">()[:</span><span class="mi">4</span><span class="p">],</span> <span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">()[</span><span class="o">-</span><span class="mi">4</span><span class="p">:]</span>
<span class="go">(['This', 'is', 'the', 'first'], ['the', 'second', 'file', '.'])</span>
</pre></div>
</div>
<p>We’re done with the test corpus:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">del_testcorpus</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
</pre></div>
</div>
<p>Test the plaintext corpora that come with nltk:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">abc</span><span class="p">,</span> <span class="n">genesis</span><span class="p">,</span> <span class="n">inaugural</span>
<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">state_union</span><span class="p">,</span> <span class="n">webtext</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">corpus</span> <span class="ow">in</span> <span class="p">(</span><span class="n">abc</span><span class="p">,</span> <span class="n">genesis</span><span class="p">,</span> <span class="n">inaugural</span><span class="p">,</span> <span class="n">state_union</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">webtext</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span> <span class="nb">repr</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">())[:</span><span class="mi">60</span><span class="p">])</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span> <span class="nb">repr</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">()[:</span><span class="mi">10</span><span class="p">])[:</span><span class="mi">60</span><span class="p">])</span>
<span class="go"><PlaintextCorpusReader in '.../nltk_data/corpora/ab...'></span>
<span class="go"> ['rural.txt', 'science.txt']</span>
<span class="go"> ['PM', 'denies', 'knowledge', 'of', 'AWB', ...</span>
<span class="go"><PlaintextCorpusReader in '.../nltk_data/corpora/genesi...'></span>
<span class="go"> ['english-kjv.txt', 'english-web.txt', 'finnish.txt', ...</span>
<span class="go"> ['In', 'the', 'beginning', 'God', 'created', 'the', ...</span>
<span class="go"><PlaintextCorpusReader in '.../nltk_data/corpora/inaugura...'></span>
<span class="go"> ['1789-Washington.txt', '1793-Washington.txt', ...</span>
<span class="go"> ['Fellow', '-', 'Citizens', 'of', 'the', 'Senate', ...</span>
<span class="go"><PlaintextCorpusReader in '.../nltk_data/corpora/state_unio...'></span>
<span class="go"> ['1945-Truman.txt', '1946-Truman.txt', ...</span>
<span class="go"> ['PRESIDENT', 'HARRY', 'S', '.', 'TRUMAN', "'", ...</span>
<span class="go"><PlaintextCorpusReader in '.../nltk_data/corpora/webtex...'></span>
<span class="go"> ['firefox.txt', 'grail.txt', 'overheard.txt', ...</span>
<span class="go"> ['Cookie', 'Manager', ':', '"', 'Don', "'", 't', ...</span>
</pre></div>
</div>
</section>
<section id="tagged-corpus-reader">
<h4>Tagged Corpus Reader<a class="headerlink" href="#tagged-corpus-reader" title="Link to this heading">¶</a></h4>
<p>The Tagged Corpus reader can give us words, sentences, and paragraphs,
each tagged or untagged. All of the read methods can take one item
(in which case they return the contents of that file) or a list of
documents (in which case they concatenate the contents of those files).
By default, they apply to all documents in the corpus.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">root</span> <span class="o">=</span> <span class="n">make_testcorpus</span><span class="p">(</span>
<span class="gp">... </span> <span class="n">a</span><span class="o">=</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> This/det is/verb the/det first/adj sentence/noun ./punc</span>
<span class="gp">... </span><span class="s2"> Here/det is/verb another/adj sentence/noun ./punc</span>
<span class="gp">... </span><span class="s2"> Note/verb that/comp you/pron can/verb use/verb </span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> any/noun tag/noun set/noun</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> This/det is/verb the/det second/adj paragraph/noun ./punc</span>
<span class="gp">... </span><span class="s2"> word/n without/adj a/det tag/noun :/: hello ./punc</span>
<span class="gp">... </span><span class="s2"> """</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">b</span><span class="o">=</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> This/det is/verb the/det second/adj file/noun ./punc</span>
<span class="gp">... </span><span class="s2"> """</span><span class="p">)</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus.reader.tagged</span><span class="w"> </span><span class="kn">import</span> <span class="n">TaggedCorpusReader</span>
<span class="gp">>>> </span><span class="n">corpus</span> <span class="o">=</span> <span class="n">TaggedCorpusReader</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="nb">list</span><span class="p">(</span><span class="s1">'ab'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['a', 'b']</span>
<span class="gp">>>> </span><span class="nb">str</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">root</span><span class="p">)</span> <span class="o">==</span> <span class="nb">str</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['This', 'is', 'the', 'first', 'sentence', '.', ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">sents</span><span class="p">()</span>
<span class="go">[['This', 'is', 'the', 'first', ...], ['Here', 'is', 'another'...], ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">paras</span><span class="p">()</span>
<span class="go">[[['This', ...], ['Here', ...], ...], [['This', ...], ...], ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">()</span>
<span class="go">[('This', 'DET'), ('is', 'VERB'), ('the', 'DET'), ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">tagged_sents</span><span class="p">()</span>
<span class="go">[[('This', 'DET'), ('is', 'VERB'), ...], [('Here', 'DET'), ...], ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">tagged_paras</span><span class="p">()</span>
<span class="go">[[[('This', 'DET'), ...], ...], [[('This', 'DET'), ...], ...], ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">raw</span><span class="p">()[:</span><span class="mi">40</span><span class="p">]</span>
<span class="go">'This/det is/verb the/det first/adj sente'</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">()),</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="n">d</span><span class="p">))</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">()]</span>
<span class="go">(38, [32, 6])</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">sents</span><span class="p">()),</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">sents</span><span class="p">(</span><span class="n">d</span><span class="p">))</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">()]</span>
<span class="go">(6, [5, 1])</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">paras</span><span class="p">()),</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">paras</span><span class="p">(</span><span class="n">d</span><span class="p">))</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">()]</span>
<span class="go">(3, [2, 1])</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'a'</span><span class="p">))</span>
<span class="go">['This', 'is', 'the', 'first', 'sentence', '.', ...]</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'b'</span><span class="p">))</span>
<span class="go">['This', 'is', 'the', 'second', 'file', '.']</span>
<span class="gp">>>> </span><span class="n">del_testcorpus</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
</pre></div>
</div>
<p>The Brown Corpus uses the tagged corpus reader:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">brown</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['ca01', 'ca02', 'ca03', 'ca04', 'ca05', 'ca06', 'ca07', ...]</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">categories</span><span class="p">()</span>
<span class="go">['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor',</span>
<span class="go">'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">brown</span><span class="o">.</span><span class="n">root</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="se">\\\\</span><span class="s1">'</span><span class="p">,</span><span class="s1">'/'</span><span class="p">))</span>
<span class="go">FileSystemPathPointer('.../corpora/brown')</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['The', 'Fulton', 'County', 'Grand', 'Jury', ...]</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">sents</span><span class="p">()</span>
<span class="go">[['The', 'Fulton', 'County', 'Grand', ...], ...]</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">paras</span><span class="p">()</span>
<span class="go">[[['The', 'Fulton', 'County', ...]], [['The', 'jury', ...]], ...]</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_words</span><span class="p">()</span>
<span class="go">[('The', 'AT'), ('Fulton', 'NP-TL'), ...]</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_sents</span><span class="p">()</span>
<span class="go">[[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'), ...], ...]</span>
<span class="gp">>>> </span><span class="n">brown</span><span class="o">.</span><span class="n">tagged_paras</span><span class="p">()</span>
<span class="go">[[[('The', 'AT'), ...]], [[('The', 'AT'), ...]], ...]</span>
</pre></div>
</div>
</section>
<section id="categorized-markdown-corpus-reader">
<h4>Categorized Markdown Corpus Reader<a class="headerlink" href="#categorized-markdown-corpus-reader" title="Link to this heading">¶</a></h4>
<p>This corpus reader class provides additional methods to select features
present in markdown documents.</p>
<p>First, let’s make a test corpus:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">root</span> <span class="o">=</span> <span class="n">make_testcorpus</span><span class="p">(</span><span class="n">ext</span><span class="o">=</span><span class="s1">'.md'</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">a</span><span class="o">=</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> # Section One</span>
<span class="gp">... </span><span class="s2"> Here's the first sentence of section one. Then the second sentence.</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> First section, second paragraph. Let's add a [link](https://example.com).</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> # Section Two</span>
<span class="gp">... </span><span class="s2"> This section is more fun. It contains an  followed by a list:</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> 1. First list item</span>
<span class="gp">... </span><span class="s2"> 2. Second list item</span>
<span class="gp">... </span><span class="s2"> """</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">b</span><span class="o">=</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> This is the second file. It starts without a section, but then adds one.</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> # Section 1</span>
<span class="gp">... </span><span class="s2"> This section has a sub-section!</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> ## Section 1a</span>
<span class="gp">... </span><span class="s2"> And here's a quote:</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> > Carpe diem</span>
<span class="gp">...</span>
<span class="gp">... </span><span class="s2"> HTML tags <em>are</em> removed.</span>
<span class="gp">... </span><span class="s2"> """</span><span class="p">)</span>
</pre></div>
</div>
<p>Now, import the <code class="docutils literal notranslate"><span class="pre">CategorizedMarkdownCorpusReader</span></code> class.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus.reader.markdown</span><span class="w"> </span><span class="kn">import</span> <span class="n">CategorizedMarkdownCorpusReader</span>
</pre></div>
</div>
<p>Note that this class requires the following Python packages:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">markdown-it-py</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">mdit-py-plugins</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">mdit-plain</span></code></p></li>
</ul>
<p>The corpus provides usual methods like <code class="docutils literal notranslate"><span class="pre">words()</span></code>, <code class="docutils literal notranslate"><span class="pre">sents()</span></code>,
<code class="docutils literal notranslate"><span class="pre">paras()</span></code>, etc. Each of these methods accepts a list of file IDs
which can be a Python list or a comma-separated string.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">corpus</span> <span class="o">=</span> <span class="n">CategorizedMarkdownCorpusReader</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="p">[</span><span class="s1">'a.md'</span><span class="p">,</span> <span class="s1">'b.md'</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">()</span>
<span class="go">['a.md', 'b.md']</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">()</span>
<span class="go">['Section', 'One', 'Here', "'", 's', 'the', 'first', ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'b.md'</span><span class="p">)</span>
<span class="go">['This', 'is', 'the', 'second', 'file', '.', 'It', ...]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'a.md, b.md'</span><span class="p">)</span> <span class="o">==</span> <span class="n">corpus</span><span class="o">.</span><span class="n">words</span><span class="p">([</span><span class="s1">'a.md'</span><span class="p">,</span> <span class="s1">'b.md'</span><span class="p">])</span>
<span class="go">True</span>
</pre></div>
</div>
<p>Here are some methods specific to the
<code class="docutils literal notranslate"><span class="pre">CategorizedMarkdownCorpusReader</span></code> class to retrieve markdown features:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">links</span><span class="p">()</span>
<span class="go">[Link(label='link', href='https://example.com', title=None)]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">images</span><span class="p">()</span>
<span class="go">[Image(label='image', src='https://example.com/image.png', title=None)]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">lists</span><span class="p">()</span>
<span class="go">[List(is_ordered=True, items=['First list item', 'Second list item'])]</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">blockquotes</span><span class="p">()</span>
<span class="go">[MarkdownBlock(content='Carpe diem')]</span>
</pre></div>
</div>
<p>The corpus can also be broken down into sections based on markdown headings:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">sections</span><span class="p">(</span><span class="s1">'a.md'</span><span class="p">)</span>
<span class="go">[MarkdownSection(content='Section One\n\nHer...'), MarkdownSection(content='Section Two\n\nThi...')]</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">corpus</span><span class="o">.</span><span class="n">sections</span><span class="p">():</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="sa">F</span><span class="s2">"</span><span class="si">{</span><span class="n">s</span><span class="o">.</span><span class="n">heading</span><span class="si">}</span><span class="s2"> (level </span><span class="si">{</span><span class="n">s</span><span class="o">.</span><span class="n">level</span><span class="si">}</span><span class="s2">)"</span><span class="p">)</span>
<span class="gp">...</span>
<span class="go">Section One (level 1)</span>
<span class="go">Section Two (level 1)</span>
<span class="go">Section 1 (level 1)</span>
<span class="go">Section 1a (level 2)</span>
</pre></div>
</div>
<section id="categories">
<h5>Categories<a class="headerlink" href="#categories" title="Link to this heading">¶</a></h5>
<p>The <code class="docutils literal notranslate"><span class="pre">CategorizedMarkdownCorpusReader</span></code> relies on YAML front matter to
read metadata defined in markdown documents. This metadata is optional,
and may define one or more categories for each document.</p>
<p>Let’s create another test corpus, this time with some metadata:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">del_testcorpus</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">root</span> <span class="o">=</span> <span class="n">make_testcorpus</span><span class="p">(</span><span class="n">ext</span><span class="o">=</span><span class="s1">'.md'</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">a</span><span class="o">=</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> ---</span>
<span class="gp">... </span><span class="s2"> tags:</span>
<span class="gp">... </span><span class="s2"> - tag1</span>
<span class="gp">... </span><span class="s2"> - tag2</span>
<span class="gp">... </span><span class="s2"> ---</span>
<span class="gp">... </span><span class="s2"> Document A: category metadata.</span>
<span class="gp">... </span><span class="s2"> """</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">b</span><span class="o">=</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> ---</span>
<span class="gp">... </span><span class="s2"> author: NLTK</span>
<span class="gp">... </span><span class="s2"> tags:</span>
<span class="gp">... </span><span class="s2"> - tag2</span>
<span class="gp">... </span><span class="s2"> - tag3</span>
<span class="gp">... </span><span class="s2"> ---</span>
<span class="gp">... </span><span class="s2"> Document B: additional metadata.</span>
<span class="gp">... </span><span class="s2"> """</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">c</span><span class="o">=</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2"> Document C: no metadata.</span>
<span class="gp">... </span><span class="s2"> """</span><span class="p">)</span>
</pre></div>
</div>
<p>Load the new corpus and see the <code class="docutils literal notranslate"><span class="pre">metadata()</span></code> and <code class="docutils literal notranslate"><span class="pre">categories()</span></code>
methods in action:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">fileids</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'a.md'</span><span class="p">,</span> <span class="s1">'b.md'</span><span class="p">,</span> <span class="s1">'c.md'</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">corpus</span> <span class="o">=</span> <span class="n">CategorizedMarkdownCorpusReader</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">fileids</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">metadata</span><span class="p">()</span>
<span class="go">[{'tags': ['tag1', 'tag2']}, {'author': 'NLTK', 'tags': ['tag2', 'tag3']}]</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">fid</span> <span class="ow">in</span> <span class="n">fileids</span><span class="p">:</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">fid</span><span class="p">,</span> <span class="n">corpus</span><span class="o">.</span><span class="n">metadata</span><span class="p">(</span><span class="n">fid</span><span class="p">))</span>
<span class="gp">...</span>
<span class="go">a.md [{'tags': ['tag1', 'tag2']}]</span>
<span class="go">b.md [{'author': 'NLTK', 'tags': ['tag2', 'tag3']}]</span>
<span class="go">c.md []</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">categories</span><span class="p">()</span>
<span class="go">['tag1', 'tag2', 'tag3']</span>
<span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">categories</span><span class="p">(</span><span class="s1">'a.md'</span><span class="p">)</span>
<span class="go">['tag1', 'tag2']</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">fileids()</span></code> method also accepts categories and returns all file
IDs that match any of the specified categories:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">corpus</span><span class="o">.</span><span class="n">fileids</span><span class="p">(</span><span class="s1">'tag2'</span><span class="p">)</span>
<span class="go">['a.md', 'b.md']</span>
<span class="gp">>>> </span><span class="n">del_testcorpus</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
</pre></div>
</div>
</section>
</section>
<section id="verbnet-corpus-reader">
<h4>Verbnet Corpus Reader<a class="headerlink" href="#verbnet-corpus-reader" title="Link to this heading">¶</a></h4>
<p>Make sure we’re picking up the right number of elements:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">verbnet</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">verbnet</span><span class="o">.</span><span class="n">lemmas</span><span class="p">())</span>
<span class="go">3621</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">verbnet</span><span class="o">.</span><span class="n">wordnetids</span><span class="p">())</span>
<span class="go">4953</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">verbnet</span><span class="o">.</span><span class="n">classids</span><span class="p">())</span>
<span class="go">429</span>
</pre></div>
</div>
<p>Selecting classids based on various selectors:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">classids</span><span class="p">(</span><span class="n">lemma</span><span class="o">=</span><span class="s1">'take'</span><span class="p">)</span>
<span class="go">['bring-11.3', 'characterize-29.2', 'convert-26.6.2', 'cost-54.2',</span>
<span class="go">'fit-54.3', 'performance-26.7-2', 'steal-10.5']</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">classids</span><span class="p">(</span><span class="n">wordnetid</span><span class="o">=</span><span class="s1">'lead%2:38:01'</span><span class="p">)</span>
<span class="go">['accompany-51.7']</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">classids</span><span class="p">(</span><span class="n">fileid</span><span class="o">=</span><span class="s1">'approve-77.xml'</span><span class="p">)</span>
<span class="go">['approve-77']</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">classids</span><span class="p">(</span><span class="n">classid</span><span class="o">=</span><span class="s1">'admire-31.2'</span><span class="p">)</span> <span class="c1"># subclasses</span>
<span class="go">['admire-31.2-1']</span>
</pre></div>
</div>
<p>vnclass() accepts filenames, long ids, and short ids:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">tostring</span><span class="p">(</span><span class="n">verbnet</span><span class="o">.</span><span class="n">vnclass</span><span class="p">(</span><span class="s1">'admire-31.2.xml'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">tostring</span><span class="p">(</span><span class="n">verbnet</span><span class="o">.</span><span class="n">vnclass</span><span class="p">(</span><span class="s1">'admire-31.2'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">c</span> <span class="o">=</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">tostring</span><span class="p">(</span><span class="n">verbnet</span><span class="o">.</span><span class="n">vnclass</span><span class="p">(</span><span class="s1">'31.2'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">a</span> <span class="o">==</span> <span class="n">b</span> <span class="o">==</span> <span class="n">c</span>
<span class="go">True</span>
</pre></div>
</div>
<p>fileids() can be used to get files based on verbnet class ids:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">fileids</span><span class="p">(</span><span class="s1">'admire-31.2'</span><span class="p">)</span>
<span class="go">['admire-31.2.xml']</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">fileids</span><span class="p">([</span><span class="s1">'admire-31.2'</span><span class="p">,</span> <span class="s1">'obtain-13.5.2'</span><span class="p">])</span>
<span class="go">['admire-31.2.xml', 'obtain-13.5.2.xml']</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">fileids</span><span class="p">(</span><span class="s1">'badidentifier'</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="gr"> . . .</span>
<span class="gr">ValueError</span>: <span class="n">vnclass identifier 'badidentifier' not found</span>
</pre></div>
</div>
<p>longid() and shortid() can be used to convert identifiers:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">longid</span><span class="p">(</span><span class="s1">'31.2'</span><span class="p">)</span>
<span class="go">'admire-31.2'</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">longid</span><span class="p">(</span><span class="s1">'admire-31.2'</span><span class="p">)</span>
<span class="go">'admire-31.2'</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">shortid</span><span class="p">(</span><span class="s1">'31.2'</span><span class="p">)</span>
<span class="go">'31.2'</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">shortid</span><span class="p">(</span><span class="s1">'admire-31.2'</span><span class="p">)</span>
<span class="go">'31.2'</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">longid</span><span class="p">(</span><span class="s1">'badidentifier'</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="gr"> . . .</span>
<span class="gr">ValueError</span>: <span class="n">vnclass identifier 'badidentifier' not found</span>
<span class="gp">>>> </span><span class="n">verbnet</span><span class="o">.</span><span class="n">shortid</span><span class="p">(</span><span class="s1">'badidentifier'</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="gr"> . . .</span>
<span class="gr">ValueError</span>: <span class="n">vnclass identifier 'badidentifier' not found</span>
</pre></div>
</div>
</section>
<section id="corpus-view-regression-tests">
<h4>Corpus View Regression Tests<a class="headerlink" href="#corpus-view-regression-tests" title="Link to this heading">¶</a></h4>
<p>Select some corpus files to play with:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">nltk.data</span>
<span class="gp">>>> </span><span class="c1"># A very short file (160 chars):</span>
<span class="gp">>>> </span><span class="n">f1</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'corpora/inaugural/README'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c1"># A relatively short file (791 chars):</span>
<span class="gp">>>> </span><span class="n">f2</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'corpora/inaugural/1793-Washington.txt'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c1"># A longer file (32k chars):</span>
<span class="gp">>>> </span><span class="n">f3</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'corpora/inaugural/1909-Taft.txt'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">fileids</span> <span class="o">=</span> <span class="p">[</span><span class="n">f1</span><span class="p">,</span> <span class="n">f2</span><span class="p">,</span> <span class="n">f3</span><span class="p">]</span>
</pre></div>
</div>
<section id="concatenation">
<h5>Concatenation<a class="headerlink" href="#concatenation" title="Link to this heading">¶</a></h5>
<p>Check that concatenation works as intended.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus.reader.util</span><span class="w"> </span><span class="kn">import</span> <span class="o">*</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">c1</span> <span class="o">=</span> <span class="n">StreamBackedCorpusView</span><span class="p">(</span><span class="n">f1</span><span class="p">,</span> <span class="n">read_whitespace_block</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">c2</span> <span class="o">=</span> <span class="n">StreamBackedCorpusView</span><span class="p">(</span><span class="n">f2</span><span class="p">,</span> <span class="n">read_whitespace_block</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">c3</span> <span class="o">=</span> <span class="n">StreamBackedCorpusView</span><span class="p">(</span><span class="n">f3</span><span class="p">,</span> <span class="n">read_whitespace_block</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">c123</span> <span class="o">=</span> <span class="n">c1</span><span class="o">+</span><span class="n">c2</span><span class="o">+</span><span class="n">c3</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">c123</span><span class="p">)</span>
<span class="go">['C-Span', 'Inaugural', 'Address', 'Corpus', 'US', ...]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">l1</span> <span class="o">=</span> <span class="n">f1</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">l2</span> <span class="o">=</span> <span class="n">f2</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">l3</span> <span class="o">=</span> <span class="n">f3</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">l123</span> <span class="o">=</span> <span class="n">l1</span><span class="o">+</span><span class="n">l2</span><span class="o">+</span><span class="n">l3</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">list</span><span class="p">(</span><span class="n">c123</span><span class="p">)</span> <span class="o">==</span> <span class="n">l123</span>
<span class="go">True</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="p">(</span><span class="n">c1</span><span class="o">+</span><span class="n">c2</span><span class="o">+</span><span class="n">c3</span><span class="p">)[</span><span class="mi">100</span><span class="p">]</span> <span class="o">==</span> <span class="n">l123</span><span class="p">[</span><span class="mi">100</span><span class="p">]</span>
<span class="go">True</span>
</pre></div>
</div>
</section>
<section id="slicing">
<h5>Slicing<a class="headerlink" href="#slicing" title="Link to this heading">¶</a></h5>
<p>First, do some tests with fairly small slices. These will all
generate tuple values.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.util</span><span class="w"> </span><span class="kn">import</span> <span class="n">LazySubsequence</span>
<span class="gp">>>> </span><span class="n">c1</span> <span class="o">=</span> <span class="n">StreamBackedCorpusView</span><span class="p">(</span><span class="n">f1</span><span class="p">,</span> <span class="n">read_whitespace_block</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">l1</span> <span class="o">=</span> <span class="n">f1</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">c1</span><span class="p">))</span>
<span class="go">21</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">c1</span><span class="p">)</span> <span class="o"><</span> <span class="n">LazySubsequence</span><span class="o">.</span><span class="n">MIN_SIZE</span>
<span class="go">True</span>
</pre></div>
</div>
<p>Choose a list of indices, based on the length, that covers the
important corner cases:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">indices</span> <span class="o">=</span> <span class="p">[</span><span class="o">-</span><span class="mi">60</span><span class="p">,</span> <span class="o">-</span><span class="mi">30</span><span class="p">,</span> <span class="o">-</span><span class="mi">22</span><span class="p">,</span> <span class="o">-</span><span class="mi">21</span><span class="p">,</span> <span class="o">-</span><span class="mi">20</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span>
<span class="gp">... </span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">21</span><span class="p">,</span> <span class="mi">22</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">60</span><span class="p">]</span>
</pre></div>
</div>
<p>Test slicing with explicit start & stop value:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">assert</span> <span class="nb">list</span><span class="p">(</span><span class="n">c1</span><span class="p">[</span><span class="n">s</span><span class="p">:</span><span class="n">e</span><span class="p">])</span> <span class="o">==</span> <span class="n">l1</span><span class="p">[</span><span class="n">s</span><span class="p">:</span><span class="n">e</span><span class="p">]</span>
</pre></div>
</div>
<p>Test slicing with stop=None:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">assert</span> <span class="nb">list</span><span class="p">(</span><span class="n">c1</span><span class="p">[</span><span class="n">s</span><span class="p">:])</span> <span class="o">==</span> <span class="n">l1</span><span class="p">[</span><span class="n">s</span><span class="p">:]</span>
</pre></div>
</div>
<p>Test slicing with start=None:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">assert</span> <span class="nb">list</span><span class="p">(</span><span class="n">c1</span><span class="p">[:</span><span class="n">e</span><span class="p">])</span> <span class="o">==</span> <span class="n">l1</span><span class="p">[:</span><span class="n">e</span><span class="p">]</span>
</pre></div>
</div>
<p>Test slicing with start=stop=None:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">list</span><span class="p">(</span><span class="n">c1</span><span class="p">[:])</span> <span class="o">==</span> <span class="nb">list</span><span class="p">(</span><span class="n">l1</span><span class="p">[:])</span>
<span class="go">True</span>
</pre></div>
</div>
<p>Next, we’ll do some tests with much longer slices. These will
generate LazySubsequence objects.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">c3</span> <span class="o">=</span> <span class="n">StreamBackedCorpusView</span><span class="p">(</span><span class="n">f3</span><span class="p">,</span> <span class="n">read_whitespace_block</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">l3</span> <span class="o">=</span> <span class="n">f3</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">c3</span><span class="p">))</span>
<span class="go">5430</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">c3</span><span class="p">)</span> <span class="o">></span> <span class="n">LazySubsequence</span><span class="o">.</span><span class="n">MIN_SIZE</span><span class="o">*</span><span class="mi">2</span>
<span class="go">True</span>
</pre></div>
</div>
<p>Choose a list of indices, based on the length, that covers the
important corner cases:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">indices</span> <span class="o">=</span> <span class="p">[</span><span class="o">-</span><span class="mi">12000</span><span class="p">,</span> <span class="o">-</span><span class="mi">6000</span><span class="p">,</span> <span class="o">-</span><span class="mi">5431</span><span class="p">,</span> <span class="o">-</span><span class="mi">5430</span><span class="p">,</span> <span class="o">-</span><span class="mi">5429</span><span class="p">,</span> <span class="o">-</span><span class="mi">3000</span><span class="p">,</span> <span class="o">-</span><span class="mi">200</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span>
<span class="gp">... </span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">200</span><span class="p">,</span> <span class="mi">3000</span><span class="p">,</span> <span class="mi">5000</span><span class="p">,</span> <span class="mi">5429</span><span class="p">,</span> <span class="mi">5430</span><span class="p">,</span> <span class="mi">5431</span><span class="p">,</span> <span class="mi">6000</span><span class="p">,</span> <span class="mi">12000</span><span class="p">]</span>
</pre></div>
</div>
<p>Test slicing with explicit start & stop value:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">assert</span> <span class="nb">list</span><span class="p">(</span><span class="n">c3</span><span class="p">[</span><span class="n">s</span><span class="p">:</span><span class="n">e</span><span class="p">])</span> <span class="o">==</span> <span class="n">l3</span><span class="p">[</span><span class="n">s</span><span class="p">:</span><span class="n">e</span><span class="p">]</span>
</pre></div>
</div>
<p>Test slicing with stop=None:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">assert</span> <span class="nb">list</span><span class="p">(</span><span class="n">c3</span><span class="p">[</span><span class="n">s</span><span class="p">:])</span> <span class="o">==</span> <span class="n">l3</span><span class="p">[</span><span class="n">s</span><span class="p">:]</span>
</pre></div>
</div>
<p>Test slicing with start=None:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">assert</span> <span class="nb">list</span><span class="p">(</span><span class="n">c3</span><span class="p">[:</span><span class="n">e</span><span class="p">])</span> <span class="o">==</span> <span class="n">l3</span><span class="p">[:</span><span class="n">e</span><span class="p">]</span>
</pre></div>
</div>
<p>Test slicing with start=stop=None:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">list</span><span class="p">(</span><span class="n">c3</span><span class="p">[:])</span> <span class="o">==</span> <span class="nb">list</span><span class="p">(</span><span class="n">l3</span><span class="p">[:])</span>
<span class="go">True</span>
</pre></div>
</div>
</section>
<section id="multiple-iterators">
<h5>Multiple Iterators<a class="headerlink" href="#multiple-iterators" title="Link to this heading">¶</a></h5>
<p>If multiple iterators are created for the same corpus view, their
iteration can be interleaved:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">c3</span> <span class="o">=</span> <span class="n">StreamBackedCorpusView</span><span class="p">(</span><span class="n">f3</span><span class="p">,</span> <span class="n">read_whitespace_block</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">iterators</span> <span class="o">=</span> <span class="p">[</span><span class="n">c3</span><span class="o">.</span><span class="n">iterate_from</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">15</span><span class="p">,</span><span class="mi">30</span><span class="p">,</span><span class="mi">45</span><span class="p">]]</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">15</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">iterator</span> <span class="ow">in</span> <span class="n">iterators</span><span class="p">:</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">%-15s</span><span class="s1">'</span> <span class="o">%</span> <span class="nb">next</span><span class="p">(</span><span class="n">iterator</span><span class="p">),</span> <span class="n">end</span><span class="o">=</span><span class="s1">' '</span><span class="p">)</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">()</span>
<span class="go">My a duties in</span>
<span class="go">fellow heavy of a</span>
<span class="go">citizens: weight the proper</span>
<span class="go">Anyone of office sense</span>
<span class="go">who responsibility. upon of</span>
<span class="go">has If which the</span>
<span class="go">taken not, he obligation</span>
<span class="go">the he is which</span>
<span class="go">oath has about the</span>
<span class="go">I no to oath</span>
<span class="go">have conception enter, imposes.</span>
<span class="go">just of or The</span>
<span class="go">taken the he office</span>
<span class="go">must powers is of</span>
<span class="go">feel and lacking an</span>
</pre></div>
</div>
</section>
</section>
<section id="seekableunicodestreamreader">
<h4>SeekableUnicodeStreamReader<a class="headerlink" href="#seekableunicodestreamreader" title="Link to this heading">¶</a></h4>
<p>The file-like objects provided by the <code class="docutils literal notranslate"><span class="pre">codecs</span></code> module unfortunately
suffer from a bug that prevents them from working correctly with
corpus view objects. In particular, although the expose <code class="docutils literal notranslate"><span class="pre">seek()</span></code>
and <code class="docutils literal notranslate"><span class="pre">tell()</span></code> methods, those methods do not exhibit the expected
behavior, because they are not synchronized with the internal buffers
that are kept by the file-like objects. For example, the <code class="docutils literal notranslate"><span class="pre">tell()</span></code>
method will return the file position at the end of the buffers (whose
contents have not yet been returned by the stream); and therefore this
file position can not be used to return to the ‘current’ location in
the stream (since <code class="docutils literal notranslate"><span class="pre">seek()</span></code> has no way to reconstruct the buffers).</p>
<p>To get around these problems, we define a new class,
<cite>SeekableUnicodeStreamReader</cite>, to act as a file-like interface to
files containing encoded unicode data. This class is loosely based on
the <code class="docutils literal notranslate"><span class="pre">codecs.StreamReader</span></code> class. To construct a new reader, we call
the constructor with an underlying stream and an encoding name:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">io</span><span class="w"> </span><span class="kn">import</span> <span class="n">StringIO</span><span class="p">,</span> <span class="n">BytesIO</span>
<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.data</span><span class="w"> </span><span class="kn">import</span> <span class="n">SeekableUnicodeStreamReader</span>
<span class="gp">>>> </span><span class="n">stream</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">(</span><span class="sa">b</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2">This is a test file.</span>
<span class="gp">... </span><span class="s2">It is encoded in ascii.</span>
<span class="gp">... </span><span class="s2">"""</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">reader</span> <span class="o">=</span> <span class="n">SeekableUnicodeStreamReader</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="s1">'ascii'</span><span class="p">)</span>
</pre></div>
</div>
<p><cite>SeekableUnicodeStreamReader</cite>s support all of the normal operations
supplied by a read-only stream. Note that all of the read operations
return <code class="docutils literal notranslate"><span class="pre">unicode</span></code> objects (not <code class="docutils literal notranslate"><span class="pre">str</span></code> objects).</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="c1"># read the entire file.</span>
<span class="go">'This is a test file.\nIt is encoded in ascii.\n'</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># rewind to the start.</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="c1"># read at most 5 bytes.</span>
<span class="go">'This '</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span> <span class="c1"># read to the end of the line.</span>
<span class="go">'is a test file.\n'</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># rewind to the start.</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">reader</span><span class="p">:</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">line</span><span class="p">))</span> <span class="c1"># iterate over lines</span>
<span class="go">'This is a test file.\n'</span>
<span class="go">'It is encoded in ascii.\n'</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># rewind to the start.</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">readlines</span><span class="p">()</span> <span class="c1"># read a list of line strings</span>
<span class="go">['This is a test file.\n', 'It is encoded in ascii.\n']</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<section id="size-argument-to-read">
<h5>Size argument to <code class="docutils literal notranslate"><span class="pre">read()</span></code><a class="headerlink" href="#size-argument-to-read" title="Link to this heading">¶</a></h5>
<p>The <code class="docutils literal notranslate"><span class="pre">size</span></code> argument to <code class="docutils literal notranslate"><span class="pre">read()</span></code> specifies the maximum number of
<em>bytes</em> to read, not the maximum number of <em>characters</em>. Thus, for
encodings that use multiple bytes per character, it may return fewer
characters than the <code class="docutils literal notranslate"><span class="pre">size</span></code> argument:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">stream</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">(</span><span class="sa">b</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2">This is a test file.</span>
<span class="gp">... </span><span class="s2">It is encoded in utf-16.</span>
<span class="gp">... </span><span class="s2">"""</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-16'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">reader</span> <span class="o">=</span> <span class="n">SeekableUnicodeStreamReader</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="s1">'utf-16'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="go">'This '</span>
</pre></div>
</div>
<p>If a read block ends in the middle of the byte string encoding a
single character, then that byte string is stored in an internal
buffer, and re-used on the next call to <code class="docutils literal notranslate"><span class="pre">read()</span></code>. However, if the
size argument is too small to read even a single character, even
though at least one character is available, then the <code class="docutils literal notranslate"><span class="pre">read()</span></code> method
will read additional bytes until it can return a single character.
This ensures that the <code class="docutils literal notranslate"><span class="pre">read()</span></code> method does not return an empty
string, which could be mistaken for indicating the end of the file.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># rewind to the start.</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># we actually need to read 4 bytes</span>
<span class="go">'T'</span>
<span class="gp">>>> </span><span class="nb">int</span><span class="p">(</span><span class="n">reader</span><span class="o">.</span><span class="n">tell</span><span class="p">())</span>
<span class="go">4</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">readline()</span></code> method may read more than a single line of text, in
which case it stores the text that it does not return in a buffer. If
this buffer is not empty, then its contents will be included in the
value returned by the next call to <code class="docutils literal notranslate"><span class="pre">read()</span></code>, regardless of the
<code class="docutils literal notranslate"><span class="pre">size</span></code> argument, since they are available without reading any new
bytes from the stream:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># rewind to the start.</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span> <span class="c1"># stores extra text in a buffer</span>
<span class="go">'This is a test file.\n'</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">reader</span><span class="o">.</span><span class="n">linebuffer</span><span class="p">)</span> <span class="c1"># examine the buffer contents</span>
<span class="go">['It is encoded i']</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># returns the contents of the buffer</span>
<span class="go">'It is encoded i'</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">reader</span><span class="o">.</span><span class="n">linebuffer</span><span class="p">)</span> <span class="c1"># examine the buffer contents</span>
<span class="go">None</span>
</pre></div>
</div>
</section>
<section id="seek-and-tell">
<h5>Seek and Tell<a class="headerlink" href="#seek-and-tell" title="Link to this heading">¶</a></h5>
<p>In addition to these basic read operations,
<cite>SeekableUnicodeStreamReader</cite> also supports the <code class="docutils literal notranslate"><span class="pre">seek()</span></code> and
<code class="docutils literal notranslate"><span class="pre">tell()</span></code> operations. However, some care must still be taken when
using these operations. In particular, the only file offsets that
should be passed to <code class="docutils literal notranslate"><span class="pre">seek()</span></code> are <code class="docutils literal notranslate"><span class="pre">0</span></code> and any offset that has been
returned by <code class="docutils literal notranslate"><span class="pre">tell</span></code>.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">stream</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">(</span><span class="sa">b</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2">This is a test file.</span>
<span class="gp">... </span><span class="s2">It is encoded in utf-16.</span>
<span class="gp">... </span><span class="s2">"""</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-16'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">reader</span> <span class="o">=</span> <span class="n">SeekableUnicodeStreamReader</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="s1">'utf-16'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
<span class="go">'This is a '</span>
<span class="gp">>>> </span><span class="n">pos</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">tell</span><span class="p">();</span> <span class="nb">print</span><span class="p">(</span><span class="n">pos</span><span class="p">)</span>
<span class="go">22</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
<span class="go">'test file.'</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="n">pos</span><span class="p">)</span> <span class="c1"># rewind to the position from tell.</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
<span class="go">'test file.'</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">seek()</span></code> and <code class="docutils literal notranslate"><span class="pre">tell()</span></code> methods work property even when
<code class="docutils literal notranslate"><span class="pre">readline()</span></code> is used.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">stream</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">(</span><span class="sa">b</span><span class="s2">"""</span><span class="se">\</span>
<span class="gp">... </span><span class="s2">This is a test file.</span>
<span class="gp">... </span><span class="s2">It is encoded in utf-16.</span>
<span class="gp">... </span><span class="s2">"""</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-16'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">reader</span> <span class="o">=</span> <span class="n">SeekableUnicodeStreamReader</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="s1">'utf-16'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="go">'This is a test file.\n'</span>
<span class="gp">>>> </span><span class="n">pos</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">tell</span><span class="p">();</span> <span class="nb">print</span><span class="p">(</span><span class="n">pos</span><span class="p">)</span>
<span class="go">44</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="go">'It is encoded in utf-16.\n'</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="n">pos</span><span class="p">)</span> <span class="c1"># rewind to the position from tell.</span>
<span class="gp">>>> </span><span class="n">reader</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="go">'It is encoded in utf-16.\n'</span>
</pre></div>
</div>
</section>
</section>
<section id="squashed-bugs">
<h4>Squashed Bugs<a class="headerlink" href="#squashed-bugs" title="Link to this heading">¶</a></h4>
<p>svn 5276 fixed a bug in the comment-stripping behavior of
parse_sexpr_block.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">io</span><span class="w"> </span><span class="kn">import</span> <span class="n">StringIO</span>
<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus.reader.util</span><span class="w"> </span><span class="kn">import</span> <span class="n">read_sexpr_block</span>
<span class="gp">>>> </span><span class="n">f</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">(</span><span class="sa">b</span><span class="s2">"""</span>
<span class="gp">... </span><span class="s2">(a b c)</span>
<span class="gp">... </span><span class="s2"># This line is a comment.</span>
<span class="gp">... </span><span class="s2">(d e f</span><span class="se">\n</span><span class="s2">g h)"""</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">read_sexpr_block</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">block_size</span><span class="o">=</span><span class="mi">38</span><span class="p">,</span> <span class="n">comment_char</span><span class="o">=</span><span class="s1">'#'</span><span class="p">))</span>
<span class="go">['(a b c)']</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">read_sexpr_block</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">block_size</span><span class="o">=</span><span class="mi">38</span><span class="p">,</span> <span class="n">comment_char</span><span class="o">=</span><span class="s1">'#'</span><span class="p">))</span>
<span class="go">['(d e f\ng h)']</span>
</pre></div>
</div>
<p>svn 5277 fixed a bug in parse_sexpr_block, which would cause it to
enter an infinite loop if a file ended mid-sexpr, or ended with a
token that was not followed by whitespace. A related bug caused
an infinite loop if the corpus ended in an unmatched close paren –
this was fixed in svn 5279</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">f</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">(</span><span class="sa">b</span><span class="s2">"""</span>
<span class="gp">... </span><span class="s2">This file ends mid-sexpr</span>
<span class="gp">... </span><span class="s2">(hello (world"""</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span> <span class="nb">print</span><span class="p">(</span><span class="n">read_sexpr_block</span><span class="p">(</span><span class="n">f</span><span class="p">))</span>
<span class="go">['This', 'file', 'ends', 'mid-sexpr']</span>
<span class="go">['(hello (world']</span>
<span class="go">[]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">f</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">(</span><span class="sa">b</span><span class="s2">"This file has no trailing whitespace."</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span> <span class="nb">print</span><span class="p">(</span><span class="n">read_sexpr_block</span><span class="p">(</span><span class="n">f</span><span class="p">))</span>
<span class="go">['This', 'file', 'has', 'no', 'trailing']</span>
<span class="go">['whitespace.']</span>
<span class="go">[]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># Bug fixed in 5279:</span>
<span class="gp">>>> </span><span class="n">f</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">(</span><span class="sa">b</span><span class="s2">"a b c)"</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span> <span class="nb">print</span><span class="p">(</span><span class="n">read_sexpr_block</span><span class="p">(</span><span class="n">f</span><span class="p">))</span>
<span class="go">['a', 'b']</span>
<span class="go">['c)']</span>
<span class="go">[]</span>
</pre></div>
</div>
<p>svn 5624 & 5265 fixed a bug in ConcatenatedCorpusView, which caused it
to return the wrong items when indexed starting at any index beyond
the first file.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">nltk</span>
<span class="gp">>>> </span><span class="n">sents</span> <span class="o">=</span> <span class="n">nltk</span><span class="o">.</span><span class="n">corpus</span><span class="o">.</span><span class="n">brown</span><span class="o">.</span><span class="n">sents</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">sents</span><span class="p">[</span><span class="mi">6000</span><span class="p">])</span>
<span class="go">['Cholesterol', 'and', 'thyroid']</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">sents</span><span class="p">[</span><span class="mi">6000</span><span class="p">])</span>
<span class="go">['Cholesterol', 'and', 'thyroid']</span>
</pre></div>
</div>
<p>svn 5728 fixed a bug in Categorized*CorpusReader, which caused them
to return words from <em>all</em> files when just one file was specified.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">reuters</span>
<span class="gp">>>> </span><span class="n">reuters</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'training/13085'</span><span class="p">)</span>
<span class="go">['SNYDER', '&', 'lt', ';', 'SOI', '>', 'MAKES', ...]</span>
<span class="gp">>>> </span><span class="n">reuters</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'training/5082'</span><span class="p">)</span>
<span class="go">['SHEPPARD', 'RESOURCES', 'TO', 'MERGE', 'WITH', ...]</span>
</pre></div>
</div>
<p>svn 7227 fixed a bug in the qc corpus reader, which prevented
access to its tuples() method</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus</span><span class="w"> </span><span class="kn">import</span> <span class="n">qc</span>
<span class="gp">>>> </span><span class="n">qc</span><span class="o">.</span><span class="n">tuples</span><span class="p">(</span><span class="s1">'test.txt'</span><span class="p">)</span>
<span class="go">[('NUM:dist', 'How far is it from Denver to Aspen ?'), ('LOC:city', 'What county is Modesto , California in ?'), ...]</span>
</pre></div>
</div>
<p>Ensure that KEYWORD from <cite>comparative_sents.py</cite> no longer contains a ReDoS vulnerability.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">re</span>
<span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">time</span>
<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">nltk.corpus.reader.comparative_sents</span><span class="w"> </span><span class="kn">import</span> <span class="n">KEYWORD</span>
<span class="gp">>>> </span><span class="n">sizes</span> <span class="o">=</span> <span class="p">{</span>
<span class="gp">... </span> <span class="s2">"short"</span><span class="p">:</span> <span class="mi">4000</span><span class="p">,</span>
<span class="gp">... </span> <span class="s2">"long"</span><span class="p">:</span> <span class="mi">40000</span>
<span class="gp">... </span><span class="p">}</span>
<span class="gp">>>> </span><span class="n">exec_times</span> <span class="o">=</span> <span class="p">{</span>
<span class="gp">... </span> <span class="s2">"short"</span><span class="p">:</span> <span class="p">[],</span>
<span class="gp">... </span> <span class="s2">"long"</span><span class="p">:</span> <span class="p">[],</span>
<span class="gp">... </span><span class="p">}</span>
<span class="gp">>>> </span><span class="k">for</span> <span class="n">size_name</span><span class="p">,</span> <span class="n">size</span> <span class="ow">in</span> <span class="n">sizes</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">9</span><span class="p">):</span>
<span class="gp">... </span> <span class="n">start_t</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span>
<span class="gp">... </span> <span class="n">payload</span> <span class="o">=</span> <span class="s2">"( "</span> <span class="o">+</span> <span class="s2">"("</span> <span class="o">*</span> <span class="n">size</span>
<span class="gp">... </span> <span class="n">output</span> <span class="o">=</span> <span class="n">KEYWORD</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="gp">... </span> <span class="n">exec_times</span><span class="p">[</span><span class="n">size_name</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_t</span><span class="p">)</span>
<span class="gp">... </span> <span class="n">exec_times</span><span class="p">[</span><span class="n">size_name</span><span class="p">]</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">exec_times</span><span class="p">[</span><span class="n">size_name</span><span class="p">])[</span><span class="mi">4</span><span class="p">]</span> <span class="c1"># Get the median</span>
</pre></div>
</div>
<p>Ideally, the execution time of such a regular expression is linear
in the length of the input. As such, we would expect exec_times[“long”]
to be roughly 10 times as big as exec_times[“short”].
With the ReDoS in place, it took roughly 80 times as long.
For now, we accept values below 30 (times as long), due to the potential
for variance. This ensures that the ReDoS has certainly been reduced,
if not removed.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">exec_times</span><span class="p">[</span><span class="s2">"long"</span><span class="p">]</span> <span class="o">/</span> <span class="n">exec_times</span><span class="p">[</span><span class="s2">"short"</span><span class="p">]</span> <span class="o"><</span> <span class="mi">30</span>
<span class="go">True</span>
</pre></div>
</div>
</section>
</section>
</section>
</section>
</div>
</div>
</div>
<footer>
<div id="footer-info">
<ul id="build-details">
<li class="footer-element">
<a href="../_sources/howto/corpus.rst.txt" rel="nofollow"> source</a>
</li>
<li class="footer-element">
<a href="https://github.com/nltk/nltk/tree/3.9.2">3.9.2</a>
</li>
<li class="footer-element">
Oct 01, 2025
</li>
</ul>
<div id="copyright">
© 2025, NLTK Project
</div>
<div id="credit">
created with <a href="http://sphinx-doc.org/">Sphinx</a> and <a href="https://github.com/tomaarsen/nltk_theme">NLTK Theme</a>
</div>
</div>
</footer>
</div>
</body>
</html>