What is a corpus, and how is it used in NLP?

A corpus (plural corpora), also known as a text corpus in linguistics, is usually a large collection of texts, and it could be compared to a database in which each text is a record. It is often designed to contain various types of utterances, registers, and genres as examples of language that naturally occurs in a specific context. In natural language processing (NLP), corpora are used to train algorithms and develop statistical models. These corpora can be used by linguists, lexicographers, data scientists, and experts in NLP for various tasks, including word frequency analysis, part-of-speech tagging, and text classification. As an essential tool for anyone working with NLP, their implementation can be as varied as creating text-to-speech modules or a system for machine translation. However, not all corpora are the same, and each helps accomplish NLP tasks differently.

Corpora use cases

In the field of linguistics, a corpus is a large and structured set of texts (nowadays, usually electronically stored and processed). The texts in a corpus have been selected to represent a particular language or subject matter. The notion of a corpus has been helpful in computational linguistics, where corpus-based methods are used for statistical analysis and hypothesis testing on the data, checking the number of occurrences, or even validating linguistic rules within the confines of a specific language territory. Corpora are used to perform research in many different disciplines, not just linguistics. For example, in the field of medicine, corpora are used to help researchers develop new treatments and drugs. In the field of law, corpora can be used to help lawyers find relevant cases and precedents. And in the field of history, corpora can be used to help historians find primary sources for their research.

One of the most famous corpora is the Brown Corpus, which was compiled at Brown University in the 1960s with 500 texts, each containing a little over 2,000 words for a total of 1 million words of American English sampled from 15 different categories. Other popular corpora include the British National Corpus (BNC), created by Oxford University Press, which contains 100 million words, and the Corpus of Contemporary American English (COCA), which was initially released in 2008 and, later on, received constant additions until 2019, with it now having over 1 billion words, making it the most extensive open-source corpus of American English.

Types of corpora

Different types of corpora can be classified according to their content, size, and structure. However, a corpus can also have other characteristics or properties for its organization, and often a corpus will fit into more than one classification.

Different classifications of corpora

Corpora classifications based on content

Text corpora are the most common type of corpora that contain texts from different sources.

Speech corpora contain recordings of people speaking and verbatim audio transcriptions, and are often used to study how people speak a particular language or to develop speech recognition software.

Image corpora contain images to develop computer vision algorithms, and usually, each image is tagged to allow for identification.

Video corpora include videos and are used to create algorithms for tracking objects on video.

Corpora classifications based on size

Small corpora typically comprise just a few texts and can be used for specific research tasks. For example, small corpora of medical texts might be used to study a specific disease.

Large corpora are composed of hundreds or even millions of texts and are often used for general research tasks, such as studying the overall patterns of a language.

Corpora classifications based on structure

Monolingual corpora are the most common corpora classification and contain texts only from a single language source.

Multilingual corpora, simply put, contain more than one language. They can be classified further based on how the text was created and the relationship between both languages.

Parallel corpora are made from two or more monolingual corpora where one corpus is the source and the second one will be a direct translation. In this type of corpora, both languages will be aligned to have matching segments at the paragraph or sentence level.

Comparable corpora are made of two or more monolingual corpora built using the same principles and, therefore, offer similar results. However, as the text is not a translation of each other, they are not aligned.

Corpora classifications based on purpose and other factors

General corpora contain various types of texts that can be utilized in different research fields, offering a baseline resource for general studies. The source can be written text or spoken language, along with transcriptions.

Specialized corpora are designed for specific research goals containing a particular text type. These constraints can refer to a specific time frame or a particular subject, among other things.

Diachronic corpora contain language data from different historical periods, and language experts use these to study the changes and development in a specific language.

Synchronic corpora would be the opposite of diachronic corpora, and all texts must be compiled from the same period.

Monitor corpora are diachronic and expandable. They are continuously updated to reflect the changes in language usage by incorporating new words and expressions.

National corpora contain texts that represent language used in a specific country.

Reference corpora, in general terms, are used as the base of comparison with other corpora. However, these are expected to be large general corpora that offer comprehensive coverage, which the community of users can regard as the standard for the particular use case.

Learner corpora include samples produced by non-native speakers of a language. This type of corpora allows researchers to compare the texts created by native speakers against those produced by language learners.

Developmental corpora contain language data from monolingual speakers at different stages in their language development. These can track and understand first language acquisition and vocabulary development.

Raw corpora provide no annotations or additional information and are given as originally collected.

Annotated corpora contain texts annotated with information about their structure, content, or meaning. For example, a corpus of medical texts might be annotated with information about the diseases mentioned in each text. Annotated corpora are often used to develop computational linguistics applications, such as question-answering systems.

So, how do you build a corpus?

As mentioned, a corpus is an extensive collection of texts. Building one provides an essential resource to investigate language and the learning data necessary to create different tools that can be implemented in numerous applications. Here are some steps on how to go about building a corpus for your specific project needs.

Processes in corpora building

1. Define the scope

Decide what kind of corpus you want to create. As there are many different types of corpora, each type serves a specific purpose. Understanding exactly what kind of data you need is the first step to building an effective corpus for your project.

2. Define the format

Collect texts in whatever format your project requires. The text collection could be digital (e.g., websites or other digitally stored files) or physical (e.g., books or other printed documents). The collection stage could also require samples of spoken language that will need to be transcribed before the text can be used.

3. Organize the data

Organize your texts into a coherent structure. Doing this will make it easier to search and analyze the language data in your corpus later. Having your text divided into different categories or topics is a common first approach for text organization.

4. Use the right tools

Use a corpus-building tool or service to help create and manage your corpus. Many software options and platforms are available to help you collect or even generate new text for your project.

5. Annotate the data

Annotate your corpus with metadata. Tagging or annotations will describe the contents of each text and can be used to categorize and search the corpus for further implementation.

6. Analyze the data

Explore your corpus! Once you have built it, you can start to carry out all sorts of interesting analyses, such as looking at word frequencies or finding collocations and interesting language patterns.

How can BAVL help you build the perfect corpus for your NLP project?

In short, a corpus is a large set of language training data for statistical NLP applications. Here at BAVL, we have all the tools you need to collect and annotate text and voice data. And suppose your project requires generating new data from spontaneous communication or based on different scenarios. In that case, with BAVL you can generate new training data on any language and subject based on your specific needs and even translate an existing monolingual corpus to create a bilingual one. Visit bavl.io or schedule a call with us today to book a demo and build the perfect corpus for your project!