Summary of the Summer School “Coding for Language Communities 2014”

[Intro]

Here is a list of the projects that were implemented during the summer school:

Project: Roma dictionary

Team Members: Bruce Birch, Ricardo Filipe, Benoit Legouy, Pedro Manha, Eleni Ntalambyra Elodie Paquette, Claire Ulrich

With graphic design assistance from Rosana Margarida

Brief Project Description

ROMA Dictionary (working title) is an app (iOS, Android, Web) which is aimed at crowdsourcing a corpus (text and audio) of Roma as spoken and written by Roma teenagers on Facebook. In a recent interview for the website “Czech Books”, writer Lukáš Houdek spoke about the upsurge in the use of Romani online:
 
“I think Facebook is very important in the life of Roma, because part of their families live abroad: in England, Canada or the US. That’s how they keep in touch and how they get information as well. It’s very interesting that thanks to Facebook many Roma have started to write in Romany, because until now there have not been so many opportunities to write Romany. It was always a language of verbal communication, but now they have to communicate in writing. In this way they are becoming more used to writing and reading Romany.”
 
An important inspiration for this project is the “Urban Dictionary” project, a crowdsourcing site originally set up as a repository for slang or cultural words or phrases in (American) English which couldn’t easily be found elsewhere. One of the first definitions on the site was “the man”, defined as “the faces of ‘the establishment’ put in place to ‘bring us down’.” The intention was clearly to give a voice to “non-establishment”, or “anti-establishment” sectors of society. This remains a major feature of the site, which has now broadened out into a general dictionary containing over 8 million entries.
 
The position of Roma throughout the world as outsiders means that they are by definition a “non-establishment” and marginalized group who have suffered from discrimination over a period of centuries.
 
The idea is therefore to take advantage of the increasing usage of written Romany online, by providing a safe place to deposit risky and funny language, which at the same time can grow into a useful resource both for the Romany-writing community and potentially for linguistic (and other) research.
 
The app will be monolingual, a space for interaction between Romani speakers. Definitions are not standard dictionary definitions containing information such as part of speech, IPA transcription, etymology, etc. Rather they are impressionistic depictions of terms and their usage with an emphasis on example sentences and mini-dialogues which are highly evocative of cultural context.

The Roma dictionary was the winning project of our internal vote for the best project!

Project: Finding near-matches for spellchecker improvement

The goal of the software is to identify alternative spellings of the same word, to facilitate the development and refinement of spell-checking tools for languages with widely disregarded spelling standards, or even to guide the development of orthography standards for languages with lack them. This overcomes a shortcoming with corpus-based approaches to these tasks, in which the words in the corpus must be accepted as correctly spelled, even when they contain typos or are not consistent with each other. The software, written in Python, can generate clusters of spellings which are likely to be from the same word and can also generate a list of the most common spelling alternations.  The project, which could grow to include other related tools, is dubbed “koreksyon” (“proofreading” in Haitian Creole). Some results for Haitian Creole (exploring alternations of pro/pwo and mp/np) are given in https://docs.google.com/spreadsheets/d/1F2empVjzKZMnm9_fAaXUeEWDtnyJPy-CB3X0tFUREsw/edit?usp=sharing.

The code is published on Github at https://github.com/reokatoa/koreksyon.

Project: Real time tweet map: A visualization of lesser-resourced languages

Team member: Netta Ben-Meir, Caroline Borowski Nadia Paraskevoudi

About the project

Given the focus of the course, we wanted to create a project that would provide us with information about the use of specific languages in social media. Considering that social media is the meeting point people all over the world today, we decided to use data from Twitter to document languages considered to be under-resourced. Social media research should be regarded as an interdisciplinary area. Undoubtedly, there is a clear need for new ways to access and demonstrate useful information for researchers and students in the fields of linguistics, language documentation, and anthropology. This map can be used as a tool to capture all the data that may reveal the community network of people using under-resourced languages in social media. This project should not be seen as a tool designed especially for Basque. Contrary to this, it could be used for a wide variety of under-resourced languages to provide information about the existing networks among people using them. The idea of creating a map that displays the location of every user tweeting in Basque constitutes an effort to capture the data described above and visualize it. Visualization of data is part of the interdisciplinary field of Digital Humanities, a field that flourished in the beginning of the 21st century, as it was an inevitable consequence of the culture of digitization. The abstract visual representations used in various projects in this field are a useful tool for analyzing textual content as well as related metadata such as spatiotemporal information. Therefore this project should be regarded as an interdisciplinary work which was made possible by combining knowledge, methods and tools from three different fields: linguistics, computer science, and digital humanities.

Methods and Tools

The project consisted of three parts:

• First, we created a twitter application (dev.twitter.com) in order to get streaming data from twitter. In particular all the data was collected using the Twitter streaming application programming interface (Streaming API), which enabled us to filter the streaming data with specific keywords, user lists, etc. For the purposes of this project we used the python package tweepy, which supports all three twitter APIs. (http://tweepy.readthedocs.org/en/v2.3.0/) It is important to mention that before we started working on our code, we accessed two different corpora which have proven to be very useful for this project. The first corpus consisted of a wordlist of the most frequent words, affixes and suffixes in Basque, and made it possible for us to extract Basque tweets.
• The second corpus included a list of users who are known to tweet in Basque regularly.In addition to their username, this list included the latitude and longitude coordinates of each user which we used to update our map every time someone posted a new tweet. This list proved to be very important, as the feature to add location information to the tweets is off by default, meaning that there are many users whose coordinates are not being displayed when they post.
• At last, we created a map using Flask, a web application framework written in Python and based on Jinja2, which is a full featured template engine for Python. In particular, we created a html page which we serve to the user from the Flask Web App. In that page (index.html) we used the google maps API to display the map and create the points. The webpage polls the Flask Web App every second to check and get new data.

The whole project is available on Github: https://github.com/nadje/Real-time-map-for-tweets- written-in-Basque

Acknowledgements

This project was made possible through the help and support of everyone participating in the summer school “Coding for language communities”. First and foremost, we would like to thank Vera Ferreira, Peter Bouda, Rita Pedro, Felix Rau, Kevin Scannell and Dorothee Beermann for organizing this summer school. They gave us the opportunity to enhance our knowledge about projects working on under-resourced languages by collaborating with people from different disciplines. Especially, we would like to thank Kevin Scannell for contributing to our project by providing us with the Basque user corpus mentioned above. Secondly, we would like to thank José Ramom Pichel for his feedback. Finally, we would like to thank Ricardo Felipe. His help and support have proven invaluable throughout this process.