Introduction

BOLT Arabic Discussion Forums was developed by the Linguistic Data Consortium (LDC) and consists of 813,080 discussion forum threads in Egyptian Arabic harvested from the Internet using a combination of manual and automatic processes.

The DARPA BOLT (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference. The material in this release represents the unannotated Arabic source data in the discussion forum genre.

Data

Collection was seeded based on the results of manual data scouting by native speaker annotators. Scouts were instructed to seek content in Egyptian Arabic that was original, interactive and informal. Upon locating an appropriate thread, scouts submitted the URL and some simple judgments about it to a database, via a web browser plug-in. When multiple threads from a forum were submitted, the entire forum was automatically harvested and added to the collection. The scale of the collection precluded manual review of all data. Only a small portion of the threads included in this release were manually reviewed, and it is expected that there may be some offensive or otherwise undesired content as well as some threads that contain a large amount of non-Arabic content. Language identification was performed on all threads in this corpus (using CLD2), and threads for which the results indicate a high probability of largely non-Arabic content are listed in arz_suspect_LID.txt in the docs directory of this package. It should also be noted that many threads may contain a mixture of Egyptian and other varieties of Arabic, even among the threads that are primarily Arabic.

The corpus is comprised of zipped HTML and XML files. The HTML files are a raw HTML file downloaded from the discussion thread. If the thread spanned multiple URLs, it was stored as a concatenation of the downloaded HTML files. The XML files were converted from the raw HTML.

 

Acknowledgement

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Samples

Please view this html sample and xml sample.

Updates

None at this time.

Portions © 2018 Trustees of the University of Pennsylvania

Introduction

GALE Arabic-English Word Alignment -- Broadcast Training Part 2 was developed by the Linguistic Data Consortium (LDC) and contains 215,923 tokens of word aligned Arabic and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation.

Other releases available in this series are:

• GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web (LDC2012T16)


• GALE Chinese-English Word Alignment and Tagging Training Part 2 -- Newswire (LDC2012T20)


• GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web (LDC2012T24)


• GALE Chinese-English Word Alignment and Tagging Training Part 4 -- Web (LDC2013T05)


• GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 1 (LDC2013T23)


• GALE Arabic-English Word Alignment Training Part 1 -- Newswire and Web (LDC2014T05)


• GALE Arabic-English Word Alignment Training Part 2 -- Newswire (LDC2014T10)


• GALE Arabic-English Word Alignment Training Part 3 -- Web (LDC2014T14)


• GALE Arabic-English Word Alignment -- Broadcast Training Part 1 (LDC2014T19)

Data

This release consists of Arabic source broadcast news and broadcast conversation data collected by LDC from 2007-2009. The distribution by genre, words, tokens and segments appears below:

Language	Genre	Files	Words	Tokens	Segments
Arabic	BC	369	97,514	129,233	7,941
Arabic	BN	40	70,635	86,400	3,752
Totals		409	168,149	215,923	11,693

 

Note that word count is based on the untokenized Arabic source, and token count is based on the tokenized Arabic source.

The Arabic word alignment tasks consisted of the following components:

• Normalizing tokenized tokens as needed


• Identifying different types of links


• Identifying sentence segments not suitable for annotation


• Tagging unmatched words attached to other words or phrases

Samples

Please view the following samples:

• English Raw


• English Token


• Arabic Raw


• Arabic Token


• Word Alignment

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2008-2009 Al Arabiyah, © 2008 Abu Dhabi TV, Al Alam News Channel, Al Baghdadya TV, Al Hiwar, Aljazeera, Al Sharqiya, Dubai TV, Nile TV, Oman TV, PAC Ltd, Saudi TV, Syria TV, © 2008-2009, 2014 Trustees of the University of Pennsylvania

Introduction

GALE Arabic-English Word Alignment -- Broadcast Training Part 1 was developed by the Linguistic Data Consortium (LDC) and contains 267,257 tokens of word aligned Arabic and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation.

Other releases available in this series are:

• GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web (LDC2012T16)


• GALE Chinese-English Word Alignment and Tagging Training Part 2 -- Newswire (LDC2012T20)


• GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web (LDC2012T24)


• GALE Chinese-English Word Alignment and Tagging Training Part 4 -- Web (LDC2013T05)


• GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 1 (LDC2013T23)


• GALE Arabic-English Word Alignment Training Part 1 -- Newswire and Web (LDC2014T05)


• GALE Arabic-English Word Alignment Training Part 2 -- Newswire (LDC2014T10)


• GALE Arabic-English Word Alignment Training Part 3 -- Web (LDC2014T14)

Data

This release consists of Arabic source broadcast news and broadcast conversation data collected by LDC from 2007-2009. The distribution by genre, words, tokens and segments appears below:

 

Language	Genre	Files	Words	Tokens	Segments
Arabic	BC	231	79,485	103,816	4,114
Arabic	BN	92	131,789	163,441	7,227
Totals		323	211,274	267,257	11,341

 

Note that word count is based on the untokenized Arabic source, and token count is based on the tokenized Arabic source.

The Arabic word alignment tasks consisted of the following components:

• Normalizing tokenized tokens as needed


• Identifying different types of links


• Identifying sentence segments not suitable for annotation


• Tagging unmatched words attached to other words or phrases

Samples

Please view the following samlpes:

• English Raw


• English Token


• Arabic Raw


• Arabic Token


• Word Alignment

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2008 Abu Dhabi TV, © 2008 Al Alam News Channel, © 2008-2009 Al Arabiyah, © 2008 Al Baghdadya TV, © 2008-2009 Al Fayha, © 2008-2009 Al Hiwar, © 2008 Al Iraqiyah, © 2008-2009 Al Ordiniyah, © 2008 Bahrain TV, © 2008-2009 Dubai TV, © 2008 Nile TV, © 2008 Oman TV, © 2008 PAC Ltd, © 2008-2009 Saudi TV, © 2008 Syria TV, © 2008-2009 Tunisian National Television, © 2008 Yemen TV, © 2007-2009, 2014 Trustees of the University of Pennsylvania

Introduction

GALE Arabic-English Word Alignment Training Part 3 -- Web was developed by the Linguistic Data Consortium (LDC) and contains 217,158 tokens of word aligned Arabic and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation.

Other releases available in this series are:

• GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web (LDC2012T16)


• GALE Chinese-English Word Alignment and Tagging Training Part 2 -- Newswire (LDC2012T20)


• GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web (LDC2012T24)


• GALE Chinese-English Word Alignment and Tagging Training Part 4 -- Web (LDC2013T05)


• GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 1 (LDC2013T23)


• GALE Arabic-English Word Alignment Training Part 1 -- Newswire and Web (LDC2014T05)


• GALE Arabic-English Word Alignment Training Part 2 -- Newswire (LDC2014T10)

Data

This release consists of Arabic source web data collected by LDC. The distribution by genre, words, character tokens and segments appears below:

Language	Genre	Files	Words	CharTokens	Segments
Arabic	WB	2,449	154,144	217,158	7,332

 

Note that word count is based on the untokenized Arabic source, and token count is based on the tokenized Arabic source.

The Arabic word alignment tasks consisted of the following components:

• Normalizing tokenized tokens as needed


• Identifying different types of links


• Identifying sentence segments not suitable for annotation


• Tagging unmatched words attached to other words or phrases

Samples

Please view the following sampls:

• English Raw


• English Token


• Arabic Raw


• Arabic Token


• World Alignment

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2014 Trustees of the University of Pennsylvania

Introduction

GALE Arabic-English Word Alignment Training Part 2 -- Newswire was developed by the Linguistic Data Consortium (LDC) and contains 162,359 tokens of word aligned Arabic and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation.

Other releases available in this series are:

• GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web (LDC2012T16)


• GALE Chinese-English Word Alignment and Tagging Training Part 2 -- Newswire (LDC2012T20)


• GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web (LDC2012T24)


• GALE Chinese-English Word Alignment and Tagging Training Part 4 -- Web (LDC2013T05)


• GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 1 (LDC2013T23)


• GALE Arabic-English Word Alignment Training Part 1 -- Newswire and Web (LDC2014T05)

Data

This release consists of Arabic source newswire collected by LDC in 2004 - 2006 and 2008. The distribution by genre, words, character tokens and segments appears below:

 

Language	Genre	Files	Words	CharTokens	Segments
Arabic	NW	1,126	112,318	162,359	5,349

Note that word count is based on the untokenized Arabic source, and token count is based on the tokenized Arabic source.

The Arabic word alignment tasks consisted of the following components:

• Identifying and correcting incorrectly tokenized tokens


• Identifying different types of links


• Identifying sentence segments not suitable for annotation, such as those that were blank, incorrectly-segmented or containing other languages


• Tagging unmatched words attached to other words or phrases

Samples

Please view the following samples:

• English Raw


• English Token


• Arabic Raw


• Arabic Token


• Word Alignment

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2004-2006, 2008 Agence France Presse, © 2008 Al-Ahram, © 2008 Al Hayat, © 2008 Al-Quds Al-Arabi, © 2008 An Nahar, © 2008 Asharq Al-Awsat, © 2006, 2008 Assabah, © 2004-2006 Xinhua News Agency, © 2004-2006, 2008, 2014 Trustees of the University of Pennsylvania

GALE Arabic-English Parallel Aligned Treebank -- Web Training was developed by the Linguistic Data Consortium (LDC) and contains 69,766 tokens of word aligned Arabic and English parallel text with treebank annotations. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Parallel aligned treebanks are treebanks annotated with morphological and syntactic structures aligned at the sentence level and the sub-sentence level. Such data sets are useful for natural language processing and related fields, including automatic word alignment system training and evaluation, transfer-rule extraction, word sense disambiguation, translation lexicon extraction and cultural heritage and cross-linguistic studies. With respect to machine translation system development, parallel aligned treebanks may improve system performance with enhanced syntactic parsers, better rules and knowledge about language pairs and reduced word error rate.

In this release, the source Arabic data was translated into English. Arabic and English treebank annotations were performed independently. The parallel texts were then word aligned.

LDC previously released Arabic-English Parallel Aligned Treebanks as follows:

• Newswire


• Broadcast News Part 1


• Broadcast News Part 2

Data

This release consists of Arabic source web data (newsgroups, weblogs) collected by LDC in 2004 and 2005. All data is encoded as UTF-8. A count of files, words, tokens and segments is below.

Language	Files	Words	Tokens	Segments
Arabic	162	46,710	69,766	3,178

Note: Word count is based on the untokenized Arabic source, token count is based on the ATB-tokenized Arabic source.

The purpose of the GALE word alignment task was to find correspondences between words, phrases or groups of words in a set of parallel texts. Arabic-English word alignment annotation consisted of the following tasks:

• Identifying different types of links: translated (correct or incorrect) and not translated (correct or incorrect)


• Identifying sentence segments not suitable for annotation, e.g., blank segments, incorrectly-segmented segments, segments with foreign languages


• Tagging unmatched words attached to other words or phrases

This release contains four types of files - raw, tokenized, treebank, and wa. The raw format contains the original Arabic and English sentences without any annotation. The tokenized format is the treebank tokenized version of the raw data. It may contain Empty Category tokens (treebank leaves that have the POS label -NONE-). The treebank and wa files are treebank and word alignment annotations on the tokenized files.

Samples

Please view the following samples:

• English Raw


• English Token


• English Treebank


• Arabic Raw


• Arabic Token


• Arabic Treebank


• Word Alignment

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2004-2005, 2014 Trustees of the University of Pennsylvania

Introduction

GALE Arabic-English Word Alignment Training Part 1 -- Newswire and Web was developed by the Linguistic Data Consortium (LDC) and contains 344,680 tokens of word aligned Arabic and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation.

Other releases available in this series are:

• GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web (LDC2012T16)


• GALE Chinese-English Word Alignment and Tagging Training Part 2 -- Newswire (LDC2012T20)


• GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web (LDC2012T24)


• GALE Chinese-English Word Alignment and Tagging Training Part 4 -- Web (LDC2013T05)


• GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 1 (LDC2013T23)

Data

This release consists of Arabic source newswire and web data collected by LDC in 2006 - 2008. The distribution by genre, words, character tokens and segments appears below:

 

Language</TD>	Genre	Docs	Words	CharTokens	Segments
Arabic	WB	119	59,696	81,620	4,383
Arabic	NW	717	198,621	263,060	8,423

Note that word count is based on the untokenized Arabic source, and token count is based on the tokenized Arabic source.

The Arabic word alignment tasks consisted of the following components:

• Normalizing tokenized tokens as needed


• Identifying different types of links


• Identifying sentence segments not suitable for annotation


• Tagging unmatched words attached to other words or phrases

Samples

Please view the following samples

• English Raw


• English Token


• Arabic Raw


• Arabic Token


• Word Alignment

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2006, 2008 Agence France Presse, © 2006-2008 Al-Ahram, © 2006-2008 Al Hayat, © 2006-2008 Al-Quds Al-Arabi, © 2006-2008 An Nahar, © 2006-2008 Asharq Al-Awsat, © 2007 Assabah, © 2006 Xinhua News Agency, © 2006-2008, 2014 Trustees of the University of Pennsylvania

Introduction

GALE Arabic-English Parallel Aligned Treebank -- Broadcast News Part 2 was developed by the Linguistic Data Consortium (LDC) and contains 141,058 tokens of word aligned Arabic and English parallel text with treebank annotations. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Parallel aligned treebanks are treebanks annotated with morphological and syntactic structures aligned at the sentence level and the sub-sentence level. Such data sets are useful for natural language processing and related fields, including automatic word alignment system training and evaluation, transfer-rule extraction, word sense disambiguation, translation lexicon extraction and cultural heritage and cross-linguistic studies. With respect to machine translation system development, parallel aligned treebanks may improve system performance with enhanced syntactic parsers, better rules and knowledge about language pairs and reduced word error rate.

The source Arabic data was translated into English. Arabic and English treebank annotations were performed independently. The parallel texts were then word aligned. The material in this corpus corresponds to a portion of the Arabic treebanked data in Arabic Treebank - Broadcast News v1.0 (LDC2012T07).

LDC previously released GALE Arabic-English Parallel Aligned Treebank -- Broadcast News Part 1 (LDC2013T14).

Data

The source data consists of Arabic broadcast news programming collected by LDC in 2007 and 2008 from Al Arabiya, Abu Dhabi TV, Al Baghdadya TV, Al Fayha, Alhurra, Al Iraqiyah, Aljazeera, Al Ordiniyah, Al Sharqiya, Dubai TV, Oman TV, Radio Sawa and Saudi TV. All data is encoded as UTF-8. A count of files, words, tokens and segments is below.

Language</TD>	Files	Words	Tokens	Segments
Arabic	31	110,690	141,058	7,102

Note: Word count is based on the untokenized Arabic source. Token count is based on the ATB-tokenized Arabic source.

The purpose of the GALE word alignment task was to find correspondences between words, phrases or groups of words in a set of parallel texts. Arabic-English word alignment annotation consisted of the following tasks:

• Identifying different types of links: translated (correct or incorrect) and not translated (correct or incorrect)
• Identifying sentence segments not suitable for annotation, e.g., blank segments, incorrectly-segmented segments, segments with foreign languages
• Tagging unmatched words attached to other words or phrases

This release contains four types of files - raw, tokenized, treebank, and wa. The raw format contains the original Arabic and English sentences without any annotation. The tokenized format is the treebank tokenized version of the raw data which may contain Empty Category tokens (treebank leaves that have the POS label -NONE-). The treebank and wa files are treebank and word alignment annotations on the tokenized files.

Samples

Please view the following samples

• English Raw
• English Token
• English Treebank
• Arabic Raw
• Arabic Token
• Arabic Treebank
• Word Alignment

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2007-2008 Abu Dhabi TV, © 2007 Al Arabiya, © 2008 Al Baghdadya TV, © 2008 Al Fayha, © 2008 Al Iraqiyah, © 2007 Aljazeera, © 2007 Al Ordiniyah, © 2008 Al Sharqiya, © 2008 Dubai TV, © 2008 Oman TV, © 2008 Saudi TV, © 2007-2008, 2012, 2014 Trustees of the University of Pennsylvania

Introduction

MADCAT (Multilingual Automatic Document Classification Analysis and Translation) Phase 3 Training Set contains all training data created by the Linguistic Data Consortium (LDC) to support Phase 3 of the DARPA MADCAT Program. The data in this release consists of handwritten Arabic documents, scanned at high resolution and annotated for the physical coordinates of each line and token. Digital transcripts and English translations of each document are also provided, with the various content and annotation layers integrated in a single MADCAT XML output.

The goal of the MADCAT program is to automatically convert foreign text images into English transcripts. MADCAT Phase 3 data was collected from Arabic source documents in three genres: newswire, weblog and newsgroup text. Arabic speaking scribes copied documents by hand, following specific instructions on writing style (fast, normal, careful), writing implement (pen, pencil) and paper (lined, unlined). Prior to assignment, source documents were processed to optimize their appearance for the handwriting task, which resulted in some original source documents being broken into multiple pages for handwriting. Each resulting handwritten page was assigned to up to five independent scribes, using different writing conditions.

The handwritten, transcribed documents were next checked for quality and completeness, then each page was scanned at a high resolution (600 dpi, greyscale) to create a digital version of the handwritten document. The scanned images were then annotated to indicate the physical coordinates of each line and token. Explicit reading order was also labeled, along with any errors produced by the scribes when copying the text.

The final step was to produce a unified data format that takes multiple data streams and generates a single MADCAT XML output file which contains all required information. The resulting madcat.xml file contains distinct components: a text layer that consists of the source text, tokenization and sentence segmentation, an image layer that consists of bounding boxes, a scribe demographic layer that consists of scribe ID and partition (train/test) and a document metadata layer.

LDC has also released:

• MADCAT Phase 1 Training Set (LDC2012T15)


• MADCAT Phase 2 Training Set (LDC2013T09)


• MADCAT Chinese Pilot Training Set (LDC2014T13)

Data

This release includes 4,540 annotation files in both GEDI XML and MADCAT XML formats (gedi.xml and madcat.xml) along with their corresponding scanned image files in TIFF format. The annotation results in GEDI XML files include ground truth annotations and source transcripts.

Files are named as follows:

• galeID_page#_scribeID.{tif|gedi.xml|madcat.xml}

Samples

Please view the following samples.

• MADCAT XML


• GEDI XML


• Raw Image

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, MADCAT Program No. HR0011-08-1-004 and GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2006 Agence France Presse, Al-Ahram, Al Hayat, Al Quds-Al Arabi, An Nahar, Asharq Al-Awsat, Assabah, Xinhua News Agency, © 2006, 2013 Trustees of the University of Pennsylvania

GALE Arabic-English Parallel Aligned Treebank -- Broadcast News Part 1 was developed by the Linguistic Data Consortium (LDC) and contains 115,826 tokens of word aligned Arabic and English parallel text with treebank annotations. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Parallel aligned treebanks are treebanks annotated with morphological and syntactic structures aligned at the sentence level and the sub-sentence level. Such data sets are useful for natural language processing and related fields, including automatic word alignment system training and evaluation, transfer-rule extraction, word sense disambiguation, translation lexicon extraction and cultural heritage and cross-linguistic studies. With respect to machine translation system development, parallel aligned treebanks may improve system performance with enhanced syntactic parsers, better rules and knowledge about language pairs and reduced word error rate.

In this release, the source Arabic data was translated into English. Arabic and English treebank annotations were performed independently. The parallel texts were then word aligned. The material in this corpus corresponds to a portion of the Arabic treebanked data in Arabic Treebank - Broadcast News v1.0 (LDC2012T07).

Data

The source data consists of Arabic broadcast news programming collected by LDC in 2005 and 2006 from Alhurra, Aljazeera and Dubai TV. All data is encoded as UTF-8. A count of files, words, tokens and segments is below.

Language	Files	Words	Tokens	Segments
Arabic	28	89,213	115,826	4,824

Note: Word count is based on the untokenized Arabic source. Token count is based on the ATB-tokenized Arabic source.

The purpose of the GALE word alignment task was to find correspondences between words, phrases or groups of words in a set of parallel texts. Arabic-English word alignment annotation consisted of the following tasks:

• Identifying different types of links: translated (correct or incorrect) and not translated (correct or incorrect)
• Identifying sentence segments not suitable for annotation, e.g., blank segments, incorrectly-segmented segments, segments with foreign languages
• Tagging unmatched words attached to other words or phrases

This release contains four types of files - raw, tokenized, treebank, and wa. The raw format contains the original Arabic and English sentences without any annotation. The tokenized format is the treebank tokenized version of the raw data which may contain Empty Category tokens (treebank leaves that have the POS label -NONE-). The treebank and wa files are treebank and word alignment annotations on the tokenized files.

Samples

Please view the below samples.

• English Raw
• English Token
• English Tree
• Arabic Raw
• Arabic Token
• Arabic Tree
• Word Alignment

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

None at this time.

Portions © 2005-2006 Aljzaeera, © 2005 Dubai TV, © 2005-2006, 2012, 2013 Trustees of the University of Pennsylvania