texrex

Web document cleaning/crawl file processing tools
written by Roland Schäfer for the COW project.

Download texrex-hyperhyper beta

Download texrex User's Manual

Sourceforge Ohloh Email support

texrex is a free software for processing data files from crawls and turn them into a corpus of web documents.
Currently, it is limited to reading ARC files, but other input modules can be developed quickly.
It performs the following processing steps:

  1. read ARC files document by document
  2. filter perfect duplicates using a Bloom filter
  3. strip HTML, scripts, stylesheets
  4. extract meta information from crawl headers
  5. normalize encodings to UTF-8 (using ICU), optionally treating all ISO-8859-1 input as Win-1252
  6. convert all HTML entities to appropriate codepoints (including rogue Win-1252)
  7. detect, remove, and/or annotate boilerplate blocks using a Multi-Layer Perceptron trained on 38 features
    (This method achieves far over 90% correct decisions in our evaluations and is thus far better than the previous state of the art. To be published.)
  8. assess the text quality of the documents by looking at frequencies of short frequent word (requires language-specific models)
  9. create w-shingling document fingerprints and filter near-duplicate documents
  10. perform in-document deduplication (remove repeated paragraphs, insert a backreference to first copy)
  11. perform additional normalization (e.g., reduce diverse Unicode dashes and hyphens to the basic codepoint)
  12. write standard-compliant XML output
  13. add server IP geolocation meta information (country, region, city – currently based on GeoLite)

Technologically, the main features of texrex are:

  1. written in FreePascal (Object FPC mode)
  2. licensed under LGPL (Pascal units) and GPL (Pascal programs), as well as the licenses used by ICU and FANN for the header translations of those libraries
  3. uses multi-threading for single-machine parallelization
  4. uses simple INI files to configure processing jobs for the main tool
  5. can be run in the background, using an included IPC client to control the process
  6. depends only on two additional libraries: ICU and FANN
Valid XHTML 1.0 Strict Last updated 2013-11-17. This is an infrequently updated page. Please go to the Sourceforge page for up-to-date information.