Handling Big Manuscript Data
Elpida Perdiki and Maria Konstantinidou
This contribution describes a series of Handwritten Text Recognition (HTR) tests performed on medieval Greek manuscripts and run on Transkribus, a platform for collaboratively transcribing and retrieving historical documents.  Transkribus was chosen on the basis of its suitability for our research, its user-friendly interface, and its accessibility.  We mainly used the HTR component of the platform.
Our research involves various works by John Chrysostom. They are distributed in ca. 3000 manuscripts, each surviving in at least two recensions, one being a revision of the other. We seek to classify the manuscripts according to the version they transmit. Regardless of the classification method that will be used (String Matching, Neural Language Model, or other, non-traditional methods),  a digital transcription is required. Due to the size of the task, we use automated transcription, HTR. For classification purposes, only a minimum of 80% accuracy is required. Even a Character Error Rate (CER) of more than 20% might be adequate, since much of the erroneous characters involve spaces,  accents and breathings,  which can be easily eliminated in subsequent analysis.
Obtaining large training data sets for such texts is not straightforward given the scarcity of digital or digitized editions and the way information is presented (in negative or mixed type apparatus). Furthermore, matching these extracted, decontextualised data to the respective manuscript images requires a high level of human interference.  One way of creating training data is by using HTR-generated transcriptions, smoothed manually,  although it is not advised by Transkribus, which recommends accurate transcriptions as ground truth.  Other methods include crowdsourcing (requiring additional resources) and data augmentation. 
Manually producing (at least partial) transcriptions is, therefore, inevitable. In an effort to limit manual input, we experimented on: (a) training data size reduction up to 93%,  (b) transcribing new manuscripts using existing models trained entirely on other manuscripts,  and (c) training of mixed models using very small samples from multiple manuscripts.
Transkribus does provide for multiple scribes in a single archive but recommends larger training sets in such cases.  In addition to the multiple hands involved, our material lacks the uniformity often found among documents of a single archive. On the other hand, our research benefits from transcribing the same work many times, and from a relative script uniformity. 
The performance of the models trained is exhibited in the accompanying poster. Irrespective of the training set size, we applied the following settings to all models:Our results proved that Transkribus is a cost-effective tool for the transcription of medieval Greek manuscripts written in standard book-hands, which is the case for the vast majority of Chrysostom’s manuscripts. The transcription of hundreds of codices would require considerable human resources and/or highly specialized technical expertise. These are usually available only in larger-scale funded projects, with individual researchers and smaller project teams precluded from such research. The method tested in this study proposes a feasible way of compiling enough training data and of substantially automating the process of transcribing large numbers of manuscripts. That could have a major impact on the textual studies of popular authors, such as Chrysostom.
Understanding the phenomenon of the two recensions in (some of) the works of one of the most popular authors in the history of the Greek-speaking world will considerably enhance our knowledge of Byzantine scholarship, editorial activity, and manuscript production, to name but a few; it will also substantially affect modern editorial attitudes and decisions for scholars of Chrysostom. For the examination of this phenomenon, we currently rely on a few articles on in-progress work that was never completed, as well as on long-outdated editions. Time and again, attempts to examine the issue were impeded by the volume of the texts and the number of the manuscripts involved.  Thus, for lack of relevant literature, one must turn to the primary sources afresh. For centuries, the painstaking task of transcribing such a volume of manuscripts deterred any aspiring scholars and editors. The latest technological advances in HTR and tools like Transkribus have rendered this once impossible task perfectly realistic.
Boenig, M., and K.-M. Würzner. 2017. “Compilation of a Large Ground-Truth Data Set Using Transkribus.” Paper presented at Transkribus User Conference 2017, 2–3 November 2017. Vienna. Slides available at https://read.transkribus.eu/wp-content/uploads/2017/07/Boenig_Wuerzner_Groundtruth-1.pdf; accessed on February 19, 2021.
Causer, T., and V. Wallace. 2012. “Building A Volunteer Community: Results and Findings from Transcribe Bentham.” Digital Humanities Quarterly 6(2).
Hunger, H. 1977. “Minuskel und Auszeichnungsschriften im 10.-12. Jahrhundert.” La paléographie grecque et byzantine, ed. J. Glénisson, J. Bompaire, and J. Irigoin, 201–220. Colloques internationaux du Centre national de la recherche scientifique 559. Paris.
Leifert, G. 2018. “HTR+ in Transkribus.” Paper presented at Transkribus User Conference 2018, 8-9 November 2018. Vienna. Slides available at https://readcoop.eu/wp-content/uploads/2018/11/LEIFERT-CITLAB.pdf; accessed on February 19, 2021.
Muehlberger, G., L. Seaward, M. Terras, S. A. Oliveira, V. Bosch, M. Bryan, … & B. Gatos. 2019. “Transforming scholarship in the archives through handwritten text recognition.” Journal of documentation 75(5):954–976.
Devine, A. M. 1989. “The Manuscripts of John Chrysostom’s Commentary on the Acts of the Apostles. A Preliminary Study, a Critical Edition.” The Ancient World 20:111–125.
Ströbel, Ph. B., S. Clematide, and M. Volk. 2020. “How Much Data Do You Need? About the Creation of a Ground Truth for Black Letter and the Effectiveness of Neural OCR.” In Proceedings of the 12th Language Resources and Evaluation Conference, ed. Nicoletta Calzolari et al., 3551–3559. Marseille.
Van Dalen-Oskam, K. H. 2014. “Authors, scribes, and scholars: Detecting scribal variation and editorial intervention via authorship attribution methods.” In Analysis of Ancient and Medieval Texts and Manuscripts: Digital Approaches, ed. T. Andrews and C. Macé, 141–158. Turnhout.
[ back ] 1. Transkribus, initially developed by Innsbruck University, is now part of the EU project READ and can be accessed at https://readcoop.eu/transkribus/?sc=Transkribus (accessed on February 19, 2021).
[ back ] 2. As of October 19, 2020, Transkribus changed its policy to a freemium model. All users are offered up to 500 free HTR credits upon registration, but a subscription plan is required for further text recognition options (model training, transcribing, and layout analysis are still free of charge).
[ back ] 3. See for instance the Burrow’s Delta method applied in Van Dalen-Oskam 2014. Transkribus’s own tool for direct comparison of its versions aims primarily at evaluating the HTR produced results. And, although its Keyword Spotting feature is not directly relevant for the purposes of our project, the output with adjustable confidence levels can be clearly valuable.
[ back ] 4. All our manuscripts are in scripta continua and, to achieve word separation, language tools are required.
[ back ] 5. Transkribus suggests avoiding accents transcription for better HTR performance, yet we did transcribe them, for a fuller assessment of palaeographical issues possibly affecting the performance of the HTR tool.
[ back ] 6. Transkribus offers Text2Image, a transcription matching tool with document images. See https://transkribus.eu/wiki/images/6/6f/HowToUseExistingTranscriptions.pdf (accessed on February 19, 2021).
[ back ] 7. Boenig and Würzner 2017.
[ back ] 8. Leifert 2018.
[ back ] 9. Augmentation techniques could include image transformation, filter applications, randomly repeating images (already provided by Transkribus), or combinations of such methods.
[ back ] 10. With printed texts images and below 20% threshold, similar experiments were conducted by Ströbel, Clematide, and Volk 2020:3556–3557.
[ back ] 11. Training sets selection demands attention and results will occur later in the research.
[ back ] 12. More than 50,000 words appear in the UCL’s Bentham Project. See https://www.ucl.ac.uk/news/2019/aug/transcribing-brunels-illegible-handwriting-using-ai/ (accessed on February 19, 2021). For a strong case for crowdsourcing in producing training data, however big a budget is required; see https://www.ucl.ac.uk/bentham-project/transcribe-bentham (accessed on February 19, 2021) and the preliminary results in Causer and Wallace 2012.
[ back ] 13. See “Kirchenlehrerstil,” in Hunger 1977.
[ back ] 17. For example, a series of studies on the Homilies on the Acts in view of a new critical edition of the work started with a dissertation in 1936 and ended in 1998. These publications reflect a research project passed on from teacher to student. Most of the involved parties are now deceased and the edition was never completed. For a more complete account of these works up to that point see Devine 1989:123-4. Other sets of homilies were also scheduled to be edited, never to be completed. All in all, only a small portion of Chrysostom’s (shorter) works has been edited since Migne’s Patrologia Graeca.