2.1. Language aptitude in the LLAMA test

Language aptitude is seen as one of the characteristics that may explain individual differences in success in learning an L2 (Grigorenko et al., 2000). Language aptitude has been theorized to consist of a number of cognitive abilities (e.g., Carroll, 1973; Dörnyei, 2014). Phonemic coding ability, grammatical sensitivity, inductive language learning ability and associative memory are seen as key cognitive abilities for learning an L2 (Carroll, 1973), and these components are included in the MLAT developed by Carroll and Sapon (1959). Based on the MLAT, Meara et al. (2002) published the LAT tests that consist of five subtests, measuring the following elements: Phonetic memory skills, capacity of vocabulary learning, grammatical inferencing ability, memory ability for sequences of unknown sounds, and ability for sound symbol association. The LAT test was initially developed for L1 speakers of English, and it may be less suited for L1 speakers who use other orthographic systems than the Latin alphabet, such as Greek or Japanese. Moreover, one of the subtests of the LAT test is designed based on existing languages, which means the test cannot be used to measure aptitude of L1 speakers of these languages.

Based on the LAT test, Meara (2005) developed the LLAMA Language Aptitude Test, which is freely available (http://www.lognostics.co.uk/tools/llama/). The LLAMA test battery consists of four sub-components: The LLAMA_B (vocabulary learning ability), LLAMA_D (sound recognition), LLAMA_E (sound-symbol association) and LLAMA_F (grammatical inferencing). Recently (after data collection for this study had been completed), a new version of the LLAMA tests was published (Meara & Rogers, 2020).

2.2. Previous studies

Language independence or language neutrality is a crucial characteristic for aptitude measures, meaning that the scores on the language aptitude test should not be affected by the writing systems, phonetic features, or grammar characteristics of the L1 of the test users. Especially if the LLAMA test is to be used in studies that measure language learning aptitude of L2 learners with diverse L1 language backgrounds, language independence is crucial. Likewise, when comparing across studies with diverse L1 backgrounds (e.g., when comparing Artieda & Munoz, 2016; Kepinska et al., 2017; Saito, 2017; Yalçin et al., 2016; Yilmaz & Granena, 2016), language independence of the test for aptitude is important. For this reason, several studies have investigated the validity and the reliability of the LLAMA test itself. These studies have focused on internal validity of the tests (Bokander & Bylund, 2020) and on a number of individual variables of the participants, such as gender, age of onset (Granena & Long, 2013), multilingualism (Rogers et al., 2017), prior L2 instruction (Rogers et al., 2017) and L1 language background (Granena, 2013; Rogers et al., 2016, 2017). Here we briefly review the studies focusing on L1 language background.

Granena (2013) investigated the LLAMA test with 186 participants (79 males, 107 female) who were under the age of 40 (M = 25, SD = 5.29, range = 18–39) with different language backgrounds: English, Spanish, and Chinese. The results showed that the scores did not differ depending on the L1. Furthermore, the results of her study drew the conclusion that the LLAMA test measured two different dimensions of language aptitude; the LLAMA_B (vocabulary learning capacity), LLAMA_E (sound-symbol association) and LLAMA_F (grammatical inferencing) invoke explicit learning ability, whilst the LLAMA_D (sound recognition) is more related to implicit learning ability. Moreover, in their study of 350 participants with diverse L1s (L1groups with more than 20 participants were Afrikaans, English, German, and Wolof), Bokander and Bylund (2020) carried out a principal component analysis that also identified the subtest LLAMA_D (sound recognition) as the odd one out. This finding supports the conclusion that LLAMA_D, unlike the other tasks, taps into implicit processing.

Rogers et al. (2016) examined 229 participants ranging in age between 10 to 75 with diverse L1 backgrounds and educational levels. Because they wanted to compare participants who had learned different types of scripts in their L1, participants were divided into three groups: English Latin script (n = 99), non-English Latin script (n = 18), and non-Latin script (mostly Chinese; n = 17).¹ They investigated whether differences in L1 writing systems impacted the scores of language aptitude. The results did not reveal any specific effects for the L1, but they did find an effect for educational level in three components: LLAMA_B (vocabulary learning capacity), LLAMA_E (sound-symbol association) and LLAMA_F (grammatical inferencing).

Consequently, Rogers et al. (2017) carried out an even broader validation study of the LLAMA test. They examined the role of the L1 writing system and the language neutrality in general of the LLAMA test. They chose L1 speakers of English (n = 107), Chinese (n = 56) and Arabic (n = 32) as participants. The L1 writing systems of the participants therefore consisted of the English Latin alphabet, Chinese morphosyllabic, Chinese logographic system (Tolchinsky et al., 2012), and the Arabic consonantal alphabet. The results showed that there were significant differences among these three groups in the LLAMA_B, LLAMA_E and LLAMA_F, but not in the LLAMA_D. It turned out, however, that these groups were not comparable because the number of monolinguals was different across groups. Therefore, they analysed a new dataset — L1 speakers of English (n = 48), Chinese (n = 56), and Arabic (n=30), excluding all monolingual participants. For these new groups, there were no significant differences. They concluded that with respect to these writing systems and language backgrounds, the LLAMA test is language neutral. Rogers et al. (2017) also followed up on research and hypotheses by Sparks et al. (1995), among others, that showed that language learning experience may have a positive influence on language learning aptitude scores. Among the variables investigated, they found that prior L2 instruction was the strongest predictor of aptitude, especially in LLAMA_B (vocabulary learning ability) and LLAMA_F (grammatical inferencing).

These studies have examined diverse L2 learners varying in L1 typology and different writing systems in order to investigate the validity of the LLAMA test, yet none of them has taken agglutinative languages into account as L1s, nor have they investigated languages with multiple writing systems. The current study therefore includes three languages as L1 background (Hungarian, Japanese and Dutch), which are distinctive in terms of orthographic systems and typology.

2.3. Syntactic characteristics of Hungarian, Japanese, Dutch and the artificial language in LLAMA_F (grammatical inferencing)

Hungarian is member of the Finno-Ugric language family, whilst most of European languages are categorized as Indo-European languages (Kiss, 2002). One of the distinctive aspects of the Finno-Ugric languages is its agglutinative characteristics. In agglutinative languages, words contain multiple morphemes that mostly remain unchanged. It is also noteworthy that Hungarian has Subject-Verb-Object (SVO) as its main structure, followed by SOV (Kas et al., 2016). Hungarian, however, is often said to be a free word order language due to the variety of structures: SVO, SOV, OVS, OSV, VSO and VOS (Kiss, 2002).

Like Hungarian, Japanese is categorized as an agglutinative language. It is said to be part of the Altaic language family, but the exact origin of Japanese has not been determined (Maeda, 2007). The basic structure of Japanese is SOV, but the word order is usually flexible. With respect to verb position, the verb is placed at the end of a sentence. Japanese verbs have no personal conjugation, but they conjugate accompanying information such as tense. The Japanese sentence (1) is an example of a sentence without a subject and with the conjugation of the verb, showing the agglutinative nature of the language:

Dutch belongs to the West-Germanic group of Indo-European languages. Dutch is closely related to English and German and has two syntactic structures, SVO and SOV (Bennis & Israel, 2010). SVO mostly appears in the main clause, whilst SOV is found in subordinate clauses. Instead of grammatical cases, relations between words are indicated by using pronouns mostly (van der Sijs, 2005), and unlike Hungarian and Japanese, Dutch is not agglutinative in nature.

Upon analysing the sentences in the LLAMA_F (grammatical inferencing) test, it was found that the syntax of this artificial language has a rich morphology, using inflection with adpositions. For instance, the sentence ‘inut-ek ipot-arap’ is paired with a picture of ‘two red rounded objects on a board’. The sentence would be broken down as follows:

Inut-ek ipot-arap

inut: on a board or above something

-ek: two objects

ipot: red colour

arap: rounded form

A morphological element ‘-ek’ follows the adposition ‘inut’, which can be interpreted that the adposition has a characteristic of inflection. This suggests that the artificial language might be similar to an agglutinative language such as Hungarian in which postpositions can be attached to the stem of the word.

2.4. Orthographic characteristics of Hungarian, Japanese, Dutch, and the systems used in LLAMA_B (vocabulary learning)

Hungarian and Dutch are written using the Latin alphabet, which is a phonogram writing system. The two Japanese syllabic alphabets (hiragana and katakana) are also categorized as phonogram systems, whilst Japanese kanji is regarded as a logogram because each character of kanji depicts a specific meaning (Coderre et al., 2008; Tanaka, 2015). The most distinctive aspects of Japanese compared to both other languages in the present study may therefore be the number of writing systems used: Japanese uses two syllabic phonogram systems and one logographic writing system.

The Japanese writing system motivated the reinvestigation of the language neutrality of the LLAMA test because of its combination of orthographies. It is noteworthy that using both kana and kanji could affect cognitive processing (Coderre et al., 2008; Sakurai et al., 2000) because syllabic writing systems connect visual forms (graphemes) to sounds (phonemes), whilst logographic writing systems join forms (characters) with meanings (morphemes) and sounds (phonemes) (Coderre et al., 2008). It has also been shown that phonograms and logograms are processed differently (Tanaka, 2015). Such differential processing may transfer when reading in the L2, such that L2 learners with an alphabetic L1 background and L2 learners with non-alphabetic L1 backgrounds rely on different types of information during lexical processing in the L2 (Wang et al., 2003). Moreover, L1 speakers with logographic writing systems tend to rely more on visual information than on phonological information, while on the other hand, L1 readers with alphabetic language backgrounds tend to directly analyse phonological information (Wang & Geva, 2003). Because the writing experience in the L1 affects decoding and learning a new vocabulary in the L2 (Hamada & Koda, 2008), a language learning aptitude test may (unwantedly) be affected by the writing system used. L2 learners whose L1 writing system is closely related to that of the L2 are able to perform better than those who have less similar L1 language backgrounds (de Groot et al., 2002).

In previous validation studies, Chinese was included as a non-alphabetic language (Granena, 2013; Rogers et al., 2016, 2017). Although the ascendant of Japanese kanji is the Chinese character hanji, it is important here to emphasize that they have developed differently and they should not be classified in the same category (Hashimoto et al., 2017). The Japanese kanji is more graphic than the recent simplified Chinese characters mainly used in the mainland, and the written forms do not always match. Therefore, these writing systems should not be regarded as one type of logogram (Yokoyama, 2016). Furthermore, the Chinese hanji has only one mora and one pronunciation per character, by contrast, each Japanese kanji has one to three morae and it usually has more than two pronunciations. These differences argue for a separate investigation of Japanese as an L1. In addition to this need to study Japanese separately from Chinese, the L1 Japanese speakers may perform the LLAMA_B (vocabulary learning) subtest in a more efficient way because they are accustomed to processing two different writing systems simultaneously. In the LLAMA_B subtest, a picture (graphic) and a word (spelled in the Latin alphabet, thus phonographic) have to be matched and memorized, thus a graphic and phonographic system need to be processed at the same time. Given that the L1 Japanese speakers are used to processing two different writing systems in which one is graphic and the other phonographic, they may be found to score higher on this subtest.

4.1. Participants

Three groups of university students were recruited. All participants received a small gift as a token of appreciation for their participation. The first group consisted of Hungarian students (studying at Eötvös Loránd University or Károli Gáspár University) majoring in Japan Studies. The second group consisted of Japanese students (studying at Osaka University) majoring in Hungarian language and culture. The third group consisted of Dutch students (studying at Leiden University) majoring in a Romance language (e.g., French language and culture or Latin American Studies). Recruitment was initiated by e-mails to the relevant language teachers at the four universities. These teachers shared the call for participation in class or on online platforms. Additionally, the first author recruited participants in situ, by asking students just before or after class to participate. In total, 86 participants were recruited. Due to incomplete datasets or because of conditions that were not met, however, seven participants were excluded from the analyses. For instance, one of the Dutch students reported having learned Finnish, an agglutinative language, so this participant’s data of this student were not included. Table 1 shows the background information of all included participants in the three groups. All participants were informed of the purpose of the study and signed a consent form. Data were anonymized prior to data storage and analysis.

4.2. Materials

4.3. Procedure

Each participant first read and signed the consent form and filled out the language background questionnaire. The first author then provided instructions about the experiment (orally), using a script based on the LLAMA manual (Meara, 2005). For carrying out the aptitude tasks, we used the same order as Saito (2017): LLAMA_D, then _B, and finally _F. This order ensures that the most implicit task (LLAMA_D, sound recognition) is carried out first, such that potential conscious strategies are less likely to be used. Most participants carried out the experiment in a quiet environment, such as a classroom or computer room. However, this was not possible for a couple of L1 Hungarian and L1 Dutch participants; they carried out the tasks in a common room. However, all participants used headphones during the experiment for LLAMA_D and LLAMA_B (vocabulary learning). To be able to make notes, participants received pen and paper before the start of LLAMA_F (grammatical inferencing). Scores for all the tests were recorded by the software and noted down by the first author. For 16 students, the first author asked them in an informal interview how they had proceeded in carrying out the task. The answers to these questions were written down by the first author. The entire experiment including the interview after the test lasted around 30 minutes.

4.4. Data analysis

To test whether there were differences between the three groups, we carried out a MANCOVA, with the three scores for the LLAMA tests as dependent variables, language group as the independent factor (with three levels) and number of languages known as covariate. When checking the assumptions for running a MANCOVA, because some participants noted knowing eight languages but none noted seven, we rank-transformed the number of languages known. Three Levene’s tests showed that for the dependent variables, homogeneity of variance could be assumed (p’s > 0.19), and Box’s M test revealed that equality of covariance could be assumed (p = 0.91). Finally, no anomalies were found when analysing the standardized residuals. Whenever a significant difference was indicated for one of the dependent variables, we carried out post-hoc t-tests to check which groups differed from each other. Taking into account the group sizes, the analyses would reach a power of 0.88, if we assumed the effects to be large (f = 0.4). However, if we assume effects to be medium or small (thus given an f of 0.25 or 0.1), power in this study is calculated to reach only 0.48 or 0.11. The quantitative data are available on this webpage: https://osf.io/ve7xa/. The notes made during the informal interviews were summarized, and exemplary findings from these summaries are reported in the next section.

5.1. Exploring differences in aptitude scores

Table 2 shows the means and standard deviations of the original scores on the LLAMA tests by the three participant groups. Boxplots of these data can be found in Figure 1. As explained, the covariate number of languages known was rank-transformed prior to analyses.³Table 3 shows the estimated marginal means and standard errors of all subtests and groups after correcting for (rank) number of languages known.

The MANCOVA showed an overall effect of language group (F (6, 148) = 3.38, p = 0.004). From the separate analyses, it was found that there were group differences for LLAMA_D, sound recognition ability (F (2, 75) = 8.77, p < 0.001, η_p² = .19) and LLAMA_F (F (2, 75) = 3.89, p = 0.025, η_p² = 0.09), but not for LLAMA_B, vocabulary learning (F = 1.79, p = 0.174). For the LLAMA_B analysis, the number of languages known turned out to be a significant predictor (F (1, 75) = 4.97, p = 0.029, η_p² = 0.06) and in the LLAMA_F (grammatical inferencing) analysis, the number of languages as covariate approached significance (F (1, 75) = 3.74, p = 0.057, η_p² = 0.05). Pairwise comparisons revealed that for LLAMA_D, Japanese outperformed both Hungarian (p = 0.005) and Dutch (p < 0.001) participants, but Hungarian and Dutch participants did not show differential scores (p = 0.074). For LLAMA_F, pairwise comparisons revealed that Japanese outperformed both Hungarian (p = 0.014) and Dutch (p = 0.020), but Hungarian and Dutch were not different from each other (p = 0.85). The total adjusted explained variance (R²) for the models for LLAMA_D and LLAMA_F were 0.19 and 0.06, respectively.

5.2. Exploring differences in strategies

Informal interviews were held with a selection of participants, most of whom had scored relatively high on either LLAMA_B or LLAMA_F. In total, four L1 Hungarian, eight L1 Japanese, and four L1 Dutch participants answered informal questions immediately after finishing LLAMA_F.⁴

All 16 participants reported that LLAMA_D (sound recognition) had been particularly demanding but did not mention any conscious strategies they used. Regarding LLAMA_B (vocabulary learning), the most used (self-reported) strategy was to mentally connect the “words” they read to similar sounding words from their L1 (or an L2). For instance, L1 Hungarian speakers connected “CIB” to the existing commercial bank CIB. Some L1 Hungarian speakers who were learning Japanese as an L2 specifically mentioned that they tried to use similar strategies they had been using for learning Japanese kanji. L1 Japanese speakers memorized “KABAN” (bag in Japanese) with the accompanying picture of a log by remembering “a wooden bag”. Similarly, the L1 Dutch speakers noted that they used their L2 English to associate “MEN” with the picture of a little man. Furthermore, an L1 Dutch speaker mentioned resemblance of the words used in the LLAMA_B to a Mayan language. The participant thus made use of personal knowledge of a Mayan language to complete this subtest.⁵

Finally, regarding LLAMA_F (grammatical inferencing), the L1 Japanese speakers noted that the artificial language resembled Hungarian, especially in the way that the artificial language used affixes. L1 Hungarian speakers noted that the artificial language resembled an agglutinative language. In contrast, the L1 Dutch speakers mentioned no resemblances of the artificial language to a language that they knew.