MiniJudgeJava Help Page
Version 1.0.0 - Last
updated on August 17, 2009
Step-by-step manual More background information
To help isolate interesting factors from nuisance factors, MiniJudgeJava generates sets of items in accordance with a factorial design. To make this process as simple as possible, item sets are generated based on a "prototype set", similar to the example sets cited in syntax papers.
Name one or two binary factors, representing the syntactic, lexical, or other variables differentiating the sentences in your sets. You have to enter your own factor name(s) according to your research topic. You can choose factors that you predict to be directly relevant to grammar (e.g. "AdjunctExtract" in an experiment on adjunct islands), but you may also include a performance factor (e.g., "CenterEmbed" for center-embedded structures).
NOTE: If you have two factors, you must choose names starting with different characters, since they'll be abbreviated in some of the tables. Also, factor names may not contain spaces, since this will cause trouble for the R analysis.
If you use two factors, MiniJudgeJava will not only test if they show main effects, but also if there is an interaction. For example, you would expect the factors AdjunctExtract and CenterEmbed both to show main effects, but you would not expect an interaction (since they presumably involve independent "modules," namely competence vs. performance). By contrast, if your factors were Adjunct and WhMove, you wouldn't expect either to show main effects (since adjunct structures and wh-movement are both grammatical), but they should show an interaction (since wh-movement from adjuncts should be bad, while wh-movement from non-adjuncts should be good).
Enter your factor(s), then click "Approve Factors". If you want to reset your factor names, click "Clear".
If you already have a material list and want MiniJudgeJava to generate survey forms for you, click HERE.
In the spaces next to factor values, enter a set of prototype items such that each item represents one factor value (or combination of factor values). Note that factor names are represented by their initial letters to save space.
In a later optional step, MiniJudgeJava can automatically segment the prototype items into their longest repeating segments to make it easier to replace them systematically when creating additional items sets. If you want to force your own segmentation, type "|" inside your sentences. For example, "The | banana is | here" will then be segmented into "the", "banana is", and "here".
If you want sentence-internal punctuation to be editable when creating additional item sets, check "Segment Internal Punctuation". MiniJudgeJava will then parse sentence internal punctuation as separate segments.
When you are satisfied with your prototype items, click "Approve Prototype Items". If necessary, you can still modify your prototype items and reapprove them later. Click "Clear All" if you want to redo all your prototype sentences.
MiniJudgeJava can help you create multiple item sets, but since the program knows no human language it may make mistakes. If you would rather create the entire list of experimental items yourself, click HERE. Either way, generation of structurally matched item sets may be easier with the help of a thesaurus (there are many available online); see HERE for more information.
If you have gone through the previous steps, MiniJudgeJava has already segmented your prototype set into strings. These represent pieces that can be varied as wholes, if you choose, across your item sets.
For example, given the prototype set below right, MiniJudgeJava extracts the prototype segments below left. These are the largest repeated strings in the prototype sets.
|
who did |
Who did you
know because I
saw? |
Enter a number from 1 to 25 for the total number of item sets you need.
MiniJudgeJava can generate additional item sets by replacing the prototype segments with new segments. As illustrated below, segment set 1 represents the prototype segment set. When filling in the cells with new segments, make sure that they are linguistically equivalent to the corresponding prototype segment. Scroll rightward and downward to show all of the cells for new segments. When all of the cells have been filled, click "Approve Sets" to proceed. If you want to clear all of the cells, click "Clear all".
Approve master list of test sentences
Carefully check the sentences appearing in the "Master List", editing them where necessary. Do not edit the factor name(s), item numbers, or set numbers. If you want to create the entire master list of items from scratch by yourself, carefully follow the coding illustrated below.
A single speaker is not enough to produce statistically
reliable
results. Moreover, individual speakers may have unknown idiosyncrasies.
Enter a number from 1 to 50 in the space next to "Number of speakers"
and click "Approve
Master List" to proceed.
IMPORTANT: If you are restarting your work at this point, you must first paste the master list back into the window and reapprove it.
In the current version of MiniJudgeJava, responses must be binary judgments (see HERE for justification of this limitation). If speakers give their responses by email, they must use "1" to represent positive responses and "0" to represent negative responses, and they must record their judgment for an item at the beginning of the line, before the item ID number. These restrictions should be made clear in the instructions.
Suggested instructions are shown in the window below "Survey Instructions". Feel free to modify them as desired (e.g. translating them into the native language of your speakers, or deleting them entirely).
When you finish editing the instruction, click "Generate Surveys" to
generate survey forms.
"SchematicData.txt" contains information linking the items in each survey with its factor
values, item numbers and set numbers, needed for statistical analysis but which must be
hidden from the informants.. The arrangment of this information on each line of this file
is as described below.
| 01 | 01 | 01 | 12 | +Adjunct | -WhMove |
| survey ID number | Item ID number | Item set ID number | order in survey | first factor | second factor (if present) |
"SurveyIntro.txt" records the instructions for your experiment. You
can copy it for reuse for further experiments with the same design.
Note: The files in the experiment folder are
crucial for the statistical analysis of your experiment, so keep them together, and do not
modify them (aside from the survey files, to which you may add speaker responses, as
described HERE).
There are three ways to run experiments generated by MiniJudgeJava: through printed paper surveys, through email, or as a computer-based laboratory experiment using MiniJudgeJava itself.
Through printed surveys
The current version of MiniJudgeJava passes on the job of statistical analysis to R. MiniJudgeJava will talk to R for you, and have R genrate a nontechnical summary of the statistical findings.
The data collected with MiniJudgeJava are categorical (binary) and repeated-measures data (i.e. each speaker gives multiple judgments). This type of data requires a highly sophisticated kind of statistics (called GLMM) that is hard to implement efficiently in current version of MiniJudgeJava (this is a job for a future version).
MiniJudgeJava passes the hard work over to R, so you need to download R before you can continue. R is by far the most powerful and widely used free statistics package available, so it's worth owning if you do any quantitative research. Click HERE for more information about R, including information about how to download it.
Enter and load experimental result
Before using R for statistical analysis, your results must be converted into a R-readable data file. How this is done depends on how you run your experiment. If you run the experiment by printing out the surveys, you have to type the results into the survey files saved in the experiment folder (adding "1" or "0" as appropriate before each item). If you collect speakers' judgments through email, you have to save all of the surveys sent back to you in the experiment folder, where the crucial files are saved. If you run the experiment with the built-in function of MiniJudgeJava, you don't have to do anything.
No matter how you run the experiment, you next have to load the experimental results.
Do this by entering or browsing for your experiment name with
the "Load
Experiments" button.
After loading the experiment, you will be told how
many
judgments MiniJudgeJava has read and how many of them are still
missing, based on your original design.
If you are satisfied that your experiment is complete,
click "Save R data file as..."
to format and save the data for statistical analysis.
In order to run the analysis, R will need to refer the full name of the data file that you just saved. NOTE: R treats any filename extension (e.g. ".txt") as part of the filename. However, by default, Windows does not show the filename extension in directories. So if you're using Windows, be sure to include ".txt" at the end of your filename even if you don't see it when you display the file in a directory. (Mac users shouldn't have to worry about this.)
You must also decide if you want R to test for satiation, where judgment contrasts weaken over time. Satiation has been taken by some to provide a new window into competence, performance, or their relationship, but testing it in MiniJudgeJava may reduce sensitivity to other effects.
Enter the name of your data file in the box after "Filename:". If you want R to test for satiation, check the indicated box. Then click "Generate R Code" to generate the R code.
Paste or load R command code into R
If all has gone well, MiniJudgeJava should have generated the R command code needed to analyze your experimental results in the box below "Copy R Code to Clipboard". This code will automatically test the significance of the main factor(s), their interaction (if two factors), and order. If you decided to test for satiation, R will also look for interactions between order and the main factor(s).
The analysis will be summarized in a brief, easy-to-read format. This is based on R's own detailed but technical format, which will be automatically saved in a file in the same folder as your data file.
To use the command code, start the R program on your computer. Then you must change R's directory to the location of your data file using R's FILE menu. Then either copy and paste the code into the R window, or save the code as a text file (e.g. "code.txt") and load it into R using R's source() command (e.g. source("code.txt")). If this is your first time running a statistical analysis using MiniJudgeJava, R will first ask you to download the key package lme4; in future runs this step will be skipped. Whether or not you have already downloaded this package, R may pause for about several seconds while loading it into the computer memory.
After running the R code, the statistical analysis of your data should be complete.
Significant effects can be positive or negative. A positive effect for a binary factor means that the probability of getting a "1" judgment is significantly higher for the [+] value of the factor; a negative effect means the same for the [-] value. A positive effect for order means that judgments get better over the course of the experiment; a negative effect means they get worse. An interaction between some factor and order means satiation (or anti-satiation): the judgment contrast for that factor changed over the course of the experiment. The actual sign of an interaction between order and a factor, or between factors, depends partly on how you defined your factors (i.e. whether [+F] is grammatical or ungrammatical, or whether its grammaticality depends on the value of factor [G]).
For help understanding your statistical results, click HERE.
Some users may also be interested in MiniJudgeJava Deluxe. It's just like ordinary MiniJudgeJava, except that it costs US$375 and requires a dongle. Click HERE for information.
Chen, T.-Y., Yang, C.-T., & Myers, J. (2009). MiniJudgeJava (Version 1.0.0) [Computer software]. Retrieved from http://www.ccunix.ccu.edu.tw/~lngproc/MiniJudge.htm.
Citing R:
R Development Core Team (2009). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. Retrieved from http://www.R-project.org.
Citing lme4 [R package for GLMM]:
Douglas Bates and Martin Maechler (2009). lme4: Linear mixed-effects models using S4 classes [Computer software]. R package version 0.999375-31.
Help pages written by James Myers and Yuguang Ko.