MiniJudgeJava Instructions
Version 0.9.9++ - Last updated on May 10, 2007

MiniJudgeJava is a tool for minimalist experimental syntax. It helps you to design, run, and analyze syntactic acceptability judgment experiments that are as simple and powerful as possible.

Software requirements. MiniJudgeJava is an excutable Java file that works with computers that have Java Virtual Machine (JVM) installed. To test if your computer has already installed JVM, visit the test page. Or, if you have Windows, you can download MiniJudgeJava.exe, created using JSmooth, which is supposed to automatically check for JVM and help you install it if necessary.

MiniJudgeJava was primarily written by Tsung-ying Chen, with help from Yang Chen-Tsung, and these instructions were written primarily by Yu-guang Ko. MiniJudgeJava has a very different look-and-feel from MiniJudgeJS, and some of its internal algorithms also work differently, though the end products (randomized surveys and statistical analyses) should be the same. We are still testing the compatibility of MiniJudgeJava on different operating systems. Please send us any reports about incompatibilities or other bugs, comments about differences in the appearance or algorithms used by MiniJudgeJava vs. MiniJudgeJS, as well as suggestions for new features.

As with MiniJudgeJS, to analyze the data collected with MiniJudgeJava, you will also need to install R, the free yet powerful statistics software package that is fast becoming a world standard (www.r-project.org). R is the one free statistics program you might actually want to own. To learn more about R, including how to download it, see HERE.

Further basics. If this is your first time trying MiniJudgeJava, you might want to scroll through all of these instructions before starting a project, just to get a sense of what you're in for. These instructions are also printer-friendly (about 20 pages long).

Using MiniJudgeJava involves a series of guided or fully automated steps, divided into three major parts: designing, running, and analyzing. It is safest to reserve enough time for each major part to finish it in one session, and save your data whenever there is a file-saving function available.

MiniJudgeJava is open source. The source code can be seen in the CLASS files HERE. It is "copylefted" with the GNU General Public License. Basically, this means you can distribute it freely (e.g. create mirrors) and modify it any way you like, including incorporating its code in your own program to sell, but the contribution of MiniJudgeJava must be acknowledged and stay open-source in your new program. See the "about" page in MiniJudgeJava for more information.

MiniJudgeJava is far from perfect. If something weird happens when you use it, or if you have any suggestions about how to improve it, contact the owner of this page HERE, giving details on your operating system, Java runtime environment, and what you were doing when it happened. MiniJudgeJava will be upgraded eventually. If you're impatient, feel free to upgrade it yourself!

If you use MiniJudgeJava in your research, please cite it.

For more information about experimental syntax, minimalist experimental syntax, MiniJudge, and R, click HERE.

Steps

Using MiniJudgeJava involves following a series of steps. Scroll down (or, if you are continuing a project, click a step to jump directly to where you left off).

• Design experiment
• Make Sentences
  
• Choose experimental factors
• Choose set of prototype sentences
• Protoset
  
• Segment prototype set (optional)
• Choose replacement segments for additional sentence sets (optional)
• Master List
• Approve master list of test sentences
• Choose number of speakers and save master list
• Run experiment
• Survey forms
  
• Write instructions for speakers
• Copy survey forms to print or email
• Save schematic survey information
• Analyze experiment
• Download and install R (if you haven't already)
• Talk to R
  
• Enter raw results
• Generate data file
• Save data file
• R code
  
• Generate R code
• Paste R command code into R
• R will summarize your findings in an easy-to-read format

Design experiment.

To help isolate interesting syntactic factors from nuisance factors, MiniJudgeJava generates sets of sentences in accordance with a factorial design. To make this process as simple as possible, sentence sets are based on a "prototype set" of sentences, similar to the example sets cited in syntax papers.

Choose experimental factors.

Name one or two binary factors, representing the syntactic, lexical, or other variables differentiating the sentences in your sets. You have to clear the default text "Required." and "Optional." in the spaces for factor names and enter your own factor names when needed. You can choose factors that you predict to be directly relevant to grammar (e.g. "AdjunctExtract" in an experiment on adjunct islands), but you may also include a performance factor (e.g., "CenterEmbed" for center-embedded structures).

NOTE: If you have two factors, you must choose names starting with different characters, since they'll be abbreviated in some of the tables. Also, factor names may not contain spaces, since this will cause trouble for the R analysis.

If you use two factors, MiniJudgeJava will not only test if they show main effects, but also if there is an interaction. For example, you would expect the factors AdjunctExtract and CenterEmbed both to show main effects, but you would not expect an interaction (since they presumably involve independent "modules," namely competence vs. performance). By contrast, if your factors were Adjunct and WhMove, you wouldn't expect either to show main effects (since adjunct structures and wh-movement are both grammatical), but they should show an interaction (since wh-movement from adjuncts should be bad, while wh-movement from non-adjuncts should be good).

Enter your factor(s), then click "Approve Factors". If you want to reset your factor names, click "Clear All".

If you have a sentence list already and want MiniJudge to generate survey forms for you, click HERE.

Choose set of prototype sentences.

In the spaces indicated below, enter a set of prototype sentences such that each sentence represents one factor value (or combination of factor values). Note that factor names are abbreviated to just one character to save space.

When you are satisfied with your sentences, click "Approve Prototype Sentences". If necessary, you can still modify your prototype sentences and reapprove them. Click "Clear All" if you want to redo all your prototype sentences.

Segment prototype set.  SKIP TO MASTER LIST

In "Protoset", MiniJudgeJava helps you create multiple sentence sets, but since the program knows no human language the process may be a bit buggy. If you would rather create the entire list of experimental sentences yourself, click HERE. Either way, generation of syntactically matched sentence sets may be easier with the help of a thesaurus (there are many available online); see HERE for more information.

The additional sentence sets will be variants on your prototype set. If you have gone through the previous steps, MiniJudgeJava has already segmented your prototype set into strings. These represent pieces that will vary as wholes across your sentence sets (or not vary at all across sets, if you choose).

    Parsed protosegments: who did%you%know%because%I saw%knew%him

The segments found by MiniJudgeJava are listed in the window below. To understand how these segments are parsed and segmented, compare the parsed segments with your prototype set, shown below on the right.

Since MiniJudgeJava does not actually understand the issue or language under investigation, you may want to modify parsed segments by yourself to fit your specific research need. You can simply edit protosegments by adding strings separated by "%" or deleting strings at this step. But note that the format of parsed segments has to be maintained. Parsed segments have to be separated with "%". When you are satisfied that the segments are correct, click "Approve Protosegments".

Input other sets.

To help MiniJudgeJava generate variants of your prototype set, you have to enter syntactic equivalents for the prototype segments in the format identical to your parsed segments. You may first copy the text from the box under "Parsed Protoset", paste it to the space under "Input Other Sets" and replace strings separated by "%" in the text with their syntactic equivalents.  For example, to add one more set in addition to the demo prototype set, the following new segments could be used:

   which person does%Mary%suspect%because%Mom hates%suspects%him 

After this point in the process, it is no longer possible to modify parsed segments. Every sentence set has to be added in exactly the same format as the protoset, or the procedure cannot proceed. When you are satisfied with your new segments, click "Approve Sets".

Approve master list of test sentences.

Carefully check the sentences appearing in the "Ordered Master List", editing them where necessary. (Note that the first number is the item number and the second is the set number; do not edit this information or the factor name(s) shown at the top).

If you want to create the entire master list of sentences by yourself, please follow the format below or MiniJudgeJava won't be able to read it:

Choose number of speakers and save masterlist.

A single speaker is probably not enough to permit statistically reliable results. Moreover, individual speakers may have unknown idiosyncrasies. A reasonable number would be 10-25 speakers. Enter the number of speakers you want after "How many surveys do you need:" and press "Approve Master List".

After you have approved the master list, save it as a text file in your computer by clicking on "Saving Master List as...". You can give the file any name you like and save it anywhere, but you will need this information when you generate your survey forms in the next major step. If you must stop working now, restart your work at this step, first pasting your master list back into this window and reapproving it.

MiniJudgeJava doesn't actually run the experiment for you. Instead, it generates survey forms for you to distribute to native speakers. You can copy the whole list of survey forms to a word processing program and paginate it for printing, one survey per sheet of paper. Or you can copy each survey into separate emails, one for each speaker.

IMPORTANT: If you are restarting your work at this point, you must first paste the master list back into the window and reapprove it.

In the current version of MiniJudge, responses must be binary YES/NO judgments (see HERE for justification of this limitation). If speakers give their responses by email, they must use "1" to represent YES (acceptable) and "0" to represent NO (unacceptable), and they must record their judgment for a sentence at the beginning of the line, before the sentence ID number. These restrictions should be made clear in the instructions.

Suggested instructions are shown in the window below "Instruction of Surveys" (intended for use with English speakers filling out surveys by email at an institution requiring the use of informed consent forms). Modify them any way you like (e.g. translating them into the native language of your speakers, or deleting them entirely).

When you are satisfied with the instructions, click "Generate Surveys" to create the surveys. When survey forms are generated, you can either save all of the survey forms as a text file by pressing "Save Generated Surveys as..." or copy it to somewhere you like by pressing "Copy Generated Surveys to Clipboard", depending on how you are going to distribute these surveys to your subjects.

Copy survey forms to print or email.

The file you just saved in the last step contains survey forms for speakers. Note that forms are separated from each other by lines, and each form gives instruction, speaker number (preceded by "##"), and list of sentences numbered in the order in which they appear in that survey (with the numbers in parentheses).

To create printed surveys, open the file with a word processing program and replace the lines with page breaks. To create email surveys, copy each survey to a separate email message. The latter method will save you the trouble of retyping the judgment data for statistical analysis, but it has other disadvantages: you can't watch to make sure the speakers follow the instructions properly, and some speakers may be annoyed by unsolicited email!

Save schematic survey information.

If you went through all of the above steps, MiniJudgeJava has taken your master list of sentences and your instructions, and has used them to generate the number of survey forms that you requested. The sentences have been put into a different partially random order for each survey. Randomization is only partial to ensure that sentence types are spread evenly within each survey (click HERE for more on the algorithm used). Randomization and even spread have another benefit, explained below and HERE.

The surveys given to speakers do not explicitly mark the factors and sets, since this information may bias speaker judgments. But MiniJudgeJava needs this information for the statistical analyses. The table below shows how the crucial information is formatted in schematic form (removing irrelevant information like the actual sentences). It looks ugly, but only MiniJudgeJava has to read it:

Copy and save the contents of the window below to a file on your computer. You can save schematic survey information by either using "Copy Schematic Data to Clipboard" to copy data and save it somewhere in your own way or clicking on "Save Schematic Data as..." to create a file for your schematic survey information.

Analyze experiment.

MiniJudgeJava passes on the job of statistical analysis to R. MiniJudgeJava will talk to R for you, and have R give a nontechnical summary of the statistical findings.

Download and install R. SKIP

The data collected with MiniJudgeJava are categorical (binary) and repeated-measures data (i.e. each speaker gives multiple judgments). This type of data requires a highly sophisticated kind of statistics (called GLMM) that is hard to implement efficiently in current version of MiniJudgeJava (this is a job for a future version).

MiniJudgeJava passes the hard work over to R, so you need to download R before you can continue. R is by far the best free statistics package available, so it's worth owning if you do any quantitative research. Click HERE for more information about R, including information about how to download it.

Enter raw results. SKIP INTRO

MiniJudgeJava can handle two kinds of raw results: surveys completed electronically with a "1" or "0" near the start of each line, or a list of "1"s and "0"s that you typed in yourself from completed paper surveys.

Either way, MiniJudgeJava needs to know the survey ID number and the order number for each sentence (i.e. the numbers shown to the speakers).

The format that MiniJudgeJava is expecting for the raw results is as shown in the box below. Survey ID numbers are marked with "##", just as on the survey forms, followed by the data from that survey. The judgments (1 vs. 0) appear at the beginning of each line, followed by the sentence number inside parentheses, again just as on the survey forms. The sentences are numbered as on the survey forms, so sentence (1) on one survey is not necessarily the same as sentence (1) on another survey. This is why the survey ID numbers are crucial! They also permit surveys to be entered in any order that is convenient, as shown below.

Notice that the formatting above isn't completely "clean": speaker 2 didn't delete the ">" mark added when replying by email, the experimenter accidently hit "p" while typing in speaker 1's data, and speaker 3 ignored the instructions and put the judgments at the end of each line instead of at the beginning. That's OK -- MiniJudgeJava just looks for each sentence ID number in parentheses, and takes the first "1" or "0" before it on the same line as the judgment for that sentence; if it can't find a "1" or "0" there, it looks at the end of the line. It ignores everything else, including other numbers or letters accidentally typed, the instructions, and the sentences themselves.

Note also that speaker 2 is missing a judgment for sentence 3. MiniJudgeJava can handle this too, and GLMM is not as badly affected by missing data as some other statistical methods are. Of course, too much missing data (or data missing in a non-random pattern) can cause problems with interpreting the results, which is why speakers must be encouraged to judge all sentences.

Furthermore, MiniJudgeJava doesn't mind if you ended up testing fewer speakers than originally planned (e.g. you emailed 20 surveys but only 10 speakers responded). It can also handle it if you accidentally enter a speaker's results twice (e.g. #01 ... #02 ... #01 ...); it will ignore any extra copies.

MiniJudgeJava provides two ways to enter raw results. (1) You can save all of the completed surveys in one file and select "Load the result from..." to load your result file. Or (2) you can copy the completed surveys from the clipboard by selecting "Paste the Result from Clipboard".

Generate data file.

In order to convert the raw data above into something that R can analyze, MiniJudgeJava needs to match up each judgment with the factor (or factors) you are testing. The key to do this is in the schematic survey info that you saved earlier. To load the schematic survey information of your experiment, select "Load Schematic Data from..." to load schematic info and then click on "Generate TXT data file" to generate a R-readable data file.

Save data file.

After a R-readable data file is generated, you have to save it somewhere for statistical analysis since R cannot directly access MiniJudgeJava. You can either choose "Copy the Organized Result to Clipboard" and paste the data file somewhere or select "Save TXT data file as..." to save the data file as a text file in your computer.

Generate R code.

In order to run the analysis, R will need to refer the full name of the data file that you just saved. NOTE: R treats any filename extension (e.g. ".txt") as part of the filename. However, by default, Windows does not show the filename extension in directories. So if you're using Windows, be sure to include ".txt" at the end of your filename even if you don't see it when you display the file in a directory. (Mac users shouldn't have to worry about this.)

You must also decide if you want R to test for satiation, where judgment contrasts weaken over time. Satiation has been taken by some to provide a new window into competence, performance, or their relationship, but testing it in MiniJudgeJava may reduce sensitivity to other effects.

Enter the name of your data file in the box after "Filename:". If you want R to test for satiation, check the indicated box. Then click "Generate R Code" to generate the R code.

Paste R command code into R.

If all has gone well, by now MiniJudgeJava should have generated the R command code needed to analyze your experimental results in the box below "Copy R Code to Clipboard". When you paste this code into the R program window, it will automatically test the significance of the main factor(s), their interaction (if two factors), and order. If you decided to test for satiation, R will also look for interactions between order and the main factor(s).

The analysis will be summarized in a brief, easy-to-read format. This is based on R's own detailed but technical format, which will be automatically saved in a file in the same folder as your data file.

To use the command code, start the R program on your computer. Then you must change R's directory to the location of your data file using R's FILE menu. Then simply copy and paste the code below into the R window. If this is the first time using MiniJudgeJava, R will first ask you to download the key package; in future runs you can cancel this step. Whether or not you have already downloaded this package, R will pause for about 30 seconds while loading it into the computer memory.

After running the above code, the statistical analysis of your data should be complete.

Significant effects can be positive or negative. A positive effect for a binary factor means that the probability of getting a "1" judgment is significantly higher for the [+] value of the factor; a negative effect means the same for the [-] value. A positive effect for order means that judgments get better over the course of the experiment; a negative effect means they got worse. An interaction between some factor and order means satiation: the judgment contrast for that factor changed over the course of the experiment. The actual sign of an interaction between order and a factor, or between factors, depends partly on how you defined your factors (i.e. whether [+F] is grammatical or ungrammatical, or whether its grammaticality depends on the value of factor [G]).

For help understanding your statistical results, click HERE.

Version information. MiniJudgeJava 0.9 was the first attempt at a Java implementation of minimalist experimental syntax. The first version of MiniJudge was written in Excel VBA, believe it or not. There is another version of MiniJudge in JavaScript, click HERE to give it a try.

Some users may also be interested in MiniJudgeJava Deluxe. It's just like ordinary MiniJudgeJava, except that it costs US$375 and requires a dongle. Click HERE for information.

Citing MiniJudgeJava:

Contact James Myers with your questions and comments.