MiniJudgeJS Version 1.1.1 - Last updated on September 8, 2008
What is MiniJudge?
SKIP INTRODUCTION
MiniJudgeJS is a tool for small-scale judgment experimentation. It helps you to design, run, and analyze linguistic acceptability judgment experiments that are as simple and powerful as possible.
Simplicity. If you have ready access to native speakers, "live" or by email, you can go from hunch to statistically supported generalization in a matter of days, if not hours. This is far faster than the typical psycholinguistic experiment and only somewhat slower than the pace at which syntacticians traditionally collect data.
Power. MiniJudgeJS helps collect judgments from multiple speakers judging multiple linguistic items (e.g. sentences or words), and then analyzes them with sophisticated statistics to extract as much information as possible from the data. This way you can be more confident that judgment patterns really relate to the syntactic factors of interest, and not bias, idiosyncrasies of speakers or linguistic items, or certain processing effects.
Software requirements. MiniJudgeJS itself is just the page you're looking at right now: an HTML file containing JavaScript. MiniJudgeJS has been most extensively tested in Firefox 1.5 running in (Traditional Chinese) WindowsXP Professional 5.1, and it also seems to work just fine in other versions of Firefox (for Mac and Linux), as well as Internet Explorer, Safari, and Opera. Reports about any remaining incompatibilities or other bugs, as well as suggestions for new features, are most welcome!
To analyze the data collected with MiniJudgeJS, you will also need to install R, the free yet powerful statistics software package that is fast becoming a world standard (www.r-project.org). R is the one free statistics program you might actually want to own. To learn more about R, including how to download it, see HERE.
Further basics. If this is your first time trying MiniJudgeJS, you might want to scroll through the whole thing before starting a project, just to get a sense of what you're in for. MiniJudgeJS is also printer-friendly (it's about 20 pages long).
Using MiniJudgeJS involves a series of guided or fully automated steps, divided into three major parts: designing, running, and analyzing. It is safest to reserve enough time for each major part to finish it in one session, since the current version of MiniJudgeJS cannot save your work as a whole. Similarly, don't reload the page while you're working, or some information may be lost. You can copy your work from the screen at any time and paste it into a file as you work.
To use MiniJudgeJS offline, just save this page to your computer (along with this information page, if you like). Opening offline files with JavaScript may make some newer versions of IE nervous, but rest assured that MiniJudgeJS doesn't read or write anything on your hard drive (it handles input and output by asking you to paste to and copy from text windows). If you encounter security warnings using MiniJudgeJS offline in IE, either tell IE to allow blocked content, or better yet, dump IE and get a proper browser.
MiniJudgeJS is "copylefted" with the GNU General Public License. Basically, this means you can distribute it freely (e.g. create mirrors) and modify it any way you like, including incorporating its code in your own program to sell, but the contribution of MiniJudgeJS must be acknowledged and stay open-source in your new program. See the page source for more information.
MiniJudgeJS is far from perfect. If something weird happens when you use it, or if you have any suggestions about how to improve it, contact the author HERE, giving details on your operating system, browser, and what you were doing when it happened. MiniJudgeJS will be upgraded eventually. If you're impatient, feel free to upgrade it yourself!
If you use MiniJudgeJS in your research, please cite it. MiniJudgeJS has also proven useful as a classroom teaching aid.
For more information about judgment experiments, small-scale experimentation, MiniJudge, and R, click HERE.
Steps
Using MiniJudgeJS involves following a series of steps. Scroll down (or, if you are continuing a project, click a step to jump directly to where you left off).
Design experiment.
To help isolate interesting syntactic factors from nuisance factors, MiniJudgeJS generates sets of linguistic items (e.g. sentences or pseudowords) in accordance with a factorial design. To make this process as simple as possible, item sets are based on a "prototype set" of items, similar to the example sets cited in linguistics papers, particularly in syntax.
Choose experimental factors.
Name one or two binary factors, representing the syntactic, lexical, phonological, morphological, or other variables differentiating the items in your sets. You can choose factors that you predict to be directly relevant to grammar (e.g. "AdjunctExtract" in an experiment on adjunct islands), but you may also include a performance factor (e.g., "CenterEmbed" for center-embedded structures).
NOTE: If you have two factors, you must choose names starting with different characters, since they'll be abbreviated in some tables below. Also, factor names may not contain spaces, since this will cause trouble for the R analysis.
If you choose two factors, MiniJudgeJS will not only test if they show main effects, but also if there is an interaction. For example, you would expect the factors AdjunctExtract and CenterEmbed both to show main effects, but you would not expect an interaction (since they presumably involve independent "modules," namely competence vs. performance). By contrast, if your factors were Adjunct and WhMove, you wouldn't expect either to show main effects (since adjunct structures and wh-movement are both grammatical), but they should show an interaction (since wh-movement from adjuncts should be bad, while wh-movement from non-adjuncts should be good).
Enter your factor(s), then click OK.
Choose set of prototype items.
As with a set of examples in a linguistics paper, the prototype items should be matched as much as possible for irrelevant properties (e.g. lexical content in sentences), so that they differ only in terms of the factor (or factors) that you are interested in studying.
+A
+W
Who did you know because I saw?
-W
You knew because I saw him.
-A
Who did you know I saw?
You knew I saw him.
In the spaces indicated below, enter a set of prototype items such that each item represents one factor value (or combination of factor values). Note that factor names are abbreviated to just one character to save space (do not edit these).
When you are satisfied with your items, click OK.
Choose total number of item sets.
A single item set is probably not enough to permit statistically reliable results. Moreover, individual sets of items may have unknown idiosyncrasies. A reasonable number would be 5-20 sets.
Enter your chosen number in the box below, then click OK.
Total number of item sets:
Segment prototype set. SKIP INTRO / SKIP TO MASTER LIST
MiniJudgeJS can help you create multiple item sets, but since the program knows no human language the process may be a bit buggy. If you would rather create the entire list of experimental items yourself, click HERE. Either way, generation of structurally matched item sets may be easier with the help of a dictionary or thesaurus (there are many available online); see HERE for more information.
The additional item sets will be variants on your prototype set. If you have gone through the previous steps, MiniJudgeJS has already segmented your prototype set into strings. These represent pieces that will vary as wholes across your item sets (or not vary at all across sets, if you choose).
For example, for the demo prototype set, the list of unique segments is as follows. Note the separate listings for "know" and "knew", but only one listing for "you", despite varying in capitalization in the prototype set. Note also that "who did" and "I saw" are treated as whole pieces, since these word strings are never split up in the prototype set. (If you're confused about the point of segmenting, scroll down to the following two steps to see how the segments will be used.)
who did you know because I saw knew him
Who did you know because I saw? You knew because I saw him. Who did you know I saw? You knew I saw him.
TIP: You can use the special character "|" in your prototype items to demarcate strings that should not be split up (e.g. fixed phrases or words in sentences written in orthographies that don't mark them, like Chinese). If you want to edit your prototype items now, click HERE.
The segments found by MiniJudgeJS are listed in the window below. Compare this list with your prototype set, shown below on the right. If necessary, modify the list of segments to make sure that they really represent the words and word strings in your prototype set that you want to vary across the other item sets. NOTE: The current version of MiniJudgeJS only permits automatic substitution for up to ten segments (any additional segments will just be copied across all sets). To reduce the number, you can combine segments, or delete segments that will not be varied across item sets (e.g. that in a study on that-trace effects).
When you are satisfied that the segments are correct, click OK.
Choose replacement segments for additional item sets.
To help MiniJudgeJS generate variants of your prototype set, enter syntactic equivalents for the prototype segments below. For example, to add two more item sets to the demo prototype set, the following new segments could be entered (the prototype set is Set 01):
01
who did
you
know
I saw
knew
him
02
03
For each of the prototype segments, enter matching substitutes for the new item sets (Sets 02 and up) in the cells below (if arrows appear at the edge, you should scroll around to make sure you fill them all). If you want to delete a prototype segment, enter a space. If you want to reuse a prototype segment, copy it from the header. You cannot add any new segment to those in the current prototype set. If you want to lengthen your prototype items to make deletion possible, click HERE. (NOTE: Do not edit headers or set numbers.)
When you are satisfied with your new segments, click OK.
ENTER MATCHING ADDITIONAL SEGMENTS HERE
Approve and save master list of test items.
If you decided to create the entire list of experimental items yourself, complete the list below, adding variant item sets in accordance with the indicated factor values (note that factor names are abbreviated to just one character).
Otherwise, carefully check the items below that were created by MiniJudgeJS, editing them where necessary (Note that the first number is the item number and the second is the set number; do not edit this information or the factor name(s) shown at the top).
When you are satisfied with the master list of items, click OK.
CHECK/EDIT MASTER LIST HERE
After you have approved the master list in the window above, select, copy, and save it to a file on your computer. You can give the file any name you like and save it anywhere, but you will need this information when you generate your survey forms in the next major step. If you must stop working now, restart your work at this step, first pasting your master list back into this window and reapproving it.
Run experiment.
MiniJudgeJS doesn't actually run the experiment for you. Instead, it generates survey forms for you to distribute to native speakers. You can copy the whole list of survey forms to a word processing program and paginate it for printing, one survey per sheet of paper. Or you can copy each survey into separate emails, one for each speaker.
IMPORTANT: If you are restarting your work at this point, you must first paste the master list back into the window above and reapprove it.
Choose number of speakers.
A single speaker is probably not enough to permit statistically reliable results. Moreover, individual speakers may have unknown idiosyncrasies. A reasonable number would be 10-25 speakers.
Enter your chosen number in the box below, click OK.
Number of speakers:
Write instructions for speakers.
In the current version of MiniJudge, responses must be binary YES/NO judgments (see HERE for justification of this limitation). If speakers give their responses by email, they must use "1" to represent YES (acceptable) and "0" to represent NO (unacceptable), and they must record their judgment for an item at the beginning of the line, before the item ID number. These restrictions should be made clear in the instructions.
Suggested instructions are shown in the window below (intended for use with English speakers filling out surveys by email at an institution requiring the use of informed consent forms). Modify them any way you like (e.g. translating them into the native language of your speakers, or deleting them entirely).
When you are satisfied with the instructions, click OK to generate the surveys.
EDIT INSTRUCTIONS HERE
IMPORTANT: Your participation in this study is entirely voluntary. If you wish to participate, you must read and electronically sign the attached informed consent form. For each item below, decide whether or not it seems English-sounding, that is, whether it could be spoken by a native speaker of English. If you think it does, put "1" before the item (at the beginning of the line). If you don't think so, put "0". If you aren't sure, put "1" or "0" anyway -- just guess! Please give a judgment for EVERY item, in the order that they are shown. Some items are quite similar to each other, but please consider each one separately. When you are done, send back the whole email with your judgments. After I get enough surveys back, I will tell you and everybody else what this is all about. Thanks for your help!
Save schematic survey information.
If you went through all of the above steps, MiniJudgeJS has taken your master list of items and your instructions, and has used them to generate the number of survey forms that you requested. The items have been put into a different partially random order for each survey. Randomization is only partial to ensure that item types are spread evenly within each survey (click HERE for more on the algorithm used). Randomization and even spread have another benefit, explained below and HERE.
The surveys given to speakers do not explicitly mark the factors and sets, since this information may bias speaker judgments. But MiniJudgeJS needs this information for the statistical analyses. The window below shows the crucial information in schematic form (removing irrel