EDEM 690: Assessment

Final Grade Breakdown

• 30%: Participation, in-class worksheets & peer feedback
• 20%: Structured critiques:
• I: qual paper assigned by me, which may be replaced by the make up structured critique for SCI
• II: qual papers from your own lit review
• III: quant papers assigned by me
• IV: quant papers from your own lit review
• 15%: Two research memos (Instructions: Memo 1 and Memo 2)
  
• 10%: Structured outline for final paper
  
• 30% 35%: Research proposal

Some major deliverables will be submitted twice: in the first week you'll get peer feedback, then you can revise before submitting it to me (Elizabeth) in the second week.

Paper Template

• Times New Roman size 12
• 3 cm (1 inch) margins
• 1.5 line spacing
  
• Header: includes your name and assignment's title (e.g. EDEM 690 Memo 1, EDEM 690 Structured Critique I)
• Footer: page number and page count (e.g. 1 / 5)
  
• APA citations
• If there are multiple sections, they should be numbered

Research Memo I [Due at start of class, Week 4]

1. What is the topic you're interested in for your research proposal, and why are you interested in it? [25 to 100 words]
   
2. What are three draft research questions you have, on that topic? [25-100 words]
   
3. With what you presently know, what data gathering methods do you think might be suitable? (Can be different for the different draft RQs) Why do you think that? [15-100 words]
   
4. With what you presently know, what data analysis methods do you think might be suitable? (Can be different for the different draft RQs) Why do you think that? [15-100 words]
5. With what you presently know, what theories do you think might be suitable? (Can be different for the different draft RQs) Why do you think that? [15-100 words]
   
6. Which research paradigm do you think you're most likely approaching this from? Why? [25-100 words]
   
7. Which social theoretic tradition do you think you're most likely approaching this from? Why? [25-100 words]
   
8. Has your view of research changed since the start of term? If so, how? [15-100 words]
   
9. Find 5 articles that look relevant to your topic & draft research questions, and import them into your citation manager of choice. Say which citation manager you are using. [1-10 words]. Provide evidence (e.g. screenshot, print-out of .bib file) you have completed this. You can put this in an appendix.
   
10. Based on your current knowledge, what paradigm(s) seems dominant in your research area? Which social theory(ies)? Which psychological tradition(s)? Explain your answers. [50-250 words]
    

• Q1-8: out of 2, marked:
• 0.5 point: incomplete, under/over word count (writing concisely matters!)
  
• 1 points: complete, but lacks internal logic, full of typos, or other clear flaw(s)
  
• 2 points: complete and reasonable (since this is for formative feedback, "reasonable" matters here more than "correct")
• Q9: out of 3, marked:
  
• 1 point: insufficient articles, or unclear they have been properly set up
  
• 2 points: articles not all clearly relevant
  
• 3 points: five relevant articles clearly imported into a citation manager
  
• Q10: out of 6, marked:
  For each of the three components:
  
  • 0.5 point: incomplete, under/over word count (writing concisely matters!)
    
  • 1 points: complete, but lacks internal logic, full of typos, or other clear flaw(s)
    
  • 2 points: complete and reasonable (since this is for formative feedback, "reasonable" matters here more than "correct")

Research Memo II

Structured Critique I

Structured Critique II

1. Article 1: 25. By question:
   
1. 1 pt: complete, appears correct
2. 1 pt: complete, appears correct
3. 2 pts: 1 pt each:
   
• describes publication venue & its audience
• describes how author seemed to scope/frame paper
4. 3 pts: 1 pt each:
• describes motivation/"tie-in to conversation" of paper
• describes the intellectual conversation: what works is the author talking about in the intro as related work
• describes the larger sociopolitical context
5. 2 pts: 1 pt each
• reasoning consistent with paradigm, identifies relevant things for identifying a paradigm from
  
• free of misconceptions
6. 4 pts, 1 pt each
• identifies theory used
• identifies tradition that theory comes from
• argues for/against the use of that theory
• the argument has solid reasoning
7. 1 pt: makes argument with substantiation
   
8. (combined with Q9 for marking)
9. 5 pts: 1pt each except for last one
   
• identifies how data sampled, collected, analysed
• argues for/against use of method/evidence
• reasoning for argument is solid
• identifies issues/flaws with the method/evidence [2 pts]
10. 2 pts, 1 pt each:
• identifies whose POV is there
• identifies whose POV isn't there
11. 2 pts, 1 pt each:
• identifies experiences/info/etc that would affect reading
• describes how that would affect the reading
12. 2 pts, 1 pt each:
• justification is given for choice of article
• justification is clear
  
2. Article 2: 15 points
   
1. 1 pt, same as Article 1
   
2. 1 pt, same as Article 1
3. 1 pt, does at least one of the two things from the Article 1 scheme
4. 2 pts, does at least two of the things from the Article 1 scheme
5. 2 pts, same as Article 1
6. 3 pts, does at least three of the things from the Article 1 scheme
7. 5 pts:
• 1 pt: identifies at least two things the articles have in common
• 1 pt: identifies at least two things the articles do differently
• 3 pts: discusses at least 3 flaws/weaknesses of the combined two papers
  

Structured Critique III

1. RQ - 1 point
   
2. Thesis - 1 point
3. Intended audience - 2 points, 1 pt each:
   
• describes publication venue & its audience
• describes how author seemed to scope/frame paper
4. Conversation - 3 points, 1 pt each:
   
• describes motivation/"tie-in to conversation" of paper
  
• describes the intellectual conversation: what works is the author talking about in the intro as related work
• describes the larger sociopolitical context
5. Paradigm - 4 points, 1 pt each:
   
• correct paradigm identified
  
• reasoning consistent with paradigm
• identifies relevant information for identifying a paradigm from
  
• free of misconceptions
6. Social theory - 4 pts, 1 pt each
• identifies social theory used
• identifies tradition that theory comes from
• argues for/against the use of that theory
• the argument has solid reasoning
7. Other theories - 4 pt, 1 pt each
   
• identifies other theory(ies) used
• identifies tradition that theory comes from
• argues for/against the use of that theory
• the argument has solid reasoning
8. (combined with Q9 for marking)
9. Methods/Evidence: 10 pts: 1pt each except for last one
   
• identifies how data sampled, collected, analysed
• argues for/against use of method/evidence
• reasoning for argument is solid
• identifies issues/flaws with the method/evidence/reporting [7 pts - 1 pt per issue/flaw identified]
10. POV - 2 pts, 1 pt each:
• identifies whose POV is there
• identifies whose POV isn't there
11. Reflection - 2 pts, 1 pt each:
• identifies experiences/info/etc that would affect reading
• describes how that would affect the reading
12. Choice - 2 pts, 1 pt each:
• justification is given for choice of article
• justification is clear

Structured Critique IV

Make-up For Structured Critique I

1. Hewner, M. (2013, August). Undergraduate conceptions of the field of computer science. In Proceedings of the ninth annual international ACM conference on International computing education research (pp. 107-114).
2. Cukier, W. (2003, April). Constructing the IT skills shortage in Canada: the implications of institutional discourse and practices for the participation of women. In Proceedings of the 2003 SIGMIS conference on Computer personnel research: Freedom in Philadelphia--leveraging differences and diversity in the IT workforce (pp. 24-33).
   

1. Who is the intended audience? Be specific. [per paper: 25-100 words; 1 point for correct, 1 point for identifying the relevant information]
   
2. What is the paradigm? How could you tell? [per paper: 25-100 words; 1 point for correct, 1 point for identifying the relevant information]
   
3. Which social theoretic tradition is used? If there is a social theory used, which is it? [per paper: 25-100 words; 1 point for correct, 1 point for identifying the relevant information]
   
4. List flaws of this paper, such as in the sampling, choice of theory, analysis, reporting, etc. [per paper: 150-700 words]
   [13 points for both papers. Identify 13 flaws between the two papers, with at least 3 flaws identified for each paper.
   0.5 for correctly identifying a flaw in the paper and 0.5 for concisely explaining why it's a flaw.
   If you are able to identify more than 13 flaws, good job! I will give at most one bonus point for identifying more than 13 flaws.]
   

Research Proposal

Your goal is to propose a short-term research study. Deadline: April 20 April 30, 2020 - submit as a pdf on mycourses.

• Using the standard template, max 6 pages. This does not include the bibliography or any optional appendices.
• APA style citations
• Will have to clearly articulate and motivate:
  
• Your research question
• Why addressing your research question provides an expected contribution to the literature
• The theories that will be guiding your approach
  
• The methods you'll use for gathering data (including how you'll recruit participants, if applicable)
  
• The methods you'll use for analysing data
• Will have to clearly situate your work in the relevant literature (includes: what paradigm are you vs your community, what theories you use vs them, etc)
• Will have to discuss the limitations of your work
• Identify what the limitations are
• Discuss ways they could be mitigated
• Identify and address any criticisms an opponent would have of your research
• Reflect on what you would do differently if you were to do this all over again from scratch
  

Marking scheme, out of 50:

• 10 points: Organization. Easy to find important information, such as through useful section headers and appropriately separated paragraphs. Able to selectively read the paper to get important information.
• 10 points: Clarity of writing. I am not looking for flowery prose. I want a concise, accessible text. You are explicitly permitted to share drafts of your paper with classmates and friends to get proofreading assistance!
• 5 points: Research question. 1 point each for:
• There is a research question
• It has an appropriate level of specificity / can be something answered
• It is relevant to the approach you outline
  
• Motivation for why you're asking it
• Argumentation that answering this research question would contribute to the literature
• 5 points: Situating research
• You explicitly identify what research community you envision as your audience
• You correctly identify the paradigm (e.g. positivism, interpretivism) that your study plan reflects
• You correctly identify the social theoretic tradition (e.g. conflict theoretic, symbolic interactionist) that your proposed study is part of
• You compare how your study's paradigm & social theoretic tradition relates to the standards of your chosen audience
• You compare how your study's methods relate to the standards of your chosen audience
  
• 5 points: Plan for gathering data
• A plan for recruitment and sampling is articulated
• The choice of approach for recruitment & sampling is justified
• The type of the data (e.g. Likert data, diary entries) is articulated
• The elicitation of data (e.g. survey questions, interview prompts) is described
  (you may put proposed survey questions, interview protocol, etc in an appendix)
• The approach to eliciting data is justified
  
• 5 points: Plan for analysing data
• A theoretical/conceptual lens is identified
• A theoretical/conceptual lens is justified as appropriate
• How the theoretical/conceptual lens will be used in the analysis is evident
• An analytic method (e.g. thematic analysis, chi-square test) is identified
• The analytic method is justified as appropriate
• 10 points: Discussion
• 5 points, 1 point each: Identification of at least five distinct limitations / potential criticisms of your approach (e.g. threat to validity/reliability, problematic conceptual assumptions)
• 5 points, 1 point each: For each of these anticipated criticisms, crafting a response to the anticipated critic (e.g. how you could mitigate the threat to validity/reliability, justifying your choice of theory vs others)