EDEM 690: Assessment
[back
to syllabus]
Final Grade Breakdown
- 30%:
Participation, in-class worksheets & peer feedback
- 20%:
Structured critiques:
- 15%: Two
research memos (Instructions: Memo 1 and
Memo 2)
10%:
Structured outline for final paper
30%
35%: Research proposal
Some major deliverables
will be submitted twice: in the first week you'll get peer
feedback, then you can revise before submitting it to me
(Elizabeth) in the second week.
Details about each
deliverable will be posted on the website as the term goes on.
Paper
Template
For research memos,
structured critiques, outline, and research proposal.
Up to 30% may
deducted for failing to follow the paper template.
- Times New Roman size
12
- 3 cm (1 inch)
margins
- 1.5 line spacing
- Header: includes
your name and assignment's title (e.g. EDEM 690 Memo 1, EDEM
690 Structured Critique I)
- Footer: page number
and page count (e.g. 1 / 5)
- APA citations
- If there are
multiple sections, they should be numbered
An example word
template is here for you to download (File ->
Download). A LaTeX
template for you is also available.
When submitting to
me, attach pages by paper clip rather than staple.
Please use double sided printing to save trees. :)
Research
Memo I [Due at start of class, Week 4]
In this course, the final
deliverable is a research proposal. This and other deliverables
are designed to help you get formative feedback toward your
research proposal.
Using the template, answer these
questions (can just use the numbers):
- What is the topic
you're interested in for your research proposal, and why are
you interested in it? [25 to 100 words]
- What are three draft
research questions you have, on that topic? [25-100 words]
- With what you
presently know, what data gathering methods do you think might
be suitable? (Can be different for the different draft RQs)
Why do you think that? [15-100 words]
- With what you
presently know, what data analysis methods do you think might
be suitable? (Can be different for the different draft RQs)
Why do you think that? [15-100 words]
- With what you
presently know, what theories do you think might be suitable?
(Can be different for the different draft RQs) Why do you
think that? [15-100 words]
- Which research
paradigm do you think you're most likely approaching this
from? Why? [25-100 words]
- Which social
theoretic tradition do you think you're most likely
approaching this from? Why? [25-100 words]
- Has your view of
research changed since the start of term? If so, how? [15-100
words]
- Find 5 articles that
look relevant to your topic & draft research questions,
and import them into your citation manager of choice. Say
which citation manager you are using. [1-10 words]. Provide
evidence (e.g. screenshot, print-out of .bib file) you have
completed this. You can put this in an appendix.
- Based on your
current knowledge, what paradigm(s) seems dominant in your
research area? Which social theory(ies)? Which psychological
tradition(s)? Explain your answers. [50-250 words]
Marking, out of 25:
- Q1-8: out of 2,
marked:
- 0.5 point:
incomplete, under/over word count (writing concisely
matters!)
- 1 points:
complete, but lacks internal logic, full of typos, or other
clear flaw(s)
- 2 points: complete
and reasonable (since this is for formative feedback,
"reasonable" matters here more than "correct")
- Q9: out of 3,
marked:
- 1 point:
insufficient articles, or unclear they have been properly
set up
- 2 points: articles
not all clearly relevant
- 3 points: five
relevant articles clearly imported into a citation manager
- Q10: out of 6,
marked:
For each of the three components:
- 0.5 point:
incomplete, under/over word count (writing concisely
matters!)
- 1 points:
complete, but lacks internal logic, full of typos, or
other clear flaw(s)
- 2 points:
complete and reasonable (since this is for formative
feedback, "reasonable" matters here more than "correct")
Research
Memo II
This is an update to Memo
I, and gives you a chance to refine your ideas and update me on
your progress.
Using the same template as before, answer the questions #1-7 from
Memo I. Increase all the word count maxima by 50% (i.e. 50-100
words becomes 50-150 words]). If your answers have not
changed since Memo I you may leave them as they were.
Then answer:
8. What has changed since Memo 1 when it comes to questions 1-7?
Why? [15-150 words]
9. Who is the intended audience for your study? If you were to
publish it, where would you want to publish it (realistically) and
why? [50-100 words]
10. Presumably the people in that research community have been
discussing your topic or similar topics. What's the "Conversation"
that you would be entering into? [50-150 words]
Structured
Critique I
Due at start of class in
week 6.
In this milestone you'll be categorizing and critiquing a single
qualitative paper using the same framework that you'll use for the
other structured critiques.
This milestone gives you practice with the framework using a
common paper for the whole class. In the next milestone you'll
pick two papers to use this framework with. The framework is a
series of 12 questions to answer and justify about the paper.
Note: be sure to use page numbers in your in-line citations.
Common paper: Barker
et al.
Framework
and template
Please note a title page is not needed nor desired.
Structured
Critique II
Due at start of class in
week 7. (Red text added Feb 11 per Slack
discussion.)
Pick two qualitative studies related to your research
proposal topic. Using the framework from Structured Critique I,
categorize and critique these two papers the first paper.
Then for the second paper, answer questions 1-6 from the
template, and a new question 7 below.
For Q3 and Q4 the word limit is now [20-300 words]. For Q6 the
word limit is now [50-300 words].
The new Q7 is: "Compare and contrast the two papers. [100-300
words]"
Note: you may use bullet points for the answers to the
questions in this assignment.
A worked
example for one qualitative study is available here.
Marking scheme: / 40
- Article 1: 25. By
question:
- 1 pt: complete,
appears correct
- 1 pt: complete,
appears correct
- 2 pts: 1 pt each:
- describes
publication venue & its audience
- describes how
author seemed to scope/frame paper
- 3 pts: 1 pt each:
- describes
motivation/"tie-in to conversation" of paper
- describes the
intellectual conversation: what works is the author
talking about in the intro as related work
- describes the
larger sociopolitical context
- 2 pts: 1 pt each
- reasoning
consistent with paradigm, identifies relevant things for
identifying a paradigm from
- free of
misconceptions
- 4 pts, 1 pt each
- identifies
theory used
- identifies
tradition that theory comes from
- argues
for/against the use of that theory
- the argument has
solid reasoning
- 1 pt: makes
argument with substantiation
- (combined with Q9
for marking)
- 5 pts: 1pt each
except for last one
- identifies how
data sampled, collected, analysed
- argues
for/against use of method/evidence
- reasoning for
argument is solid
- identifies
issues/flaws with the method/evidence [2 pts]
- 2 pts, 1 pt each:
- identifies whose
POV is there
- identifies whose
POV isn't there
- 2 pts, 1 pt each:
- identifies
experiences/info/etc that would affect reading
- describes how
that would affect the reading
- 2 pts, 1 pt each:
- justification is
given for choice of article
- justification is
clear
- Article 2: 15 points
- 1 pt, same as
Article 1
- 1 pt, same as
Article 1
- 1 pt, does at
least one of the two things from the Article 1 scheme
- 2 pts, does at
least two of the things from the Article 1 scheme
- 2 pts, same as
Article 1
- 3 pts, does at
least three of the things from the Article 1 scheme
- 5 pts:
- 1 pt: identifies
at least two things the articles have in common
- 1 pt: identifies
at least two things the articles do differently
- 3 pts: discusses
at least 3 flaws/weaknesses of the combined two papers
Structured
Critique III
Due at start of class in
week 11 week 13. To submit,
DM a copy to me on Slack.
Using the same framework as the previous critiques. This time
there is a single quantitative paper for everybody to critique:
Zingaro, Daniel,
and Leo Porter. "Impact of student achievement
goals on CS1 outcomes." In Proceedings of
the 47th ACM technical symposium on Computing Science Education,
pp. 279-296. ACM, 2016.
You may complete this in groups of at most 3.
If doing so, ensure all your names are in the header. I
strongly recommend that if doing this in a group, to first make a
first pass over the questions on your own. Then get
together with your group and compare & flesh out your answers.
You will not learn as much if you just
divide & conquer the assignment.
Marking scheme: / 35
- RQ - 1 point
- Thesis - 1 point
- Intended audience -
2 points, 1 pt each:
- describes
publication venue & its audience
- describes how
author seemed to scope/frame paper
- Conversation - 3
points, 1 pt each:
- describes
motivation/"tie-in to conversation" of paper
- describes the
intellectual conversation: what works is the author talking
about in the intro as related work
- describes the
larger sociopolitical context
- Paradigm - 4 points,
1 pt each:
- correct paradigm
identified
- reasoning
consistent with paradigm
- identifies
relevant information for identifying a paradigm from
- free of
misconceptions
- Social theory - 4
pts, 1 pt each
- identifies social
theory used
- identifies
tradition that theory comes from
- argues for/against
the use of that theory
- the argument has
solid reasoning
- Other theories - 4
pt, 1 pt each
- identifies other
theory(ies) used
- identifies
tradition that theory comes from
- argues for/against
the use of that theory
- the argument has
solid reasoning
- (combined with Q9
for marking)
- Methods/Evidence: 10
pts: 1pt each except for last one
- identifies how
data sampled, collected, analysed
- argues for/against
use of method/evidence
- reasoning for
argument is solid
- identifies
issues/flaws with the method/evidence/reporting [7 pts - 1
pt per issue/flaw identified]
- POV - 2 pts, 1 pt
each:
- identifies whose
POV is there
- identifies whose
POV isn't there
- Reflection - 2 pts,
1 pt each:
- identifies
experiences/info/etc that would affect reading
- describes how that
would affect the reading
- Choice - 2 pts, 1 pt
each:
- justification is
given for choice of article
- justification is
clear
Feb 18, 2020: Based on student
feedback, structured critique IV will not happen.
Structured
Critique IV
Due at start of class in
week 10.
Pick two quantitative studies related to your research proposal topic.
Using the framework from Structured Critique II, categorize
and critique these two papers.
Make-up
For Structured Critique I
Due at start of class in week
12 week 14. This
is an optional assignment. If you do this assignment, and do
better on it than Structured Critique I, I will replace your SCI
grade with this one.
To submit: send it to me as a DM on Slack, as a pdf.
If you are working in a group, send it in a group conversation
that includes all of your group members.
Two papers:
- Hewner, M. (2013,
August). Undergraduate
conceptions of the field of computer science. In
Proceedings of the ninth annual international ACM conference
on International computing education research (pp. 107-114).
- Cukier, W. (2003,
April). Constructing
the IT skills shortage in Canada: the implications of
institutional discourse and practices for the participation
of women. In Proceedings of the 2003 SIGMIS conference
on Computer personnel research: Freedom in
Philadelphia--leveraging differences and diversity in the IT
workforce (pp. 24-33).
Answer for each: [25
points - one bonus point possible]
- Who is the
intended audience? Be specific. [per paper: 25-100 words; 1 point
for correct, 1 point for identifying the relevant information]
- What is the
paradigm? How could you tell? [per paper: 25-100 words; 1 point
for correct, 1 point for identifying the relevant information]
- Which
social theoretic tradition is used? If
there is a social theory used, which is it? [per paper: 25-100 words; 1 point for correct, 1
point for identifying the relevant information]
- List flaws of
this paper, such as in the sampling, choice of theory,
analysis, reporting, etc. [per paper: 150-700 words]
[13 points for both papers. Identify 13 flaws between the two
papers, with at least 3 flaws identified for each paper.
0.5 for correctly identifying a flaw in the paper and 0.5 for
concisely explaining why it's a flaw.
If you are able to
identify more than 13 flaws, good job! I will give at most
one bonus point for identifying more than 13 flaws.]
As usual,
the assignment must be formatted with the standard template
for this course. Point form answers are acceptable.
You may complete this in groups of at most 3. If doing so,
ensure all your names are in the header. I strongly
recommend that if doing this in a group, to first make a first
pass over the questions on your own. Then get together
with your group and compare & flesh out your answers. You
will not learn as much if you just divide & conquer the
assignment.
Mar 22, 2020: The Structured Outline
for the Research Proposal has been canceled due to the Covid-19
pandemic.
Research
Proposal
Your goal is to propose
a short-term research study. Deadline: April 20 April 30, 2020 -
submit as a pdf on mycourses.
- Using the standard
template, max 6 pages. This does not include the bibliography
or any optional appendices.
- Will have to clearly
articulate and motivate:
- Your research
question
- Why addressing
your research question provides an expected contribution to
the literature
- The theories that
will be guiding your approach
- The methods you'll
use for gathering data (including how you'll recruit
participants, if applicable)
- The methods you'll
use for analysing data
- Will have to clearly
situate your work in the relevant literature (includes: what
paradigm are you vs your community, what theories you use vs
them, etc)
- Will have to discuss
the limitations of your work
- Identify what the
limitations are
- Discuss ways they
could be mitigated
- Identify and
address any criticisms an opponent would have of your
research
- Reflect on what
you would do differently if you were to do this all over
again from scratch
Marking scheme, out of
50:
- 10 points: Organization.
Easy to find important information, such as through useful
section headers and appropriately separated paragraphs. Able
to selectively read the paper to get important information.
- 10 points: Clarity
of writing. I am not looking for flowery prose. I want a
concise, accessible text. You are explicitly permitted to
share drafts of your paper with classmates and friends to get
proofreading assistance!
- 5 points: Research
question. 1 point each for:
- There is a
research question
- It has an
appropriate level of specificity / can be something answered
- It is relevant to
the approach you outline
- Motivation for why
you're asking it
- Argumentation that
answering this research question would contribute to the
literature
- 5 points: Situating
research
- You explicitly
identify what research community you envision as your
audience
- You correctly
identify the paradigm (e.g. positivism, interpretivism) that
your study plan reflects
- You correctly
identify the social theoretic tradition (e.g. conflict
theoretic, symbolic interactionist) that your proposed study
is part of
- You compare how your
study's paradigm & social theoretic tradition relates
to the standards of your chosen audience
- You compare how your
study's methods relate to the standards of your chosen
audience
- 5 points: Plan
for gathering data
- A plan for
recruitment and sampling is articulated
- The choice of
approach for recruitment & sampling is justified
- The type of the
data (e.g. Likert data, diary entries) is articulated
- The elicitation of
data (e.g. survey questions, interview prompts) is described
(you may put
proposed survey questions, interview protocol, etc in an
appendix)
- The approach to
eliciting data is justified
- 5 points: Plan
for analysing data
- A
theoretical/conceptual lens is identified
- A
theoretical/conceptual lens is justified as appropriate
- How the
theoretical/conceptual lens will be used in the analysis is
evident
- An analytic method
(e.g. thematic analysis, chi-square test) is identified
- The analytic
method is justified as appropriate
- 10 points: Discussion
- 5 points, 1 point
each: Identification of at least five distinct limitations /
potential criticisms of your approach (e.g. threat to
validity/reliability, problematic conceptual assumptions)
- 5 points, 1 point
each: For each of these anticipated criticisms, crafting a
response to the anticipated critic (e.g. how you could
mitigate the threat to validity/reliability, justifying your
choice of theory vs others)