WOLLMAN, Circuit Judge.
Cully Mitchell was convicted of conspiracy to launder money, in violation of 18 U.S.C. § 1956. The district court sentenced Mitchell to forty-four months' imprisonment. Mitchell appeals his conviction and sentence. We affirm the conviction, but remand for reconsideration of the sentence.
I.
An investigation led by Postal Inspector William Witt revealed that 290 postal money orders were purchased in St. Louis, Missouri, and cashed in Los Angeles, California, between 2003 and 2006. Initially, the money orders were purchased by Damion Epherson and cashed by him and others. Beginning on April 15, 2004, Mitchell began cashing some of the money orders. The money orders were purchased in denominations that usually totaled $2,800, which was $200 less than the threshold for reporting the transactions to the Treasury Department. The investigation revealed that fifty-six money orders, totaling $52,600, listed Mitchell as the payee or payor.
On August 12, 2007, Witt and Postal Inspector James Davis, went to the residence of Erika Whitaker, Mitchell's wife. Whitaker had cashed one of the money orders that listed Mitchell as the payor. At Whitaker's residence, Witt and Davis encountered Mitchell, who was living with Whitaker at the time. Witt and Davis explained that they were conducting an investigation into money laundering involving the use of postal money orders.
Mitchell cooperated with the investigators and candidly explained his involvement. According to Witt, Mitchell stated that Epherson was a childhood acquaintance and that Epherson had paid him between $50 and $100 for every $1,000 that he cashed. Mitchell stated that he believed Epherson was a drug dealer. On one occasion, Mitchell had asked Epherson to sell him an ounce of cocaine for personal use. According to Mitchell, Epherson declined, responding that he did not sell quantities that small. Mitchell stated that he had no first-hand knowledge of Epherson's business in St. Louis, but assumed that Epherson was selling drugs there. According to Witt, Mitchell also stated that he thought the funds used to purchase the postal money orders were derived from Epherson's drug business.
In May 2008, Epherson and Mitchell were indicted for money laundering. The indictment alleged that Mitchell "knowingly and willingly" conspired to launder the proceeds of "unlawful activity." The indictment further alleged that Mitchell had done so by structuring money order purchases from the Post Office, and that he had done so "knowing that the transaction is designed in whole or in part to conceal or disguise the nature, source or ownership of the proceeds of such unlawful activity."
Mitchell was tried in May 2009. At trial, Postal Inspector Witt was the sole witness. Witt sketched out the facts as set forth above, detailing his experiences investigating money laundering, discussing the documentary record in this case, and relating the statements that Mitchell made to him. Over Mitchell's objection, the government introduced Mitchell's tax return information for 2003, 2004, 2005, and 2006. Discussing this information, Witt explained to the jury that there was no record of Mitchell filing income taxes for those years. Mitchell's defense was that the government had not proved beyond a reasonable doubt that the money orders were purchased with funds from drug dealing, that Mitchell did not know that he was laundering money for a drug dealer, and that Witt's testimony about Mitchell's inculpatory statements was unreliable. The jury returned a verdict of guilty.
At sentencing, the court considered the recommendation of the presentence investigation report (PSR). The base offense level was eight. Six levels were added because the value of the funds involved was more than $30,000, but less than $70,000, pursuant U.S.S.G. § 2B1.1(b)(1)(D). The PSR recommended a six-level increase pursuant to U.S.S.G. § 2S1.1(b)(1), because the defendant knew or believed the funds were the proceeds of, or intended to promote an offense involving the distribution of a controlled substance. The PSR also recommended a four-level increase because the defendant was "in the business of laundering funds," pursuant to U.S.S.G. 8 2S1.1(b)(2)(C), for a total offense level of twenty-four. The PSR calculated Mitchell's criminal history category as I, for an advisory guideline range of fifty-one to sixty-three months' imprisonment.
Mitchell set forth his objections in a lengthy sentencing memorandum and iterated his objections at the sentencing hearing. R. at 20-46; Sent. Tr. at 2-6, 10-18. Mitchell objected to the enhancement for laundering funds derived from drug distribution and the enhancement for being in the business of money laundering. R. at 27-34; Sent. Tr. at 12-17. He claimed that he did not know the funds were the proceeds of or intended to promote the distribution of a controlled substance. R. at 28-29; Sent. Tr. at 12. Moreover, he argued that he was not in the business of laundering funds, describing his offense as an idiosyncratic scheme with a childhood friend. R. at 34; Sent. Tr. at 13-15. He stated that the evidence presented at trial showed that he received between $2,250 and $4,500 for negotiating the money orders, and that this was not a substantial amount of revenue. R. at 33-34; Sent. Tr. at 14. Last, he argued for a three-level minor role reduction. R. at 34-39; Sent. Tr. at 15-18. In sum, Mitchell believed that the appropriate offense level was 11, with a guideline range of 8 to 14 months' imprisonment. R. at 39; Sent. Tr. at 6.
The court denied Mitchell's objections and declined to grant a minor role reduction. The court, however, varied downward from the guideline range and sentenced Mitchell as set forth above.
II. A. Sufficiency of the Indictment
Mitchell argues that the indictment was deficient because it failed to allege all of the elements of the offense charged and that this deficiency was prejudicial. Specifically, he attacks the indictment for failing to allege that he knew the funds used to purchase the money orders were proceeds from unlawful activity.
"When an indictment is challenged for the first time after the verdict is returned, we apply a deferential standard of review, upholding the indictment unless it is so defective that by no reasonable construction can it be said to charge the offense for which the defendants were convicted." United States v. Jenkins-Watts, 574 F.3d 950, 968 (8th Cir. 2009) (internal punctuation omitted), cert. denied, ___ U.S. ___, 130 S.Ct. 1915, 176 L.Ed.2d 387 (2010); see also United States v. Just, 74 F.3d 902, 904 (8th Cir. 1996) ("Although the sufficiency of an indictment is a jurisdictional issue that may be raised at any time, an indictment that is challenged after jeopardy has attached will be liberally construed in favor of sufficiency.").
As set forth above, the indictment alleged, inter alia, that Mitchell had knowingly and willfully conspired with Epherson to violate 18 U.S.C. § 1956. Mitchell did not object to the indictment prior to trial. The indictment articulated the government's allegations regarding the manner and means of the conspiracy. The indictment was not so defective that Mitchell could not have reasonably understood the offense for which he was charged. Accordingly, the indictment was sufficient. See United States v. Wydermyer, 51 F.3d 319, 325 (2d Cir. 1995) (holding that indictment need not allege defendant knew funds derived from unlawful activity when indictment alleges defendant knowingly attempted to commit the offense); see also United States v. Diaz-Diaz, 135 F.3d 572, 575-77 (8th Cir. 1998) (discussing when deficiency in the indictment warrants reversal).
B. Admission of Tax Return Information
Mitchell argues that the district court abused its discretion by admitting his tax return information. He claims that the tax return information was not relevant and that any probative value was outweighed by the potential prejudicial effect.
We review the district court's ruling on the admissibility of evidence for abuse of discretion. United States v. Chase, 451 F.3d 474, 479 (8th Cir. 2006). Evidence is relevant if it has "any tendency to make the existence of any fact that is of consequence to the determination of the action more probable or less probable than it would be without the evidence." Fed.R.Evid. 401. All relevant evidence is admissible, but relevant evidence may be excluded when its probative value is substantially outweighed by the danger of unfair prejudice. Fed.R.Evid. 402, 403.
The district court overruled Mitchell's objection, stating that the tax information was relevant to show a plan to conceal the transactions and that its relevance appeared to outweigh the potential prejudicial effects. Failure to file tax returns is relevant to the existence of, and a defendant's participation in, a money laundering conspiracy. United States v. Mangual-Santiago, 562 F.3d 411, 429 (1st Cir.), cert. denied ___U.S. ___, 130 S.Ct. 293, 175 L.Ed.2d 196 (2009). The tax information was thus relevant to the government's assertion that Mitchell was engaged in a conspiracy to launder money. Mitchell has not shown that the danger of unfair prejudice substantially outweighed the probative value of the tax information. Accordingly, the district court did not abuse its discretion by admitting the tax information into evidence.
C. Adverse Inference During Closing Argument
Mitchell argues that the district court abused its discretion by preventing him from suggesting to the jury that it draw a "missing witness" adverse inference against the government. He wanted to argue during closing argument that the government's decision not to call Inspector Davis to testify cast doubt upon the veracity of Witt's testimony.
We review for abuse of discretion a district court's ruling on the propriety of a missing witness adverse inference. Smith v. Copeland, 87 F.3d 265, 269 (8th Cir. 1996). To constitute reversible error, the district court's prohibition of an adverse inference argument must have "resulted in injury or prejudice." Id. The government stated that Inspector Davis was available to testify if Mitchell so wanted; Mitchell declined the government's invitation. Thus, an adverse inference argument based upon a missing witness was inappropriate. See United States v. Cole, 380 F.3d 422, 427 (8th Cir. 2004) (stating that in the context of jury instructions, adverse inferences from missing witnesses are not appropriate "where the defense does not show that the government possesses the sole power to produce the witness"). Mitchell has not shown that he suffered any injury or prejudice from the district court's decision. Accordingly, the district court did not abuse its discretion in ruling as it did.
D. Jury Instructions
Mitchell argues that the district court abused its discretion by rejecting a proposed jury instruction on the definition of "proceeds." Mitchell sought to narrow the definition of proceeds to include only profits, rather than gross revenue. We review for abuse of discretion the district court's formulations of jury instructions. United States v. Cruz-Zuniga, 571 F.3d 721, 725 (8th Cir. 2009). We will reverse if the failure to properly instruct the jury was prejudicial. Id.
For purposes of 18 U.S.C. § 1956, proceeds equates with gross receipts when the funds are derived from drug sales. United States v. Williams, 605 F.3d 556, 567-68 (8th Cir. 2010). Mitchell argues that United States v. Santos, 553 U.S. 507, 128 S.Ct. 2020, 170 L.Ed.2d 912 (2008), compels a different conclusion. Our precedent, however, is to the contrary. Williams, 605 F.3d at 568 (holding that Santos does not apply in the drug context); see United States v. Spencer, 592 F.3d 866, 879-80 (8th Cir. 2010) (same). Thus, Mitchell cannot show any prejudice from the jury instruction as given. Accordingly, the district court did not abuse its discretion in declining to instruct the jury as Mitchell sought, and he is entitled to no relief on this issue.
E. Sufficiency of the Evidence
Mitchell argues that the evidence presented at trial was insufficient to prove beyond a reasonable doubt that he conspired to violate the money laundering statute. Specifically, he alleges that the government failed to show that he knew the transactions were designed to conceal the source of funds.
We review de novo the denial of a motion for judgment of acquittal. United States v. Hodge, 594 F.3d 614, 617 (8th Cir.), cert. denied, ___ U.S. ___, 130 S.Ct. 3401, 177 L.Ed.2d 314 (2010). "When considering a challenge to a conspiracy conviction based upon sufficiency of the evidence, we will affirm if the record, viewed most favorably to the government, contains substantial evidence supporting the jury's verdict, which means evidence sufficient to prove the elements of the crime beyond a reasonable doubt." Id. (internal punctuation omitted).
We conclude that the evidence was sufficient to convict Mitchell of money laundering. The indictment alleged that Mitchell conspired to violate 18 U.S.C. § 1956(a)(1)(A)(i), also known as the "promotion prong" of § 1956. The jury instruction stated that Mitchell was accused of violating § 1956(a)(1)(A)(i) by promoting unlawful activity. The jury was not instructed to consider whether Mitchell cashed the money orders to conceal or disguise the source of the funds in violation of § 1956(a)(1)(B)(i), also known as the "concealment prong" of § 1956. Accordingly, Mitchell's argument about whether there was sufficient evidence of an intent to conceal is misplaced. When considered in the light most favorable to the jury's verdict, there was substantial evidence that Mitchell had violated the promotion prong of § 1956. Thus, the district court did not err in denying Mitchell's motion for acquittal.
F. Sentencing Issues
Mitchell contests the district court's calculation of his sentence. He claims that the district court erred in three ways: (1) imposition of a six-level enhancement because he knew or believed that the money he was laundering was the proceeds of or intended to promote the distribution of a controlled substance, (2) imposition of a four-level increase because he was in the business of money laundering, and (3) failure to reduce his offense level by three because he was a minor participant in the conspiracy.
We review the district court's application of the guidelines de novo. United States v. Martin, 583 F.3d 1068, 1073 (8th Cir. 2009), cert. denied, ___ U.S. ___, 130 S.Ct. 2355, 176 L.Ed.2d 570 (2010). Predicate factual findings are reviewed for clear error. 18 U.S.C. § 3742(e); Martin, 583 F.3d at 1073.
1.
The district court imposed a six-level enhancement pursuant to U.S.S.G. § 2S1.1(b)(1), which provides for an enhancement if the defendant "knew or believed that any of the laundered funds were the proceeds of, or were intended to promote (i) an offense involving the manufacture, importation, or distribution of a controlled substance or a listed chemical." Witt's uncontroverted testimony was that Mitchell said that he believed the money orders were the proceeds of Epherson's drug trafficking in St. Louis. Mitchell had sought to purchase cocaine for personal use from Epherson. As Mitchell related to Witt, Epherson demurred, stating that he did not truck in such small quantities of cocaine. Mitchell claims that this evidence was insufficient to justify the enhancement because it did not prove that the funds were, in fact, derived from drug trafficking.
Mitchell's focus is misplaced. The sentencing enhancement in § 2S1.1(b)(1) does not turn on any objective characteristic of the funds, but rather upon the subjective knowledge or belief of the defendant. See United States v. Payne, 962 F.2d 1228, 1235 (6th Cir. 1992) (holding that defendant's subjective belief that funds were proceeds of unlawful activity was sufficient for enhancement to apply). The district court found that Mitchell "knew that the laundered proceeds were intended to promote the offense involving the distribution of controlled substances." Based upon the evidence presented at trial, this finding was not clearly erroneous. Even if Mitchell lacked such knowledge, he wrould still qualify for the enhancement because he believed the funds derived from Epherson's drug dealing. Accordingly, the district court did not err in applying this six-level enhancement.
2.
The district court also imposed a four-level enhancement pursuant to U.S.S.G. § 2S1.1(b)(2)(C), determining that Mitchell was "in the business of laundering funds." Mitchell claims that he was not in the business of laundering funds and portrays his crime as a singular, albeit misguided, scheme with a childhood friend. Asserting that "Mitchell was in the regular business of money laundering," the government focuses on the length of time that Mitchell dealt with Epherson and on Mitchell's failure to report any other source of legitimate income.
The guideline commentary to § 2S1.1 instructs the court to look to the totality of circumstances to determine whether a defendant was in the business of laundering funds. U.S.S.G. § 2S1.1, cmt. n. 4(A). The commentary lists six factors for this analysis: whether (1) the defendant regularly engaged in laundering funds; (2) the defendant laundered funds for an extended period of time; (3) the defendant engaged in laundering funds from multiple sources; (4) the defendant generated a substantial amount of revenue in return for laundering funds; (5) the defendant had a prior conviction for a money laundering related offense; and (6) the defendant made statements during the course of an undercover government investigation that he had engaged in any of the conduct listed in factors (1), (2), (3), or (4). U.S.S.G. § 2S1.1, cmt. n. 4(B)(i)-(vi). The commentary to the 2001 amendment to the guidelines, which codified this enhancement, states:
With respect to third party money launderers, subsection (b)(2)(C) provides a four-level enhancement if the defendant is "in the business" of laundering funds. The Commission determined that, similar to a professional "fence",
see § 2B1.1(b)(4)(B), defendants who routinely engage in laundering funds on behalf of others, and who gain financially from engaging in such transactions, warrant substantial additional punishment because they encourage the commission of additional criminal conduct.
U.S.S.G. app. C, amend. 634, C-736 (2009). As the commentary suggests, the question is whether the defendant was similar to a professional fence, that is, someone who is in the business of laundering funds on the behalf of others and who gains financially from engaging in such transactions. Id.
A professional fence is an individual who is part of a business that moves contraband. United States v. Collins, 104 F.3d 143, 144 (8th Cir. 1997). Insofar as they run an "underworld business," they cause greater community blight than if they act in a less entrepreneurial manner. United States v. St. Cijv, 977 F.2d 698, 702 (1st Cir. 1992). Accordingly, the sentencing commission has decided that such individuals deserve longer sentences because the amount of funds involved in the case at bar likely underestimates the scope of their criminality and the extent to which they encouraged other crimes. United States v. Warshawsky, 20 F.3d 204, 214-15 (6th Cir. 1994). It is unclear whether Mitchell was akin to a professional fence and thus properly considered as being in the business of money laundering.
Mitchell did not have a prior money laundering related conviction, he did not make any statements during an undercover government investigation, nor did he deal with multiple sources. Thus, the propriety of the enhancement depended on whether he regularly laundered money for an extended period of time and generated a substantial amount of revenue therefrom. U.S.S.G. § 2S1.1, cmt. n. 4(B)(i)-(vi). It is true that Mitchell was regularly engaged in money laundering for an extended period of time, sixteen to eighteen months. The district court concluded that Mitchell had "generated a substantial amount of money" based upon the premise that he had garnered $42,000 from these transactions. The district court apparently derived this figure from the PSR, which stated that Mitchell had negotiated money orders totaling approximately $42,000. The evidence presented at trial, however, was to the contrary: Epherson paid Mitchell between $50 and $100 per money order cashed, and the evidence was that Mitchell cashed approximately forty-five money orders for Epherson. Thus, Mitchell generated between $2,250 and $4,500 in total from Epherson over sixteen to eighteen months.
The government brought this discrepancy to our attention by way of a letter, in which it stated that it would have no objection to a remand to permit the district court to reconsider the application of the enhancement if our court concluded that the factual error might have affected the district court's decision in this regard.
We conclude that such a remand is warranted, because the difference between the two calculations is significant for determining whether Mitchell derived "substantial revenue" from these transactions and whether the enhancement should apply. See United States v. Aguasvivas-Castillo, No. 07-0111 CCC, 2010 WL 883719, *2 (D.P.R. Mar. 5, 2010) (holding enhancement properly applied when defendant received at least $334,493 in kickbacks from laundering at least $4.4 million at three supermarkets); United States v. Hosseini, No. 05 CR 254, 2007 WL 2904015, *2 (N.D.Ill. Oct. 1, 2007) (holding that "massive program of structuring" $15 million in deposits rose "to the level of conducting a business"); United States v. Quarshie, No. 04 CR. 1148-5, 2007 WL 107754, *l-3 (S.D.N.Y. Jan. 16, 2007) (holding that defendant was not in the business of laundering funds when she received only $4,000 from laundering $100,000 for one person).
3.
Finally, the district court denied Mitchell's plea for a three-level minor role reduction pursuant to U.S.S.G. § 3B1.2. Section 3B1.2 provides for a reduction in the offense level when the defendant "plays a part in committing the offense that makes him substantially less culpable than the average participant." Id. at § 3B1.2, cmt. n. 3. Whether a defendant played a minor role is a question of fact reviewed for clear error. United States v. Deans, 590 F.3d 907, 909 (8th Cir. 2010). The burden of proof rests with the defendant to prove that he played a minor role. Id.
Mitchell emphasizes that he cashed approximately fifteen percent of the known money orders in the conspiracy and that this represented a similar percentage of the total amount of funds transferred. He asserts that he was not involved in the underlying drug dealing, that he did not participate in the purchase of the money orders, and that he did not transport the money orders from St. Louis to Los Angeles. When denying Mitchell's request for this reduction, the district court stated that although his culpability was less than Epherson's, Mitchell could not be considered "anything less than a full participant in the conspiracy."
Epherson and Mitchell were the only principals in the money laundering conspiracy. The fact that Mitchell was less culpable than Epherson does not necessarily lead to the conclusion that Mitchell was substantially less culpable than the average participant. See United States v. Bush, 352 F.3d 1177, 1182 (8th Cir. 2003) ("[M]erely showing the defendant was less culpable than other participants is not enough to entitle the defendant to the adjustment if the defendant was `deeply involved' in the offense."). Accordingly, Mitchell has not established that the district court clearly erred in finding him a full participant in the conspiracy and denying his request for a minor-role reduction.
III.
The conviction is affirmed. The sentence is vacated, and the case is remanded to the district court for reconsideration of the U.S.S.G. § 2S1.1(b)(2)(C) enhancement in light of the amount of revenue that Mitchell in fact derived from his transactions.
08), and the parties agree that it controls here.
As set forth more fully in the Disparate Impact Opinion, see 637 F. Supp. 2d at 109, Guardians represents both a distillation and a reconciliation of the differing test-validation approaches contained in the Equal Employment Opportunity Commission's "Uniform Guidelines on Employee Selection Procedures" ("EEOC Guidelines"), 29 C.F.R. §§ 1607.1-1607.18. The EEOC Guidelines draw a sharp distinction between "tests that measure `content' — i.e., the `knowledges, skills or abilities' required by a job — and tests that purport to measure `constructs' — i.e., `inferences about mental processes or traits, such as `intelligence, aptitude, personality, commonsense, judgment, leadership and spatial ability.'" Gulino, 460 F.3d at 384 (quoting Guardians, 630 F.2d at 91-92). A content validity approach to test validation measures the degree to which the content of a selection procedure is representative of important aspects of job performance. 29 C.F.R. § 1607.5(B). A construct validity approach assesses the degree to which the test measures identifiable characteristics that are important for successful performance of the job in question. Id. § 1607.14(D). The Guardians court observed, however, that content and construct exist on a single continuum that "starts with precise capacities and extends to increasingly abstract ones." Guardians, 630 F.2d at 93. The Guardians court therefore developed a "functional approach" to content validation that permits courts and employers to use a content validity approach to assess tests that measure some abstract abilities. Id. The court cautioned, however, that "the degree to which content validation must be demonstrated should increase as the abilities tested for become more abstract."Id. Guardians lays out a five-part test for analyzing the content validity of an employment test:
(1) The test-makers must have conducted a suitable job analysis;
(2) They must have used reasonable competence in constructing the test itself;
(3) The content of the test must be related to the content of the job;
(4) The content of the test must be representative of the content of the job; and
(5) There must be a scoring system that usefully selects from among the applicants those who can better perform the job.
Id. The first two requirements relate to the "quality of the test's development," while the final three "are more in the nature of standards that the test, as produced and used, must be shown to have met."
Id. The court turns now to the question of whether the City has carried its burden of demonstrating that it complied with these requirements.
Although Plaintiffs argue that the City has failed to meet its burden under Guardians, they also argue in the alternative that this court should rule against the City without recourse to theGuardians factors. According to Plaintiffs, many of the constructs that Exam 6019 was meant to measure are internal, unobservable traits, which means that the exam cannot be properly assessed using content-validation techniques. Because the City relies exclusively on a content-validity approach to demonstrate that Exam 6019 is valid, Plaintiffs argue that it has necessarily failed to carry its burden of proof. (See generally Pl. Mem. 8-11.) While this argument has some force, the court considers it unwise to circumvent Guardians in this manner. First, Guardians and Gulino recognize that the line between observable abilities and internal traits can be hazy, and that courts should closely scrutinize an employer's actions before declaring that a particular validation method is categorically inappropriate. Second, the 18 constructs that Exam 6019 seeks to measure are not all abstract or "internal"; some, like Number Facility, are relatively concrete. In this sense, Exam 6019 exemplifies the continuum that Guardians sought to accommodate. Third, Plaintiffs' argument could just as easily apply to the question of whether the City has carried its burden at step three of theGuardians test to demonstrate that the content of Exam 6019 is related to the content of the job. See Guardians, 630 F.2d at 95. Accordingly, the court believes that the better course is to consider Plaintiffs' points about the relevance of the City's validation efforts as part of its Guardians analysis.
1. The City's Job Analysis
The EEOC Guidelines require employers who use a content validation approach to undertake a job analysis that assesses "the important work behavior(s) required for successful performance and their relative importance." See Guardians, 630 F.2d at 95 (quoting 29 C.F.R. § 1607.14C(2)). Here, the City's job analysis appears to meet that standard. Dr. Cline asked incumbent firefighters to generate a comprehensive list of 181 tasks and 46 abilities that she then used to construct a Job Analysis Questionnaire (JAQ) that she administered to over 340 firefighters. (HT 38-39; Job Analysis Report at DCAS-08652-56.) The JAQ asked respondents if they performed the listed tasks, how important the tasks were, how frequently they performed the tasks, and whether they would perform the tasks on the first day of active duty. (Id. at DCAS-08657.) It also asked the firefighters how important the listed abilities were and if they were required on the first day of active duty. (Id.) After using the JAQ results to narrow the task and abilities list, Dr. Cline convened a Linking Panel to rate the relationship between the tasks
and abilities. She then used the Linking Panel results to eliminate one of the abilities from the list. (Id. at DCAS-08660.) This process substantially complies with both the EEOC and Guardians requirements. See 630 F.2d at 95-96.
In undertaking the Exam 6019 job analysis, the City appears to have avoided the two crucial errors that it made in its job analysis for Exams 7029 and 2043. As detailed in the Disparate Impact Opinion, the City's previous job analysis (1) failed to demonstrate any relationship between abilities and tasks, and (2) retained tasks and abilities that could be learned on the job, as opposed to tasks and abilities that firefighters must perform or possess prior to appointment. See D.I. Op., 637 F. Supp. 2d at 108-113. Here, the Linking Panel, under the supervision of Dr. Cline, appears to have understood the definitions and importance of the various abilities and meaningfully analyzed the abilities' relationship to the listed tasks. (See Job Analysis Report at DCAS-08660 and Appx. B.) By seeking information about "Day One" tasks and abilities on the JAQ, the City also appears to have made reasonable efforts to eliminate tasks and abilities that could be learned on the job.
The City used the same job analysis for Exam 7029 and 2043.
The City also did a better job of selecting relevant abilities and characteristics for testing. As described in the Disparate Impact Opinion, the designers of Exams 7029 and 2043 sought to test only nine abilities, all of which were cognitive. In doing so, the City ignored a host of non-cognitive abilities that are equally, if not more important to the job. See D.I. Op., 637 F. Supp. 2d at 119-121. By contrast, Dr. Cline consulted numerous reputable sources to compile a comprehensive list of abilities, (HT 36-37), and used a reasonable review process to narrow the list to 18 non-physical constructs. In doing so, Dr. Cline retained many of the constructs that this court previously criticized the City for excluding. (See Job Analysis Report; D.I. Op., 637 F. Supp. 2d at 119.)
Plaintiffs assert that the City failed to test for an important ability, Mechanical Comprehension. (See, e.g., Pl. PFF ¶ 70.) The City is not required to test every single potentially useful ability. Guardians, 630 F.2d at 99. Plaintiffs may address the utility and feasibility of testing for Mechanical Comprehension with the Special Master when developing the future exam.
2. The Test Construction Process
The second Guardians requirement is that the test-makers must use reasonable competence in constructing the test. As described in the Disparate Impact Opinion,Guardians offers two points of guidance on this issue. First, civil service examinations should be constructed by testing professionals rather than lay employees.Guardians, 630 F.2d at 96 ("[E]mployment testing is a task of sufficient difficulty to suggest that an employer dispenses with expert assistance at his peril."). Second, exam questions should be tested on a sample population to roughly determine whether they accurately measure the targeted abilities and to ensure that they are comprehensible and unambiguous. Id.
The City retained a test-preparation specialist, Dr. Cline, to oversee the construction process. But just as it did with Exams 7029 and 2043, the City erred by permitting firefighters to write test items, in this case for the cognitive component of the exam. (See Pl. PFF ¶¶ 75-77; D.I. Op., 637 F. Supp. 2d at 115.)Guardians explicitly warns employers not to leave item-drafting to lay employees, "who may have . . . expertise in identifying tasks involved in their job but [are] amateurs in the art of test construction." 630 F.2d at 96. It is no defense to say, as the City does, that the items written by firefighters were later reviewed by an expert, since this was precisely the process condemned in Guardians. Id. at 84, 96.
The City also sample-tested Exam 6019 on firefighters during the Final SME Review. That review occurred so late in the construction process, however, that the City was unable to integrate its findings into a revised exam. Dr. Cline conducted the Final SME Review on December 26 and 27, 2006 — one day before Exam 6019 was scheduled to go to print. (Pl. Exs. 14, 62; HT 82.) As described above, the SME reviewers provided a large amount of negative feedback that, had it been actually considered by the test designers, would have required a substantial rethinking or reorganization of the exam. Dr. Cline admitted at the 6019 Hearing, however, that she largely ignored the firefighters' ratings and comments, and that at any rate it was "way too late" to alter the major components of the exam. (HT 92-95; see also fn. 14, above.)
The City does not dispute that Exam 6019 went to print on December 28, 2006. Although Dr. Cline testified at the 6019 Hearing that she made changes to exam items on the basis of the Final SME Review results, she also admitted that she did not know when the exam was printed. (HT 204.)
Dr. Cline also administered a pretest to DCAS personnel on January 12, 2007 to test the exam instructions and the length of the timed section — but not to test whether the questions were comprehensible or accurately measured the 18 constructs. (Test Dev. Rep. 9.) For Guardians purposes, this pretest suffers the triple infirmity of having been given too late, for the wrong reasons, and to the wrong people. Therefore, despite its belated efforts, the City failed to comply with Guardians' requirement to conduct appropriate sample testing.
3. Relationship Between Test Content and Job Content Guardians' third requirement that the content of the exam be directly related to the content of the job reflects "[t]he central requirement of Title VII" that a test be job-related. 630 F.2d at 97-98. The correspondence between exam and job is a function of two conditions: whether the constructs that the employer sought to test were required for successful job performance, and whether the exam actually tested those abilities. See D.I. Op., 637 F. Supp. 2d at 117. As detailed above in Section IV.B.1, the City has successfully demonstrated that the 18 constructs it sought to test were related to the job of entry-level firefighter. But the City has failed to demonstrate that Exam 6019 actually tested those 18 constructs. This failure is fatal to the City's showing underGuardians' third requirement and, ultimately, to its ability to demonstrate that Exam 6019 is valid.
i. Appropriateness of a Content-Validity Approach
The EEOC Guidelines, and test-construction professionals generally, recognize at least three distinct methods of test validation: criterion-related validation, construct validation, and content validation. (See Pl. Ex 18 ¶ 49; HT 32.) Criterion-related validation examines whether candidate performance on an exam correlates with actual job performance. (Pl. Ex 18 ¶ 51; HT 33.) This inquiry is retrospective, and requires job-performance data such as appraisals or supervisor ratings. (Pl. Ex 18 ¶ 51.) Construct validation evaluates an exam's relationship to other well-researched or validated exams, including whether the exams share common characteristics and whether performance on a given exam correlates with performance on another exam that has been shown to have criterion-related validity in a similar setting. (Pl. Ex 18 ¶¶ 58-59; HT 33.) Content validation, according to Plaintiffs' expert, "seeks to use logical, and sometimes statistical, evidence to evaluate whether the selection procedure can be viewed as `mapping onto' and calling for a candidate to display behaviors that are important to performing the job." (Pl. Ex 18 ¶ 53; see also HT 33 (testimony of Dr. Cline that content validity "involves looking at the test content and see[ing] whether it matches something that it's desirable to match").)
The City relied solely on a content-validity approach to validate Exam 6019. (Id. 34; Pl. PFF ¶ 48.) Dr. Cline conducted a job analysis, identified relevant constructs, and attempted to create items that measured those constructs — all hallmarks of a content-validation approach. (See, e.g., HT 41-43.) Plaintiffs argue that this approach was fundamentally inappropriate, and that as a result the City cannot prove that Exam 6019 is valid.
According to Plaintiffs' expert, Dr. Jones, "[c]ontent validation becomes an increasingly appropriate strategy as the opportunity to show a linkage between observable aspects of the selection procedure and observable aspects of the job increases. The ubiquitous example is content validation of a computer-based word processing test to assess candidates where word processing is a key part of the job." (Pl. Ex 18 ¶ 54.) Conversely,Guardians, the EEOC Guidelines, and Dr. Jones's testimony all make clear that content validation is an inappropriate validation strategy if the exam seeks to measure internal, unobservable mental traits or constructs. See Guardians, 630 F.2d at 93, 98; Pl. Ex 17 ¶¶ 39-40; HT 308-311. The EEOC Guidelines state that:
Dr. Cline offered a similar example at the 6019 Hearing: "For example, if a test for algebra matches the content of an algebra course. That would be content validity." (HT 32.)
A selection procedure based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity. Thus, a content strategy is not appropriate for demonstrating the validity of selection procedures which purport to measure traits or constructs, such as intelligence, aptitude, personality, commonsense, judgment, leadership, and spatial ability.
29 CFR § 1607.14C(1). This prohibition reflects the considered judgment of courts, agencies, and test-construction experts that it is extremely difficult to determine ex ante whether a multiple-choice, paper-and-pencil exam accurately measures internal mental processes or personal attributes. Thus, according to Dr. Jones, test-makers seeking to measure such abstract constructs typically use a criterion-related or construct validation approach in place of, or in conjunction with, a content-based approach. (Pl. Ex. 17 ¶ 48.)
This is not to say that an exam can never measure such traits — only that it is nearly impossible to tell if it will do so before it has been administered.
Here, it is clear that Exam 6019 seeks to measure abstract, unobservable mental constructs such as Flexibility of Closure, Speed of Closure, and Problem Sensitivity, as well as personality characteristics like Integrity, Adaptability, Tenacity, Work Standards, and Resilience. These are precisely the sort of "traits or constructs" that render an exam unfit for content-based validation. The City's exclusive reliance on a content validation approach necessarily means that it has failed to demonstrate that the content of Exam 6019 is related to the content of the job of entry-level firefighter.
The City argues that a content-validation approach is appropriate in this case because Dr. Cline sufficiently "operationalized" the traits and mental constructs she sought to measure — i.e., she defined them in terms of observable behaviors rather than unobservable internal processes. (HT 67.) This argument is undercut by the results of the item classification procedure, in which Dr. Cline asked five DCAS Test and Measurement Specialists to match each item on Exam 6019 to the ability it measured. As Dr. Cline admits, the entire purpose of the classification procedure was to determine whether the Exam 6019 constructs were sufficiently operationalized. (HT 68.) The DCAS specialists, however, were unable to correctly classify approximately 40% of the items. (Test Dev. Rep. at DCAS-15235-36.) This result should not have been surprising: as described above, the inherent difficulty of ascertaining what an exam measures before it is administered is precisely what animates the Guidelines' prohibition against using content validation to support exams that measure internal traits or unobservable mental constructs. In this case, the City's own data demonstrate the inappropriateness of relying on a content-validity approach to assess Exam 6019, above and beyond the admonitions in the EEOC Guidelines and professional literature.
ii. Factor and Reliability Analyses
After the candidates took Exam 6019, Dr. Cline conducted a "factor analysis" and a "reliability analysis" on the scoring data to determine whether the exam measured the constructs it was intended to measure. (See Exam Analysis.) According to the City, the results of these analyses demonstrate that Exam 6019 accurately measured whether candidates possessed the 18 abilities and characteristics that Dr. Cline identified. Plaintiffs' evidence demonstrates the opposite.
According to Dr. Jones, factor analysis is a means of identifying patterns in a set of data — in this case, the candidates' answers to each item on Exam 6019. (HT 323.) Factor analysis is used to determine whether exam items group together into "factors" or clusters consisting of items intended to measure the same construct. The underlying assumption is that a given candidate is likely to perform consistently on similar items; for example, if a candidate correctly answers a Spatial Reasoning question, that candidate is more likely to correctly answer other questions that measure Spatial Reasoning. The degree to which candidates' answers aggregate into identifiable clusters, and the number of such clusters, can be compared to the employer's exam plan to determine whether the exam actually measured what it was supposed to measure. (Id.; see also D.I. Op., 637 F. Supp. 2d at 116.) For example, had the City designed Exam 6019 perfectly, a factor analysis of the 195 items on the exam would reveal 18 clusters, each defined by the items written to assess one of the 18 constructs. (See, e.g., Pl. Ex. 17 ¶¶ 140-41.)
The City's submissions do not explain the purpose or methodology of factor analysis.
Although Dr. Cline conducted a factor analysis, she did not factor analyze the individual items on Exam 6019. Instead, she sorted the 195 items into 18 subgroups, based on which construct each item was intended to measure, and added up candidates' scores on each subgroup of items to create 18 "subscores." Dr. Cline then factor analyzed those subscores. (Id. ¶ 135; HT 99-100, 330.) As Plaintiffs point out, by factor analyzing the subscores rather than the individual items, Dr. Cline's factor analysis assumed, rather than proved, that the sets of items written to measure each ability correlated with one another more strongly than they correlated with other sets of items that were written to assess other constructs. (Pl. PFF ¶ 124 (emphasis in original).) In other words, Dr. Cline's factor analysis did not do what the City needed it to do: reveal approximately 18 clusters, each composed primarily of items written to test identical constructs. What Dr. Cline's subscore factor analysis did do, according to her testimony, is reveal a strong correlation among items within each of the three broad components of the exam — cognitive, SJE, and timed. (HT 102-04.) This result is irrelevant. The question is whether Exam 6019 accurately measured the 18 constructs that it was designed to measure, not the three external categories that those constructs were grouped into.
Dr. Jones, by contrast, conducted an "item-level" factor analysis of Exam 6019. (Pl. Ex. 17 ¶ 138.) His factor analysis found that items that were intended to measure completely different constructs clustered together, and, conversely, that items that were intended to measure identical constructs were scattered across clusters rather than bunched together. (See id. ¶¶ 145-46, 220-21.) Dr. Cline disputes the accuracy of his findings, but even if her contention were correct, it would not alter the fact that her factor analysis does not show that Exam 6019 did what it was supposed to do. The burden to make this showing lies with the City.
Dr. Cline also conducted a reliability analysis on the Exam 6019 scoring data. (See Exam Analysis.) According to Dr. Jones' testimony, reliability analysis, also called internal consistency analysis, is in some senses a mirror version of factor analysis. Instead of looking for patterns in the scoring data as a whole, reliability analysis pulls out sets of items that were meant to correspond to one another according to the test plan — in this case, the 18 subgroups of items that corresponded to each intended construct. (HT 333.) The reviewer then assesses the degree to which the test-takers performed consistently on the items in each subgroup. (Id.)
Testing professionals use a statistic called "coefficient alpha" to calculate how reliable each subgroup is. (Pl. Ex. 17 ¶ 158.) According to Dr. Cline, coefficient alpha measures how much error there is in a test score. (HT 111.) A coefficient alpha of .60 indicates that 40% of the score is due to error variance. (Id.) A coefficient alpha of .70 or greater is typically accepted as indicating a reliable measure of a given construct. (Pl. Ex. 17 ¶¶ 158-59.) When Dr. Cline calculated the internal consistency of the 18 subgroups of Exam 6019 items, she obtained the following results:
Cognitive Ability the Items Coefficient Were Intended to Alpha Measure No. of Test Items Reliability Deductive Reasoning 4 .36 Flexibility of Closure 5 .43 Information Ordering 5 .57 Inductive Reasoning 5 .45 Number Facility 5 .50 Problem Sensitivity 6 .42 Spatial Orientation 5 .48 Visualization 5 .48 Speed of Closure 5 .70 Perceptual Speed 40 .95 Memory 5 .26 Adaptability 15 .44 Coordination 15 .54 Establishing and Main- taining Interpersonal Relationships 15 .40 (Exam Analysis at DCAS-19483-19484.) Only two subgroups of items — those intended to measure Speed of Closure and Perceptual Speed — had coefficient alphas of .70 or greater. Twelve of the 18 subgroups had coefficient alphas of .50 or lower, indicating that 50% or more of the score was due to error. The remaining four subgroups had coefficient alphas between .53 and .57, indicating that 43% to 47% of the score was due to error. According to Dr. Jones, these reliabilities are no better than those produced when he randomly selected items to be placed in each subgroup using a random number generator. (Id. ¶¶ 168-71.)
Dr. Cline asserted at the 6019 Hearing that these calculations do not indicate that Exam 6019 failed to test for the intended constructs, because "when the [subscores] are correlated all together and added into a single score, the overall score is going to be reliable." (HT 112; see also Cline Expert Rep. 9 ("The reliability of the total score was .94, which . . . is generally acceptable.").) Dr. Cline did not explain or defend this claim, which is problematic for the City, given that it bears the burden of demonstrating job-relatedness. What is clear is that Dr. Cline admitted on the stand that her reliability analysis does not say "anything" about whether Exam 6019 measures 18 specific constructs. (HT 113.) In this phase of the Title VII inquiry, that concession is dispositive of the issue.
In sum, the City has not established that Exam 6019 did what it was designed to do: determine whether candidates possessed the 18 abilities and characteristics that the City deemed necessary to the job of entry-level firefighter. As confirmed by the classification procedure, the City's reliance on a content-based approach to validate an exam that tested internal, unobservable constructs was fundamentally misplaced. To the extent that content validation was appropriate for the portions of the exam that tested observable constructs, the City's statistical analyses undermine any contention that the exam actually measured those constructs, and Dr. Jones's statistical analyses refute it.
4. Representativeness of the Test Content
The fourth Guardians requirement is that a test be a "representative sample of the content of the job." 630 F.2d at 98 (internal quotation marks omitted). This requirement has two components: "[t]he first is that the content of the test must be representative of the content of the job; the second is that the procedure, or methodology, of the test must be similar to the procedures required by the job itself." Id. These requirements are not strict, since rigorous interpretation of either component would "foreclose any possibility of constructing a valid test."Id.
With respect to content representativeness, "it is reasonable to insist that the test measure important aspects of the job, at least those for which appropriate measurement is feasible, but not that it measure all aspects, regardless of significance, in their exact proportions." Id. at 99. The reason for this requirement is to "prevent either the use of some minor aspect of the job as the basis for the selection procedure or the needless elimination of some significant part of the job's requirements from the selection process entirely; this adds a quantitative element to the qualitative requirement that the content of test be related to the content of the job." Id.
Plaintiffs argue, and this court agrees, that the City's complete failure to carry its burden at step three of theGuardians analysis necessarily means that it has not demonstrated content representativeness at step four. As detailed above, the City cannot establish that Exam 6019 accurately measures any of the constructs that it was intended to measure. A fortiori, the City cannot establish that the exam measures any "important aspects of the job." This does not mean that the fourth Guardians requirement is redundant: had the City demonstrated that Exam 6019 tested some or all of the intended constructs, this court would be required to undertake the additional "quantitative" inquiry into whether "some minor aspect of the job" was used as a basis for selection or "some significant part of the job's requirements" was eliminated from the exam. See id. But where the employer has failed to show any significant correlation between exam content and job content, the content-representativeness component of step four is extraneous.
The requirement that a test's procedures be representative is intended "to prevent distorting effects that go beyond the inherent distortions present in any measuring instrument." Id. In particular, "although all pencil and paper tests are dependent on reading, even if many aspects of the job are not, the reading level of the test should not be pointlessly high." Id. Dr. Cline analyzed the reading level of Exam 6019 after this court's July 2009 liability ruling, which criticized the unnecessarily high reading level of Exams 7029 and 2043. (See Readability Rep. (Def. Ex. A-7); D.I. Op., 637 F. Supp. 2d at 122-23). Dr. Cline compared Exam 6019 with the Probationary Firefighter's Manual that the FDNY uses at its Fire Academy. (Readability Rep.) According to Dr. Cline, the median reading grade level for Exam 6019 is 7.7, while the median reading level of the Manual is 10.0. (Id.) This analysis sufficiently demonstrates that the reading level of Exam 6019 is not too high. See D.I. Op., 637 F. Supp. 2d at 123 (reading level of written test questions "should be below the reading level of material Firefighters may be given to read at the Fire Academy").
5. Utility of the Scoring and Selection System
Under the fifth Guardians requirement, the City must demonstrate that its scoring system usefully selects those candidates who can better perform the job of entry-level firefighter. Guardians, 630 F.2d at 100. The court notes at the outset that it is approaching this analysis in a different posture than the Guardians court. In Guardians, the Second Circuit held that the defendant had submitted adequate evidence of content-relatedness and representativeness under requirements three and four — in other words, that the challenged exam accurately assessed important job constructs. 630 F.2d at 99. TheGuardians court therefore went on to consider whether the defendant's use of a cutoff score and a rank-order mechanism was justified by business necessity. Id. at 100, 105. Here, by contrast, the City has failed to make its showing under requirements three and four. The City has not established that Exam 6019 reliably measures any of the abilities or characteristics required by entry-level firefighters. This fact obviates the need to inquire into the City's scoring system. If candidates' exam scores provide no useful information about their qualifications, then by definition there is no "valid" cutoff score that will usefully differentiate between qualified and unqualified candidates, and no "valid" justification for ranking candidates based on their exam scores. Step five of Guardians is only appropriate for determining whether an otherwise valid exam was used improperly. See Barbara Lindemann Paul Grossman, Employment Discrimination Law 158 (3d ed. 1996). That situation is not present here.
Even if the City could demonstrate that Exam 6019 has some level of validity, it could not establish that its use of the exam as a pass/fail and rank-ordering device is job-related. With respect to pass/fail use, the City admits that it used a cutoff score of 70 on Exam 6019 because 70 is the default cutoff score provided by the City's civil service rules. (Pl. PFF ¶ 12.) As this court held in the Disparate Impact Opinion, the City may not simply rely on civil service rules to set a passing score. Instead, the City must present evidence that its chosen cutoff score bears a direct relationship to the necessary qualifications for the job of entry-level firefighter. D.I. Op., 637 F. Supp. 2d at 125. The City has not attempted to make such a showing.
The City, to its credit, acknowledges that its chosen cutoff score is improper. (See Def. Mem. 12.) The City suggests, instead, that it would be appropriate to use an entirely different cutoff score to distinguish qualified from unqualified candidates. At the 6019 Hearing, Dr. Cline testified that she reanalyzed the Exam 6019 data using something called a "Modified Angoff Method," and that the results of this reanalysis show that a cutoff score of 80, rather than 70, is appropriate — i.e., job-related. (HT 55-57; see also Angoff Report.) The City now contends that using Exam 6019 on a pass/fail basis with a cutoff score of 80 is a valid selection procedure under Title VII.
The court is perplexed by the City's approach. The City is essentially asking the court to ratify a hypothetical use of Exam 6019, rather than the City's actual use of the exam. But the City could have done any number of things with the exam results. It could have banded candidates' scores, or combined them with candidates' CPAT scores, or decided that there was a strong basis in evidence that Exam 6019 violates Title VII and then discarded the exam scores altogether. The City took none of these actions, so the court does not need to consider them. And for the same reason, the court does not need to consider what would have happened if the City had set the cutoff score at 80. The only question before the court is whether what the City actually did — use Exam 6019 as a pass/fail device with a cutoff score of 70 — was justified by business necessity. If, as the City admits, that action was not justified — if the cutoff score that it used did not differentiate between qualified and unqualified candidates — then the inquiry is over. The candidates who were harmed by the City's invalid cutoff score have already been harmed, and the candidates who benefited from the invalid cutoff have already benefited. The only question now is what the court should do to fix this problem.
Viewed in proper perspective, the City's proposal to use Exam 6019 with a different cutoff score is actually a proposed remedy for its admittedly invalid selection procedure. The practical effect of the City's proposal is that 2,100 candidates who scored between 70 and 80 on Exam 6019 would be retroactively stricken from the eligibility list. (See Def. Mem. 13.) As the City admits, implementing the new cutoff score would mean that "about 15% of the candidate pool would fail, with the result being a greater impact on minorities." Id. The benefit, according to the City, is that the new pass/fail system would be job-related. But for the reasons outlined above, the City cannot demonstrate that any cutoff score, however derived, would be valid. The problem lies with the exam, not the cutoff. The court therefore declines to implement the City's proposed remedy.
With respect to rank-ordering, Dr. Cline asserts that her Angoff analysis demonstrates that "[d]ifferences in test scores . . . differentiate different types of candidates and provide support under the [EEOC] Guidelines for the use of Examination 6019 in a rank-order fashion." (Angoff Report at DCAS 19537.) Mere differentiation, however, is insufficient to support the rank-ordering of an eligibility list containing 21,000 candidates, the vast majority of whom are bunched tightly at the top. "Permissible use of rank-ordering requires a demonstration of such substantial test validity that it is reasonable to expect one- or two-point differences in scores to reflect differences in job performance. . . . Without some substantial demonstration of reliability it is wholly unwarranted to make hiring decisions, with a disparate racial impact, for thousands of applicants that turn on one-point distinctions among their passing grades."Guardians, 630 F.2d at 100-101. Dr. Cline admitted in a deposition, however, that her Angoff analysis provides no information about the expected difference in job performance associated with even a three-point difference in scores on Exam 6019. (Pl. PFF ¶ 268.) Nor could it, given that the City has failed to demonstrate that Exam 6019 is content valid.
According to Plaintiffs' analysis, of the top 10,000 candidates on the eligibility list, only five scored below 80 on the exam. (Pl. Ex. 4a ¶ 15.) The 2,039th ranked candidate on the eligibility list has an Adjusted Final Average score of 98.727. (Id. ¶ 18.) The 886th ranked candidate has an Adjusted Final Average score of 100.593. (Id. ¶ 16.) In other words, fewer than two points separate the 1,153 highest-ranked remaining candidates.
V. CONCLUSION
The parties do not dispute that the City's use of Exam 6019 to select entry-level firefighters disproportionately impacts black and Hispanic firefighters. For the reasons stated above, the City has not met four of the five requirements that it must meet to demonstrate that Exam 6019 is content-valid under Guardians Assoc. of New York City Police Dept., Inc. v. Civil Service Comm'n., 630 F.2d 79, 100 (2d Cir. 1980). The City has therefore failed to carry its burden to demonstrate that its use of Exam 6019 as a pass/fail and rank-ordering device is job-related and justified by business necessity. The court therefore concludes that the City's use of Exam 6019 does not comply with Title VII.
Accordingly, the court restrains and enjoins the City from taking any further steps to initiate or finalize a fire academy class using the Exam 6019 eligibility list until October 1, 2010. The court will hold a hearing at the earliest possible date to consider the remedial measures it should take in light of this order.
SO ORDERED.