Free Reply to Response to Motion - District Court of Federal Claims - federal

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 1 of 32

Department the Army of Headquarters,United States Army Training and Doctrine Command Fort Monroe, Virginia 23651-1047 12 January2004

TRADOC Pamphlet 350-70-4

Training SYSTEMSAPPROACH TRAINING: EVALUATION TO Summary This pamphletprovides implementingguidance,formats, and techniques for evaluation andquality assurance programs the U.S. Army of Training and Doctrine Command (TRADOC) described in TRADOC as Regulation 350-70, Part III.

activities and TheArmySchool Applicability This pamphletapplies to TRADOC System (TASS) Training Battalions responsible for managing pedorming Training Development or TD-relatedfunctions, including (TD) evaluation/qualityassurance the training, products,andinstitutions that of present the training. It also applies to non-TRADOC agencies/ organizations having Memorandums Understanding, Memorandums of of Agreement, contracts for developing and training or training productsfor TRADOC TASS and agencies and organizations. Forms The"R" formsat the backof this pamphlet for local reproduction. are

Suggested Theproponent this pamphlet the Deputy for is Chief of Staff for ImproveOperations and Training (DCSOPS&T). Send comments and suggested improvements on DA Form 2028 (Recommended Changesto ments Publications and Blank Forms)through channelsto Commander, TRADOC (ATTG-CD),5 Fenwick Road, For~ Monroe, VA 23651-1049. Suggestedimprovements mayalso be submitted using DAForm1045 (ArmyIdeas for ExcellenceProgram (AIEP) Proposal). homepage at Availablity This publication is distributed solely throughthe TRADOC http://www.tradoc.army.mil. Contents Paragraph Page Introduction Purpose References Explanations abbreviations andterms of Systems Approach Training (SAT) overview to Regulation,pamphlet, job aid (JA) relationships and

1-;2 4 1-__~_3 4 1-___~4 4 1-__#_5 5

1422

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 2 of 32

TRADOC Pam 350-70-4 Contents (cont) Paragraph Page Overview quality assurance, of quality control, andevaluation1-6 1-7 Typesevaluations of ................................................................ Overview training development of management/planning ....... 1-8 Electronic .................................................................... 1-9 support Quality control criteria .............................................................. 1-10 Ch_.~ter 2 Evaluation Process 2-1 Evaluation ................................................................. overview Evaluator's........................................................................ 2-2 role 2-3 Evaluation description process ................................................ PlanningEvaluations Planning ................................................................... overview Planning anevaluation ............................................................ Evaluation project management ..................................... plans Quality control cdteria planning for .......................................... Collecting EvaluationData Data collection overview .......................................................... Data collection procedures ...................................................... Quality control criteriafor data collection ................................. AnalyzingEvaluationData Evaluation analysis overview .................................................. Analysis description ................................................................. Quality control criteria for evaluation analysis ......................... Preparing EvaluationReports Evaluation preparation report overv=ew ................................... Evaluation descriptions report ................................................. Report preparation ......................................................... steps Prepare report draft ................................................................. Staffdraftreportfor review/concurrence ................................. Obtain final approval recommendations of .............................. Distribute report/recommendations action, andconduct for follow-up ...................................................................... check Qualitycontrolcriteria for preparing evaluation reports........... 6 10 11 13 13

13 14 21

3-1 3-2 3-3 3-4

23 23 28 30

4-1 4-2 4-3

31 31 39

5-1 5-2 5-3

39 39 43

6-1 6-2 6-3 6-4 6-5 6-6 6-7 6-8

43 44 44 44 46 47 47 48

1423

2

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 3 of 32

TRADOC Pam 350-70-4 Contents (cont) Paragraph Page Conducting Evaluation Follow-Ups Evaluation follow-up overview ................................................. 7-__~1 48 Follow-up description ............................................................... 7-__~2 9 4 Quality control criteria for conducting follow-upevaluations....7-~3 51 Chapter 8 Internal Evaluation Internal evaluation overview .................................................... Internal evaluation description .................................................. Internal evaluation procedures ................................................ Areas consider/review to duringinternal evaluations ............... Outputsinternal of evaluations ................................................ Internal evaluation issues/concerns ......................................... Quality control criteria for internal evaluations ......................... Ch._~ter 9 External Evaluation External evaluation overview ................................................... External evaluation description ................................................ External evaluation procedures ............................................... Unit training evaluation ............................................................ AUTOGEN program software .................................................. Outputs external of evaluations ............................................... External evaluation issues/concerns ....................................... Quality control criteria for external evaluations ........................ Chapter 10 Accreditation Accreditation ............................................................. overview Accreditation description ......................................................... Self-assessment description .................................................... Quality control criteriafor accreditation ................................... Appendixes A__~. References ......................................................................... B_~. Process SAT ...................................................................... C._~. Aid Job Hypeflinks ............................................................. Glossary_ .................................................................................

8-'1 51 8-2. 52 54 8-__~3 8-.._~4 6 5 8-__~59 5 8-.__~6 60 8-__~71 6

9-'1 9-2 9-3 9-4 9-5 9-6 9-7 9-8

61 62 63 63 64 66 66 67

10-~167 10-2. 68 10-~370 10-~4 71 72 73 77 78

1424
3

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 4 of 32

TRADOC Pam 350-70-4

1-1.

Chapter1 Introduction Purpose. This pamphletprovides detailed guidancein support of TRADOC 350-70on the following areas of the evaluation process Req for TRADOC courses and courseware: o Evaluation process. ,, Internal evaluations. Externalevaluations. Accreditation. References. referencesfor this pamphletappearin appendixA. The Explanations abbreviationsandterms. Abbreviations and terms of appear the glossaryof this publication. in

1-2.

1-4,

Systems Approach Training (SAT) overview. to a. It is important to havean understanding the Army'speacetime of mission, and the System's Approach Training (SAT), before to addressing specifics of this pamphlet. appendix for a detailed the See B explanationof the SAT process, andeachof the five phases. b. TheArmy'speacetime missionis to preparethe Army,i.e., Active Component (AC), ReserveComponent (RC), and National Guard Components, perform(fight, win, andsurvive) acrossthe entire to spectrum military operations.Soldiers andunits mustbe trained to of performtheir missionto standard,andsurvive. c. Education/trainingdevelopment is a vital component the (TD) of mission to prepare the Armyfor war. TheArmy'sTDprocessis subsumed the SATprocess. Thegoal of the SATprocessis to in support the Army'smissionby providing mission-focused, task-based, hard- andsoft-skill education/training. This training mustbe rigorous; relevantto units, soldiers, andleaderstrained; andconducive safety to andenvironmental protection. d. TheSAT a systematic,spiral approach making is to collective, individual, andself-development education/trainingdecisionsfor the Army.It is usedto determine whether not training is needed; or whatis trained; whogets the training; how,how well, andwhere training is the presented;andthe training support/resources required to produce, distribute, implement, evaluatethe requirededucation/training and products.

1425

4

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 5 of 32

TRADOC Pam 350-70-4 1-5. Regulation, pamphlet, and job aid (JA) relationships. a. This pamphlet supports and provides procedural guidance for the policy established in TRADQC Req 350-70. The regulation directs the use of this pamphlet in the planning and conduct of evaluations. Job aids, product templates, product samples, information papers, and other supporting documents/products support this pamphlet. Print the pamphlet and JAs as individual files, or as a single document. Supporting JAs b. Figure 1-1 provides a list of JAs referenced in this pamphlet. Appendix C provides a consolidated list of JAs with hyperlinks. Figure 1-2 depicts the relationship of this pamphlet and supporting documents/products with TRADOC Reg 350-70.

Number

Title Format an Evalua~n ut PrOjectMarragem~qt Plan Guidance Davelopin.3 Project M~a~m~nt for a Plan '~aster E~luahon Plah F0~at Devel~ng ~es~o~a~s and Intewi~w Guides rnte~ewGuide Guldel=n~s D~ermln¢~g for Sat~pl~ng size Summa~ng Oualita~ve Data (Wr~enComme~) Execut~ Summa~ =ormat D~il~ E~lua[bn R~pohForma~ Format ~r Memorandum WD~cisicnma~r Format F011o~Up for Evalua~ds Chec~i~ SAT Evalua@r's Chec~ist:TrainingIns~[u~ndFac=lities Evalua~r's ChecklisEPmduGs Evalua~r'sC~eCkliSE TO Mana~rial Internal E~lu~lonQuestions Mana~dal ~ornal Evalua~on ~ons ~lobtication of k~redi~on Format ~ Propone~ Accredta~onT~am Recomm~datlon [o their Commandant NCOA ~CTASS of or BN Accred~at~on Sample Final ~cmdi~tionReport to NCOA a~l~ RC TASS by Pr~0P~t BN Guidelinesfor Pt~p~;ng Co~uc~ng SelfAssessment and a Cov~ ~ [or Self-Assessment Rep0n Figure 1-1. List of JAs

JA350.70~ 10c JA350-70~.10~ JA350-70-4,10e

1426
5

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 6 of 32

TRADOC Pam 350-70-4

Job-Aids

1427

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 7 of 32

TRADOC Pam 350-70-4 Overview quality assurance, of quality control, andevaluation. Quality training andtraining productsresult in soldiers whocanperform andsurvive in the full spectrum operations.This requires timely of training andtraining productsthat conform establishedstandards to and meet identified requirements, i.e., are efficient andeffective. TheSAT is the management control systemthe Armyuses to producequality education/training, andtraining productsthat meet needs the the of Army. Introduction a. Quality assurance (QA), quality control (QC), and evaluation employed throughout the SATprocess to ensurequality. These functions are not synonymous. has a distinct purposewithin Each training to achieve ultimate goalof quality. A brief descriptionof the quality, QA,QC,andevaluationfollows. (1) Quality is the timeliness, accuracy,andconformance specified standardsfor products, processes,and/or programs. Quality is engineered a process; is not an attribute that is added into it later. It is the goal at all stagesof development, is achieved and through continuousevaluation during eachphaseof the SAT process. Built-in checksin eachphaseensurethe quality of the SATprocessand instructional products,with emphasis the units' or graduates' on performance. (2) Quality assurance the function involving evaluative is processes that assurethe command training is efficient and that effective, andmeets current, Stryker, andfuture training needs the of the operationalforce. Theprimeaimof QAis to furnish the chain of command the confidence that the TRADOC with mission is being achieved, while minimizing risk of error or failure. It provides an oversightfunction for increasing organizational effectiveness,efficiency, andeconomy. objective of QAis to: The (a) Provide the Armywith the maximum return on investment. (b) Ensure maintainquality up-to-dateproducts fulfill and to the needs the operational Army. of (c) Ensure quality training andtraining productsare delivered in a timely manner,andcomplywith Department the Army(DA) and of TRADOC policy. Quality Assurance achievedthroughdecisions basedon results of is accreditation, internal andexternal evaluation, andQC functions.

Quality

Quality assurance

1428
7

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 8 of 32

TRADOC Pam 350-70-4 Quality control (3) Quality control is an evaluativeaction or event, conducted effect QA,that ensuresall education/TD implementation and procedures andprocesses,and/or education/training products, met or exceeded prescribed standards. EveryQCactivity provides a degreeof QA.The SATprocess provides a series of QCmechanisms/checks are that appliedto the development all education/trainingproducts, of procedures,and processes.Thesechecksare formal or informal. Note: The SAT"How-To"pamphlets provide the QC requirements/checks.Thegeneral QCprocessis discussedin more detail in cha~ter 2. Evaluation (4) Evaluationis a systematic, continuous processto appraisethe quality (efficiency, deficiency,andeffectiveness)of a program, process, or product. It may determinethe worth of a training program;determine if objectives havebeen met; and/or appraisethe value of a newtraining technique. It is the means which an evaluator provides management by (i.e., decisionmakers) with information/recommendations can so it decideon actions to improve education/training. It also provides the information/recommendations provethe value/worth of the education to training (summative evaluation). Evaluations: (a) Identify both intended andunintendedoutcomes decisionmakers make can necessary adjustmentsin the instructional program. (b) Providefeedback usedto modifythe education/training program,as necessary. b. Figure 1-3 showshowthis pamphletis organized. Some chapters Pamphlet by providedin other chapters; refer to eachof organization are supported guidance these to accomplish evaluation. TheproceduralJAs, product the templates,productsamples, informationpaperswill help in and accomplishing work. the Evaluation/ QC relationship c. Evaluations are the feedbackmechanism within the SATprocess to completethe QC.TheQCprocessbegins with the prescribed minimum quality standardsfor the relevant productor process. Within the QC process,the function of the evaluationelement to is gather/collect information to analyze, andthen provide management with data on whichto make quality judgments. Figure1-4 illustrates the role evaluationplays within the QC process.

1429
8

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 9 of 32

TRADOC Pare 350-70-4 (1) Evaluation a keycomponent the QC is of process, is not but synonymous it. with (2) Evaluation a tool that not onlymeasures, also is but contributes the success Army to of education/training.

Figure 1-3. Pamphlet organization

1430

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 10 of 32

TRADOC Pam 350-70-4

1-7,

Figure 1-4. Quality control process Typesof evaluations. Evaluationsare categorizedinto two types: internal andexternal. a. Internal evaluation gathers internal feedbackand management data from the education/traininginstructional system environment to determine if-(1) TheSAT processis being appropriately applied in the development products or programs. of (2) Theinstructional baseis providingthe appropriate/intended training. (3) Theobjectives of the training havebeenmet. (4) Theinstructional system producing requiredqualified is the graduates. (5) Thestaff andfaculty receivethe requiredtraining. (6) Instructorsprovidequality instruction. (7) Required infrastructure is in placeto supporttraining, whether residentor Distributed Learning (DL).

Internal evaluation

1431
10

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 11 of 32

TRADOC Pam 350-70-4 b. This processis conducted matter where howthe training no or takesplace(i.e., institution, TASS TrainingBattalion, DL,unit, or at home).Thepurpose an internal evaluationis to improve quality of the andeffectivenessof the instructional system.Internal evaluationsare discusseddetailin ch__hg_pter in 8_. External evaluation c. Externalevaluationdetermines soldiers canmeetjob if performance requirements, require all the instruction received, andneed anyadditional instruction not received.Ultimately, this evaluation processdetermines the training meets needsof the Army.This if the processgathers data from the field to assessa graduate'sper[ormance in a job environment, i.e., to determine the graduate trained to if was meetreal-world job performance requirements.External evaluations are discussed detailin ch_.~._pter in 9..

1432

11

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 12 of 32

TRADOC Pam 350-70-4

Overviewof training development managemenUplanning. Through planning,a manager develop realistic estimateof the will a resources required to accomplish evaluationprojects, establish milestones, allocate the availableresources the project. Initial and to TDmanagement planning begins with an education/TDrequirement that is a result of a needs analysis, or a new/revised training strategy. Typesof planning a. Planningservesas a top-level toot to manage education/training requirementsand resources. Proponentschools develop two types of plans: (1) A Training Development Project Management Plan (TDPMP) ~s developed every education/TD for processor product(see table 1-1 for procedure).This plan is simpleandinformal, or detailed and complex.TheTDPMP tell the "who", "what", "when","where", "why", will and"how",andultimately estimatethe cost of everyproject. (a) The TDPMP documentsworkload and resource requirements the duration of a specific project. Examples for include-= Conduct job analysis of the 84B30. a = Design/developdrill. a (b) The TDPMP identifies manpower resourcing and requirements the specific project to include suchthings asfor Personnel. o Milestones. ¯ Costs. o Material. o Temporary duty (TDY). o Anyother factor requiredfor the TDproject. Table1-1 Project Management Plan procedures StepI Action 1. Identify project teammembers, consult as needed plan and for requirements. 2. Identify TDresourcesrequired to complete the development project. Establish TDmilestones starting andcompleting project, for the and intermediate milestones, as needed. Determine budgetrequirements provide to the budgetoffice. and Include such things as--

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 13 of 32

TRADOC Pam 350-70-4 TDY requirements,to include TDY location, length, transportation requirements,personnel,andcost estimates. Material requirements. Distribution requirements costs. and Printing/reproduction costs. Determine audiovisual support requirements(personneland equipment). Identify training productrequirements. Coordinate between directorates and departments all within the proponent school, andappropriateexternal units. Complete the TDPMP. Obtain appropriate command approval of the TDPMP. Update Proponent Plan, Individual Training Plans (ITPs), the TD System TrainingPlan, etc., as appropriate the project. to

10.

(2) Proponent Plan is a proponent's TD internal, living document that includes all requirements (resourcedandunresourced). a new If TDrequirement identified or an existing oneis changed, is adjust the TD Plan. TheTDPlanis a roliup of the requirements outlined in the TDPMP. is a long-range document It covering multiple years, and provides data to various resource, budget, andmanpower reports, such as the MonthlyStatus Report, Installation Contract, Program Objective Memorandum, Command and Operating Budget. (a) The TD Plan documents workload and resource TD requirementsfor the programming, planning, budgeting,and execution years. Thedetails within the TDPlanwill increaseeachyear. (b) TheTDPlan addresses required collective and all individual training products (e.g., resident andnonresident courses, training supportpackages (TSP),manuals, etc.) as well as all processes (e.g., missionanalysis, evaluation, etc.). Table1-2 provides the procedures. Table 1-2 Training Development Plan procedures Step Action Summarize TDPMP all requirements prioritized order. in Determinewhich requirements can be accomplishedbasedon available resources (i.e., resourced requirements). Identify the impactof eachof unmet training requirement unit on mission and task accomplishment. Finalize the TDPlan. Obtain command approval of the TDPlan.

1434

13

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 14 of 32

TRADOC Pam 350-70-4 b. For further information, seeTRADOC 350-70,chapterstl-._.~2 Reg and 11-_~3.
1-9,

Electronicsupport.Training developersshould take advantage of TRADOC-provided automationtools available to support the applicable TDprocess. Theautomationtools and programs available changeover time, both in what they can accomplish,andeasein use. Keep cognizant of these changes, and bring the needfor upgraded automationrequirements the attention of the chain of command. to A goodsourceof current information is the ArmyTraininq SupportCenter website. Throughout this pamphlet there are referencesto the current automation tools, and guidance howa tool wilt help performa on specific process/procedure. current, primarytraining automated The development tool is the Automated Systems Approach Training to (ASAT).EachTraining/TD (task) proponentshould have an ASAT point of contact(POC) the TDorganizationthat canassist with obtaining in and using the program.Another resourceis the ASAT Help Desk. Qualitycontrolcriteria. Eachchapterin this pamphlet will include QC criteria for applying evaluation the process. Chapter 2 Evaluation Process

1-10. Quality control

2-1,

Evaluation overview. Theevaluation processis usedto appraise quality (efficiency andeffectiveness)of education/training,andTD. Thereis a process follow to conduct evaluationof training to an programs, products, and/or process, whetherevaluating-¯ Theeffectivenessof a training program, the meeting training or of objectives; appraisingthe effectivenessof new training; ¯ Equipment environmental or factors; or Extentthat training andtraining productsmeet needs the the of Army (i.e., determine training receivedis whatis requiredin if thefield). Theevaluation processinvolves the following phases:planning, collecting, analyzing,reporting, andfollow-up.

1435
14

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 15 of 32

TRADOC Pam 350-70-4 Evaluator's role. Theprevailing role of an education/trainingevaluator is to identify, articulate, andprovideinformationto decisionmakers to assist themin making education/training decisions. These could include decisions on whetherto continue training, whetherand howto improve the training, and/orthe cost effectiveness the training. Evaluation of is the processusedto completeQCchecksthroughout the entire SAT process. As an evaiuator, look for both strengths andweaknesses in the entire instructional system.Focuson,, How well the graduatesare meetingjob performance requirements. o Whether instruction is being providedthat is not needed. ° Whether instruction that is needed not being provided. is o Ways improvethe learner's performance the job, as well as to on the instructional system. ¯ How well eachinstructional system component contributing to is overall instructional system quality. This includes,but is not limited to, lessonplans, instructors, equipment, training devices, interactive courseware, training schedules, audiovisualmedia, facilities, manpower, costs. and = Whether SATprocessis being appropriately applied. the o How efficiently the education/training products meet identified the needs. Supportto decisionmakers a. Evaluatorssupportdecisionmakers all levels by-at (1) Collecting, analyzing,evaluating,anddistributing feedback concerningsuch areas as: (a) Quality of current training andtraining support. (b) Sufficiencyof doctrine. (c) Operability andmaintainability of equipment weapon and systems from field user's viewpoint. (d) Readiness meetnewtraining requirements. to (2) Providing standardsand guidance evaluating and for accreditingArmy training, training products,andinstitutions. (3) Identifying performance deficiencies. (4) Providingsuccessful initiatives fromthe collection andanalysis of trendsdata.

1436
15

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 16 of 32

TRADOC Pam 350-70-4 (5) Ensuring quality of training by determining the: if (a) Instruction follows objectives andimplementation procedures listed in the lesson plan, coursemanagement student plan, evaluationplan, andthe approved program instruction (POI). of (b) Training aids usedin classrooms supportthe objectives, are appropriate, understandable, readable. and (c) Environmental conditionscontribute to a properlearning environment. (d) Instructor performance meets instructional standards. (e) Training development training management and are effectiveandefficient. (f) Collectivetraining products effective andefficient. are (6) Ensuring staff andfaculty havereceivedrequiredtraining. Evaluation team makeup b. Anevaluator is an independent observerthat provides guidance andassistance while ensuringa quality processis applied, andquality productsare produced.Conducting evaluationis a teameffort, an guidedby the training developer, will function as the project leader. who Executing evaluationas a teameffort is the mosteffective wayto an accomplish this process. Building the teammay involve a matrix management approach.Theteamwill consist of a project leader and subject matterexperts(SMEs) (e.g., training developers, soldiers units, instructors, task performers, their supervisors, and etc.). Depending the purposeof the evaluation, the other teammembers on will vary (i.e., ad hocteams built on the requirements whatis are of being evaluated). Thereare several major players that performon every team. (1) Evaluators the evaluationteam on primarily consist of training developer expert in evaluation), andexpertsin other areas, (an depending the evaluationinitiative. upon (a) Thetraining developer,a GS-1750 (Instructional Systems Specialist), is normally charge the project. Thisis the individual in of trained in the conduct evaluations.This person the "training of is development" SME.

1437

16

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 17 of 32

TRADOC Pam 350-70-4 (b) Ensurethe "content/technical" SMEs masterexperts are in the military occupational specialty (MOS), area of concentration (AOC),or other areasbeing evaluated. (2) A difficulty encountered setting up this teamis selecting when master SMEs. There are three levels of SMEs--apprentice, journeyman,and master. Makesure mastertraining developersand mastercontent area SMEs on this team. are Team member roles c. Everyone taking part in an evaluationis part of the evaluation team. TRADOC 350-70, paraqraph Re,q 11-3-4, provides the basic policy on TDteams.A variety of peopleare needed during the evaluation process,but the number mix of personnel and will vary basedon the evaluation. (1) Theevaluationteamshall ensurethe evaluation: (a) Incorporatesall elements that make a SAT. up (b) Determines the right things are beingtrained; students if learned;andtraining transferredto the job, metthe unit's needs,and was efficient. (c) Provides supportto decisionmakers all levels. at (d) Is thoroughand comprehensive. (e) ts technicallycorrect. (f) Results in a quality productby applyingQC measures. (g) Complieswith TRADOC guidanceand policy. TD (h) Meetsmilestone requirements. Training developer role (2) Thetraining developer (GS-1750) a key player on is evaluation team, whousually leads and manages internal evaluation the efforts (within the proponent school) and, depending the purpose, on may lead the external evaluationefforts. This role includes-(a) Keeping peopleinvolved in the evaluationprocess all informed of progress, problemsencountered,developments, changes, andconstraints. (b) Beingresponsive. (c) Providingresults in a timely manner. (d) Keepingthe project management updated, plan appropriate.

Evaluation teamroles

1438
17

(e) Providing guidanceto the teammembers concerninghow

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 18 of 32

TRADOC Pam 350-70-4

(j) Conducts collection andanalysis; identifying data deficienciesandefficiencies. (k) Providing recommendations actual or potential to problems/deficiencies. (I) Sharing identified efficiencies with appropriate organizations. (m) Assisting with the center/school'sself-assessment efforts. (n) Ensuringall necessary corrective actions werecompleted. SME role (3) TheSME the content or technical expert on the team.This is SME or shouldbe, the master is, performer the action/activity being of evaluated. Nomatter whatthe job--a training developer,a combat/doctrine developer,or an instructor--the SME involved is directly in the evaluationfunction. TheSME specifically responsible is for the accuracyandcompleteness the technical content, andthe of comprehensiveness the content presented. The SME perform of may other duties the teamleader assigns, suchas data collection. (4) Depending the purposeof the evaluation, the composition on of the evaluationteam include, in addition to the training developer may andSME, following: the (a) Visual informationspecialists. (b) Installation supportperson. (c) SafetyOfficer. (d) Resourcemanagement personnel.

Additional members

1439
18

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 19 of 32

TRADOC Pam 350-70-4 Evaluation areas d. Evaluatorsshouldroutinely evaluateall aspectsofthe production, management, conductof the Armyeducation/training system. This and includes such areasas: (1) Enlisted,officer, warrant officer, andDA civilian training. (a) Evaluate quality of instruction, standardization, test objectives,etc., to prepare students perform to effectively. (b) Ensure horizontal andver[ical integration of tasks and training. (c) Ensure progressiveandsequentialtraining. (d) Evaluate quality of individual training strategiesand Career Development Model. (2) Unit Training Products. Evaluation areas (a) Evaluate quality of unit training strategies(unit long shor[-range training strategies). (b) ReviewCombined ArmsTraining Strategies (CATS), MissionTraining Plan (MTP),drills, exercises,WarfightingTSP,and Training Aids, Devices, Simulator and Simulations (TADSS) ensure-to o Product usefulness. ¯ Technical/doctrinalcontent accuracy. Designand development consistent with analysis data. are Effective training sequence. o Product meets needs. o Safety, environment, and risk assessment/management considerationsare incorporated. ¯ Deficiencies identified werecorrected. Note: Unit feedbackon MTP, Soldier Training Publication (STP)and TSPare supposed come to through the automated unit training management program(i.e., StandardArmyTraining System/Unit Training Management Configuration).

1440
19

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 20 of 32

TRADOC Pare 350-70-4 Arms (3)Combined Training Strategies. (a) ArmyTraining Strategy (Headquarters(HQ) TRADOC only). (b) Unit LongRange Training Strategy. Training Strategy. (c) Unit Short Range (d) Individual LongRange Training Strategy. Training Strategy. (e) Individual Short Range Training Strategy. (f) Self Development (4) Threat. Evaluateto determineschool's compliance with TRADOC regulations on threat, i.e., ensuringvalid, accurate, and consistentthreat portrayalis in the training andTDareas. (5) Doctrinal development. Evaluatethe organization, procedures and management the development of process, and the integration of doctrineinto training. (6) School evaluation system. Evaluate feedbacksystemsand how training institutions usefeedback verify/validate decisions,or to cause change. (7) Education/training management. Evaluate the management implementation educationandtraining. of (8) Training ResourceManagement. (a) Evaluatehowwell the schoolaligns resourceswith mission/training requirements toAccomplish training student workload standard. to Allocate resourcesto support Commanding General, TRADOC priorities. o Identify tradeoffs, andimpactsof these tradeoffs. o Considerefficiencies and economies. ¯ Applysavingsto meetunresourced priorities.

2O

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 21 of 32

TRADOC Pam 350-70-4 (b) Evaluatetraining resourcesinvolving programming and budgeting, manpower management, ammunition, equipment, and facilities. (c) Evaluate assignmentand management contracts. of (d) Evaluate management personnel assignments of and staff andfaculty training. (9) Training Requirements Analysis System(TRAS). Determine howwell the institution is managing TRAS implementation process and to include: (a) TRAS document implementation and updates. (b) Maintenance milestone schedules. of (c) Meetingprogramming budgeting windows and for resources. (10) Systems Approach Training process. Ensurethe to appropriateapplication of minimum essential elements the SAT of process. (11) Staff andfaculty training. Evaluatethe adequacy and relevancyof institution's training anddevelopment programs the staff for andfaculty. (12) Automationsystemswhich provide(a) Proponent approved functionality (operational) requirements. (b) Seamless share data. (c) Automated systemseasy to operate. (d) Multiple levels of help, to includeuseof program, technical assistance, andembedded training. (e) Thecapability to maintainthe integrity of currentdata. (f) Keydatabases the productioncycle (available through in the TDautomated support system).

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 22 of 32

TRADOC 350-70.-4 Pam Remember, evaluators must have access to decisionmakers,and the evaluation findings, conclusions, and recommendations be must credibleto be useful.
2-3.

Evaluation process description.Evaluations assessthe quality of training andTD. Evaluatorsconductevaluationsby analyzingthe current status of unit andindividual performance, training products,programs, andprocesses using a 5-phased by process(planning, collecting data, data analysis, providing recommendations/reportingfindings, and of following up on recommendations ensureimplementation).This to process produces valid andreliable results that identify training deficiencies. Findingsprovide the basis for corrective recommendations throughthe chain of command. Findingsalso identify those areasof the training program performing efficiently andeffectively. a. Evaluationefforts will determine whether: (1) Training andTDefforts haveaccomplished their intended purpose (i.e., the objectiveof the training hasbeen met). (2) The right soldier was trained(i.e., all students within fell target audience). (3) All studentsmetspecified entry-level standards. (4) Accreditation wasperformed properly, and accreditation data wasobtained. (5) The TDprocess met minimum essential requirements analysis, design, development, implementation. and (6) Training productsmetunit needs. (7) Soldiers' training metunit needs. (8) Appropriate quantity of training wasreceived. (9) Soldiers needed instruction they did not receive. (10) Soldiers receivedinstruction they did not need. (11) Training institutions are adheringto command training and educationguidance. Note: An overviewof eachevaluation phaseis provided in paragraph 2-3b throughf below.

Evaluation findings

443

22

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 23 of 32

TRADOC Pam 350-70-4 Planning overview b. Thefirst step in doing anythingis properplanning. Evaluations are no exception and require thoroughplanning. Some routine evaluation duties, suchas conducting classroom a observation,reviewinga test, or analyzing groupof studentend-of-course a critiques, do not require indepthevaluation plans. Explain procedures performingthese routine for duties in a local standardoperatingprocedure,or equivalentdocument. However, majorevaluations(i.e., in-depth evaluations schooltraining of program,products, or processes) require an evaluation plan--the end product of the planning phase.This phaseof the processis discussed in detail in ch__~ter 3. c. Thedata collection processinvolves determining whattype of data is required, whatdata to collect (student performance, student feedback, audit trails, supervisors,graduates, etc.), where/who is collected data from (source), howmuch,and howdata is collected (method/technique/ instrument). Next, structure anddevelopthe data collection method/instrument collect data. Thefinal step is to administerthe to instrument/technique collect the data. This phase the processis to of discussed in detail in ~. d. Analysisis the processof reviewing, synthesizing, summarizing, andprocessing evaluationdata collected to develop initial findings concerningthe issue being evaluated. Analysis reducesthe huge volumes rawdata collected into a series of initial findings. The of method usedwill depend the type of data collected. This phase on of the process discussed detail in ch_.~ter5. is in

Data collection overview

Analysis overview

Reporting e. After evaluationdata is collected andanalyzed,the next step is to findings and identify majorfindings andrecommendations. identified, write a Once recommend- report to include: ations overview (1) References. (2) Background/problem. (3) Purpose evaluation. of (4) Summary data collection procedures. of (5) Results - major findings, conclusions, andrecommendations. Thisphase the process discussed detail in ch__h~oter of is in 6.

1444
23

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 24 of 32

TRADOC Pam 350-70-4 Follow-up overview f. The follow-up phasedeterminesif recommendations were implemented. This phaseshould take place within oneyear of the evaluationapproval. This phase the processis discussed detail in of in ch__h~oter 7. g. Evaluatorsreceive feedback from students, supervisors,soldiers, instructors, commanders, training developers,andaffected organizations. Provide feedback the same.Theentire evaluation to processis based feedback,either formal or informal. Informal on feedback consistsof unsolicited information,whichis transmittedeither directly or indirectly to an actionoffice, verbally, or in writing. Formal feedback, however, solicited, specific in content,andtransmitted is throughchannelsby means a document of designed specifically for the purpose (i.e., a survey or questionnaire). Thefeedback system, therefore, includes either the instruments mechanisms to elicit or used the information,andthe means whichit is transmittedto the by appropriate action office. It is the evaluator'sresponsibility to ensure that the feedback system functioning, andevaluationresults are is relayed backto the appropriateaction point in the system.Formalor informationfeedback,via external andinternal evaluations, may drive future evaluation plans. Chapter 3 PlanningEvaluations
3-1.

Feedback system

Planningoverview, This chapter addressesthe planning phaseof the evaluationprocess.It will assist evaluatorsin planningan evaluation and creating the MasterEvaluation Plans (MEP) and Evaluation Project Management Plans. Addor delete additional actions, as the specific evaluationconducted requires. Evaluationsthat are not well-plannedor managed result in the collection of informationthat cannotbe trusted will (i.e., invalid andof no use). TheQuality Assurance Office (QAO) responsible for the development MEPs Evaluation Project of and Management Plans. However,involvementfrom other directorates and departments essential in their development. is Planningan evaluation. first step to any evaluationis planning, The whichis probably mostdifficult step. Everything the that follows depends howwell the evaluation is conceptualized.This entails how on well thoughts,ideas, and/orintuitions are expressed. the planning In process, assessall major goals/objectives, needs,resourcesand capabilities, and/orany other areas, whichare eventuallyrepresented in an evaluationplan. Thereare several reasons carefully plan an to evaluationanddocument plan in a written evaluationplan before the conductingan evaluation. Planning--

3-2.

1445

24

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 25 of 32

TRADOC Pam 350-70-4 Summarize data f. When data is summarized, simply condense data for analysis. the (1) Theeasiest and mostaccurate method summarizing for large amounts quantitative data is to use an automated of statistical program. (2) Summarizing qualitative data is much moredifficult accurately andeffectively summarize. Aid 350-70-4.5provides Job guidance summarizing for qualitative data. Analyzedata g. Beforeanalyzinganydata, ensure quantitative data is entered all into a computer,andall qualitative data is summarized condensed and into categories. Keep mindexactly whythe data is being analyzed in (i.e., identify whatspecific questions needed). are h. Statistical analysisis requiredfar analyzing quantitativedata. (Note: Statistics simply show things aboutthe data that are otherwise not seen.) There twotypesof statistics: are (1) Descriptive statistics give measurements simply that "describe" the data collected. This includes suchmeasures as: (a) Mean the averagevalue. (b) Range the extremevalues. (c) Standard deviation - the degreeto whichvaluesare dispersed. (2) Inferential statistics "infer" something aboutthe total population from whichthe data wascollected, usually by a random process. It includes such measurements as-(a) tests--a statistical analysis us to er som ed inf ething abouttwo different populations,based data collected from a sample on of eachof the populations. (b) Chi-square--astatistical test commonly to compare used observedor actual frequencywith expected hypothesized or frequency of twovariables. (c) Analysisof Variance---a statistical analysisusedto determine whether there are anystatistically significant mean differences between or moregroups, or one or morevariables, or two factors.

1446

43

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 26 of 32

TRADOC Pam 350-70-4 individual descriptions on howto use these various methods for analyzingdata are not provided.If the evaluationplan requires using unfamiliarstatistical analysis,consultwith a statistician or evaluator experienced statistics. in Interpret the analysis results i. Interpret the analysis in common senseterms, andbe able to explainthe results. Interpretationof analysisis oneof the mostdifficult steps in this phase evaluation. Keep mindthe purpose the of in of evaluation data results are interpreted. Annotate trends identified, as all andinclude in the final report. Qualitativedata is often considered less objective than quantitative data, but can providevery useful information, if procedures followed, especially when are looking at themes and relationshipsat the caselevel. Quantitativedata, though more scientific, requiresstatistical manipulation represent to findings. Aswith analysis,interpretationof results may call for the useof unfamiliarstatistical procedures. Solicit the assistance a statistician, of or moreexperienced evaluator, when usingunfamiliar statistical methods.
5-3.

Quality controlcriteria for evaluationanalysis. TD/taskproponents perform QCactions as an inherent part of the SATprocess. When analyzing data, ensure-a. Theintegrity of the rawdata, by verifying the completeness and understandabilityof responses. b. Accuracyof data summaries verified. was c. Dataanalysis wasappropriate the type of data collected. for d. Resultsof the analysis wereinterpreted to provide usable, accurate, and pertinent recommendations. Chapter6 PreparingEvaluation Reports

6-1,

Evaluationreport preparationoverview.This chapter provides guidance howto developa report, whichcapturesthe results of an on evaluation(phasefour of the evaluationprocess). Anevaluationreport will include findings, conclusions,and recommendations. chapter This wilt assist in preparing, staffing, anddistributing the report/recommendations.

447
44

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 27 of 32

TRADOC Pam 350-70-4

6-2.

Evaluation report descriptions.Formaland informal reports are an ongoing processduring the life of an evaluation; do not considerthem as just an "end of evaluation" product. Establish the primaryaudience during the planningphase,andwrite the report to that audience's interest. Thereport shouldbe conciseandpresented a waythat will in ensure evaluationinformationis usedto its fullest potential. The the report summarizes results of the evaluation the (i.e., findings, conclusions, and recommendations). report is providedto senior The leadersfor their informationandguidance. Reportpreparation steps. Evaluatorsshould perform the steps in table 6-1, whichprovides the generalproceduralguidance preparing for the evaluationreports. Table6-1 Ste
NO.

6-3.

[ an evaluation report Steps Prepare draft report, to includefindings, conclusions, the and recommendations. Developfindings, conclusions, and/or recommendations howthey relate to the objective of the on evaluation. Staff draft report for review/concurrence. internally, andwith Staff involved organizations. Obtain review andconcurrence of recommendations. Revise draft reportinto a final report. Obtain final approvalof recommendations. Provide to approving authority(i.e., decisionmaker) final approval final report, for of and recommendations. Distribute report/recommendations action. Distribute final for report with approvalauthority documentation all organizations to I with implementing requirements.

6"4,

Prepare draft report. Theevaluationreport is a vital document that summarizes results of the evaluation(i.e., findings, conclusions, the and recommendations). Fromthe findings, develop conclusions, and for eachidentified problemarea, provide specific recommendations for management consider. Thereare different waysof reporting to information, depending howthe report is used, the target audience, on andthe impact that the evaluation will have. Beas conciseandsimple as possible, while ensuring requiredinformationis included. all

1448

45

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 28 of 32

TRADOC Pam 350-70-4 Typesof evaluation reports a. Thelength and format of an evaluationreport can vary significantly. Factorsto considerare the audience whom report for the is intended,complexity the evaluation, and/orthe impactthe of evaluationresults will haveon the organization.For purposes of standardization,the reports are categorizedinto two types--Executive Summary Detailed Evaluation Report. and (1) TheExecutiveSummary a synopsisof the evaluation, and is .requires a responseto the evaluation recommendations. The Executive Summary a memorandum is designedfor senior leadership that includes: (a) Background. (b) Purpose objectives. and (c) Methods. (d) Major findings, recommendations, whois responsible and to implement lead the implementationo9 recommendations. (or (e) Suspense date for a responseto recommendations. Job Aid 350-70-4.6aprovides a format of an Executive Summary. For mostevaluations, the ExecutiveSummary sufficient. Keepon file the is supporting data anddocumentation required to explain and support findings. Detailed Evaluation Report (2) Detailed EvaluationReportsare lengthy, formal evaluation reports that explainin detail thosefindings includedin the Executive Summary listed above.A detailed report is requiredif: (a) Intended audience outsidethe installation, andthe is evaluationspecifics (to include supportingdata) cannotbe explained person. (b) Evaluation will havea very significant impact training/training resources. (c) Evaluationinvolves very complex data collection analysis methodologies require explanation, andare supported that with data. (3) In addition to the itemsin an executivesummary, detailed the report includes:

Executive Summary

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 29 of 32

TRADOC Pam 350-70-4 (a) An Executive Summary. (b) A discussion limitations. of (c) A list of assumptions. (d) Anidentification of EssentialElements Analysis. of (e) Appendices data summaries, for which mayinclude Data Collection Instruments, EvaluationPlan, etc. the (f) A memorandum requesting concurrence the director of the recommendations, organization responsible for implementing and recommendations. Formatthe memorandum similar to paragraphfive of the ExecutiveSummary cited in JA 350-70-4.6a.A format for a Detailed Reportis providedin JA 350-70-4.6b. Report preparation hints b. Whetheran Executive Summary a Detailed Evaluation Report or is prepared,keepthe report as brief as possible. Thefollowing suggestions assist in preparing will either report. (1) Onlyinclude information that the intendedreaderneeds. (2) Keep simple. it (3) Do not use acronyms unknown readers. to (4) Donot write at a readinglevel drastically above,or below,the reader. (5) Explain in simpletermsany complex data collection, analysis methodologies. (6) Stay focusedon the Essential Elements Analysis when of writing the report. Staff draft report for review/concurrence. a. Theprocedures staffing a draft report shouldbeginwithin the for organization, and then to involved organizations.Thesteps are as follows: (1) Distributethe draft report internally to individualswithin the office to reviewthe report content.

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 30 of 32

TRADOC Pam 350-70-4 (2) Provide director a copyof the draft report for the review/approvaL (3) Ensuredirector signs a memorandum to distribute organizations responsible for implementing recommendations. (Note: tf an Executive Summary prepared, director will sign it. If a is the DetailedEvaluationReportis prepared,the director signs the accompanying memorandum.) (4) Distribute draft report to all organizations affectedby the evaluation for their review andconcurrence nonconcurrence) (or recommendations made.Requesta responseback within 30 days. b. If nonconcurrence comments received, resolve any are disagreements prior to sending final report to the decisionmaker. the Usecoordinationmeetings,if necessary,to resolve disagreements. c. If no agreements reachedon nonconcurrences, are sendthe report with explanationsof all nonconcurrences the decisionmaker to for a final decision. Obtainfinal approval recommendations.this phase, a final of In report is providedto the decisionmaker (i.e., assistant commandant) for final approvalof the overall report andthe individual recommendations. Draft report a. Senda memorandumthe decisionmaker,along with the report to andcopiesof the responses receivedas a result of the staffing from the Staffing Draft Reportstep. Job Aid 350-70-4.6c provides a sample format for the memorandum to the decisionmaker. sent b. After the decisionmaker approved disapprovedthe has or recommendations, distribute the final report.
6-7.

Distribute report/recommendations action, and conduct for followupcheck. this phase, final report is distributed to all In the organizationsthat are responsiblefor implementing approved the recommendations, with a copy of the memorandum the from decisionmaker, showingfinal approval of recommendations. a. Providea suspense date for written responses responsible to organizations. b. Responses must include:

1451
48

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 31 of 32

TRADOC Pam 350-70-4 (1) Actionsthey havetakenthus far. (2) Milestones future actionstheywill take. for c. Inform the organizationthat a follow-up checkis conducted to ensurethat they havetaken the actions, and provide a predetermined time to conduct this check-up.
6=8,

Quality control criteria for preparing evaluation reports. Training development/task proponentsperform QCactions as an inherent part of the SAT process. When developingan evaluation report, ensure--a. Thefinal evaluationreport wasfocused the objectives of the on evaluation. b. Thefinal evaluationreport includes, at a minimum: (1) Findings. (2) Conclusions. (3) Recommendations. c. Reports werestaffed to all appropriateorganizations. d. Final report was approved chain of command by before distribution. e. Final report wasdistributedto all relevantorganizations (i.e., all organizationsthat are responsiblefor implementing approved the recommendations). Chapter7 Conducting Evaluation Follow-Ups

7-1,

Evaluationfollow-up overview.This chapter provides guidanceon howto conduct follow-up of the evaluationeffort (phase of the a 5 evaluationprocess).Duringthis phase,it is determined if recommendations as a result of the evaluation wereactually made implemented. chapterwill assist in conducting This follow-ups, and includesn ,, Preparing a follow-up evaluation. for o Conducting follow-up evaluation. the ¯ Preparingandstaffing the report.

1452

,~9

Case 1:00-cv-00644-NBF

Document 114-38

Filed 07/18/2005

Page 32 of 32

TRADOC Pam 350-70-4
7-2,

Follow-up description. The follow-up phaseis sometimes considered the mostimportantpart of the evaluationprocess,but often is the most overlooked. Conducting follow-up will determineif recommendations a wereimplemented, if they resulted in an improvement the and to training. This phase shouldtake place within oneyear of the evaluation approval. If the recommendations not followed up, do not conduct are the evaluation. a. Though decisionmaker signed off on the evaluation and the has approved recommendations, is still no guarantee the there that those responsible will implement recommendations. the Conducta follow-up to ensureimplementation. table 7-1 for proceduralsteps for See conducting follow-up. a Table7-1 Stepsfor conducting follow-up a NO. Steps 1~ Input action milestonesinto a tracking system.Collect responses from organizationsresponsiblefor implementing recommendations. Input milestonesinto a systemfor tracking actions taken. Conduct foliew-~p to ensure~ctions havebeentaken to the implement recommendations. . ..... 3. Prepareand write a foll0w-up report. 4. Staff the follow-upreport.

Follow-up steps

Input action milestones into tracking system

b. Duringthe reporting phase,determineactions the responsible organizations havetakenso far, andtheir plansfor future actions. To begin the follow-up, updateinformation from these organizationsto determinestatus of actions taken and planned.Theseactions are put into a tracking system. Thetracking system: (1) Includesall presentor future actions takenby organization. (2) Is automated simplify useof data collected. to (3) Hasthe projecteddate for the follow-up. Projecteddate will depend the milestonesestablishedfor the actions, andon local on policies andprocedures performing for follow-ups. (4) Provides audit trail on all actionstakenas a result of the an evaluation.

1453

50