A year is long enough to go wrong slowly
A one-year analytics study rarely fails in a single moment. It drifts. The data arrives late and nobody adjusts the plan, the senior analyst from the proposal appears at kickoff and then fades from the calls, the method changes quietly to fit what the data allows, and in month eleven a draft report arrives that answers a slightly different question than the one you asked. Every monthly update was pleasant. The year was lost anyway.
Most of that is decided when you choose the partner. A proposal for a study is a set of promises about people, method, data and time, and some firms make those promises in ways you can check while others make them in ways you cannot. Your job during selection is to tell the difference before you sign.
This piece is for the program manager, analyst lead or nonprofit director who has to choose. It covers what separates strong partners for a twelve-month analysis, the red flags that predict trouble, the questions to ask in an interview, and a simple way to score candidates.
The people who will actually do the work
A study is the judgment of two or three people applied over a year. Everything else, the firm's size, its brand, its other contracts, matters only insofar as it affects those people. So the first question for any proposal is who, by name, will do the thinking.
Look for named individuals in the roles that carry judgment: the study lead, the senior methodologist or analyst, and whoever will build the data pipeline. For each, look for what they have personally done that resembles your study, and how many hours they are assigned to it. A strong lead with 80 hours across a year is an advisor, not a lead.
Then look at the shape of the hours. A sound twelve-month study usually has senior people heavily involved at the design stage, steadily present through analysis, and back in force for interpretation and writing. A plan where senior hours appear only at the start and the end, with junior analysts carrying the middle alone, is a plan where nobody senior will see the problems as they happen.
Ask for commitment in the contract. Designate the key roles, and require approval before anyone named is replaced, with a substitute of equal or better qualifications. Firms that intend to staff the work as proposed will accept that without argument.
A method plan you can argue with
A useful method plan is specific enough that you could disagree with it. It names the data, the comparison, the model or test, the main threats to the answer, and what the result will look like when it is done.
For an evaluation of a program's effect, that means saying what the comparison group is and why it is credible. For a forecasting study, it means saying what will be forecast, at what level, against what baseline, and how accuracy will be judged. For a descriptive study, it means saying what population, what measures, and what precision is achievable with the data available.
Watch for plans built entirely of method names. "We will apply advanced machine learning and statistical techniques" is a list of words. "We will estimate the effect using a difference-in-differences comparison against counties that adopted the program two years later, test the parallel trends assumption on the five prior years, and report the estimate with confidence intervals" is a plan. The second can be wrong, which is exactly what makes it useful.
Good firms also say what could go wrong with their own method and what they will do if it does. A method plan that cannot be wrong cannot be checked, and a plan that cannot be checked is a promise to figure it out later on your budget.
A data access plan, not a data access hope
In most year-long studies, data access is the largest single schedule risk. The strongest proposals treat it that way.
A real data access plan lists each data source, who controls it, what agreement or approval is needed to use it, how it will be transferred or accessed, and how long that is expected to take. It names what the firm needs from you and by when. It says what the firm will do in the meantime if access is delayed, such as building the pipeline against a sample or a synthetic file with the same structure.
It also addresses handling. Where will the data be stored, who will have access, how will personal information be protected, and what happens to the data at the end of the study. If your data involves health, education or criminal justice records, the firm should already know which rules apply and should ask you about the agreements in place.
A schedule with interim deliverables
A twelve-month study with one deliverable at month twelve is a year of trust with no checkpoints. Ask for a schedule with interim deliverables that let you see the work while there is still time to correct it.
A Twelve-Month Study With Checkpoints, Illustrative
Illustrative schedule. The data quality report and the preliminary briefing are the two checkpoints that most often save a study.
The data quality report in the early months deserves emphasis. It is the moment when the firm tells you, in writing, what the data can and cannot support. A firm willing to deliver that report early is a firm that will tell you bad news while it is still useful.
Reproducibility: code and a data list, delivered
At the end of a year, you should own the means to rerun the analysis, not only the report. That means the analysis code, a list of every data source with the version or extract date used, a data dictionary, and a README that lets a competent analyst reproduce the main tables and figures.
Ask candidates how they will deliver this and what tools the code depends on. A firm that works in open languages and commits to a shared repository from month one makes reproduction easy. A firm that relies on proprietary platforms can still be a good choice, but you should know before signing that rerunning the work will require that platform.
Reproducibility also protects the quality of the work during the year. Analysts who know their code will be read and rerun write cleaner, better documented analysis. It is a quality control step disguised as a delivery requirement.
How often you will hear from them
Communication cadence is easy to agree on and easy to let lapse. Ask for it in writing: a short standing call every two weeks, a written monthly status covering progress, risks and decisions needed, and a briefing at each major deliverable.
The content matters more than the frequency. A useful status names the decisions the firm needs from you and the risks that have changed since last time. A status that only lists completed tasks tells you activity, not direction. Ask candidates to show you an example status report from a past engagement with sensitive details removed.
Evidence of similar analysis
The best predictor of whether a firm can do your study is whether its people have done a study like it. Similar does not mean the same agency or the same topic. It means the same kind of analytic problem: the same type of data, the same design challenge, the same kind of audience for the result.
Ask for two or three examples, each with the question, the data, the method, the deliverable and a reference you can call. Published reports or public summaries are strong evidence, because you can read them. Call the references and ask what went wrong and how the firm handled it. Every year-long study has a problem; the reference will tell you whether this firm surfaced it or hid it.
Red flags and what to ask instead
These patterns show up in proposals and interviews often enough to be worth naming, along with the question that tests each one.
| Red flag | Why it matters | What to ask instead |
|---|---|---|
| Senior names with few hours, or names "to be determined" | The classic setup for bait-and-switch staffing | "How many hours will each named person spend, and in which months?" |
| A method section made of technique names | Nothing in it can be checked or disagreed with | "Walk me through how you would answer research question one, step by step." |
| Junior-heavy hours across the middle months | Nobody senior sees problems as they happen | "Who reviews the analysis each month, and how?" |
| "We will work with your team to obtain data" | The largest schedule risk has no plan | "What do you need from us, by when, and what do you do if it is late?" |
| One deliverable at the end | No checkpoint until it is too late to correct course | "What will I see in writing by month three?" |
| Examples that are capability claims, not studies | No evidence the people have done this kind of work | "Show me a report your proposed lead wrote, and give me that client's contact." |
None of these flags is disqualifying alone. Several together usually mean the proposal was written by a business development team and the study will be staffed by whoever is available in month two.
Questions to ask in the interview
If you can, interview the people who will do the work, not the people who sell it. Ask the proposed lead and senior analyst to attend, and direct the questions to them.
- What is the first thing you would want to know about our data, and why?
- What is the most likely reason this study gives an unclear answer, and what would you do about it?
- Describe a past study where the data did not support the planned method. What did you change?
- How many hours will you personally spend on this study, and on which parts?
- What are you working on now, and how much of your time will it take over the next year?
- What will we receive in the first ninety days?
- How will we be able to rerun your analysis after the study ends?
- When would you tell us that a finding was going against what we expected?
Listen for specifics and for candor. A good answer to the second question names a real weakness in your data or design. An answer that says the method will handle it is an answer from someone who has not thought about your study yet.
A simple scoring approach
You do not need an elaborate scheme. Choose a handful of criteria, weight them, score each candidate on a short anchored scale, and write one or two sentences explaining each score. Keep price as a separate consideration so you can see the tradeoff plainly rather than burying it in the totals.
How We Would Weight the Criteria
Our weighting, out of 100, based on our judgment of what predicts a useful study. It is not measured data. Adjust it to your study and fix your weights before reading proposals.
For the scale, three or four levels with written anchors is enough. For example: zero for absent, one for stated without specifics, two for specific and plausible, three for specific, plausible and supported by evidence such as a past report or a named reference. Anchors keep two reviewers from scoring the same proposal far apart for reasons neither can explain.
If you are a public agency, your procurement rules and the evaluation criteria stated in your solicitation govern how you evaluate. Use an approach like this to shape those criteria before the solicitation goes out, then evaluate exactly as the solicitation says.
Frequently asked questions
The named people who will do the work and how many hours each will spend on it. A study is the judgment of a few individuals over many months. Method, data planning and schedule matter, but they are only as good as the people executing them, so start there and make the key roles contractual.
Ask for named people with hours by month, interview those people rather than the sales team, and designate them as key personnel with replacement allowed only by approval and only with equal or better qualifications. Firms that intend to staff the work as proposed accept these terms easily.
At minimum, a design memo in the first month, a data quality report once data arrives, a preliminary analysis briefing around mid-year, an interim findings memo, and a draft report with a review window before the final. Each gives you a chance to correct course while there is still time.
Because without them you own a report, not an analysis. With the code, a data list and a README, another analyst can rerun, check and extend the work after the study ends. Knowing the code will be read also tends to improve the quality of the analysis during the year.