01Research design and problem statement
We start from your research gap and questions, then decide which data, which type of analytics and which validation strategy can answer them, and how big data analysis fits alongside conventional methods such as surveys, experiments or interviews.
02Data access, governance and ethics
Before analysis, we check that you are allowed to use the data for your purpose: licences and terms of service, ethics approval, consent, and data protection law such as the UK and EU GDPR. Personal data are minimised, pseudonymised or anonymised where possible and stored securely. We also consider bias in how the data were collected and who is missing from them.
03Data preparation and integration
Cleaning, transforming and joining data from CSV files, relational (SQL) and NoSQL databases, JSON, XML and APIs, with every step scripted rather than done by hand, so the path from raw data to analysis file can be rerun.
04Distributed processing
When data outgrow one machine, frameworks such as Apache Hadoop and Apache Spark split storage and computation across a cluster. Hadoop combines a distributed file system (HDFS), a resource manager (YARN) and the MapReduce programming model; Spark keeps intermediate data in memory and offers Spark SQL, MLlib for machine learning and Structured Streaming. Many PhD datasets do not need a cluster at all, and a well-written script on one machine or a cloud notebook is simpler and easier to reproduce. Your mentor helps you choose the lightest tool that works.
05Train, validation and test data
For predictive models, data are split before any modelling: a training set to fit models, a validation set (or cross-validation within the training data) to tune hyperparameters and compare models, and a held-out test set used once, at the end, to estimate performance on new data. For time series, splits respect time order, for example with rolling-origin evaluation; for grouped data, such as several records per patient, all records from one group stay in the same split.
06Cross-validation, leakage and overfitting
k-fold cross-validation gives a more stable estimate than a single split; nested cross-validation keeps hyperparameter tuning from biasing the estimate. Data leakage, where information from the test data or the future reaches the model (for example scaling or feature selection on the full dataset before splitting), produces results that look excellent and do not generalise. Overfitting is checked by comparing training and validation performance and with learning curves, and controlled with regularisation, simpler models, early stopping and more data. A confusion matrix describes errors; it does not on its own detect overfitting.
07Evaluation metrics
Metrics follow the problem. For regression: MAE, RMSE and R-squared. For classification: precision, recall, F1-score and ROC AUC, with precision-recall AUC preferred when one class is rare, and calibration when predicted probabilities are used for decisions. Accuracy alone misleads on imbalanced data. For clustering: internal measures such as the silhouette coefficient, together with a check that clusters make sense in your field. Results are compared against a simple baseline and reported with uncertainty, such as cross-validation spread or bootstrap confidence intervals.
08Statistical inference at scale
With millions of records, almost any difference is statistically significant. We help you report effect sizes and confidence intervals rather than relying on p-values from ANOVA, t-tests, chi-square tests or correlations, and to correct for multiple testing where many hypotheses are tested.
09Interpretation and reporting
Your mentor helps you interpret model outputs, explain them in plain terms for your results and discussion, and justify your choice of method against alternatives. Where they suit your study, interpretability methods such as feature importance, partial dependence or SHAP values support the explanation. Reporting follows guidance such as TRIPOD+AI for clinical prediction models where it applies.
10Reproducibility
Your analysis is kept as version-controlled code (Git) with a README, a record of software and package versions (for example a requirements or environment file, renv or a Docker image), fixed random seeds and, where data can be shared, a data availability statement. Notebooks such as Jupyter and R Markdown or Quarto are useful for documenting analyses, as long as they run cleanly from top to bottom.