% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/resample.R
\name{resample}
\alias{resample}
\title{Resample a Learner on a Task}
\usage{
resample(
  task,
  learner,
  resampling,
  store_models = FALSE,
  store_backends = TRUE,
  encapsulate = NA_character_,
  allow_hotstart = FALSE,
  clone = c("task", "learner", "resampling"),
  unmarshal = TRUE,
  callbacks = NULL
)
}
\arguments{
\item{task}{(\link{Task}).}

\item{learner}{(\link{Learner}).}

\item{resampling}{(\link{Resampling}).}

\item{store_models}{(\code{logical(1)})\cr
Store the fitted model in the resulting object=
Set to \code{TRUE} if you want to further analyse the models or want to
extract information like variable importance.}

\item{store_backends}{(\code{logical(1)})\cr
Keep the \link{DataBackend} of the \link{Task} in the \link{ResampleResult}?
Set to \code{TRUE} if your performance measures require a \link{Task},
or to analyse results more conveniently.
Set to \code{FALSE} to reduce the file size and memory footprint
after serialization.
The current default is \code{TRUE}, but this eventually will be changed
in a future release.}

\item{encapsulate}{(\code{character(1)})\cr
If not \code{NA}, enables encapsulation by setting the field
\code{Learner$encapsulate} to one of the supported values:
\code{"none"} (disable encapsulation),
\code{"try"} (captures errors but output is printed to the console and not logged),
\code{"evaluate"} (execute via \CRANpkg{evaluate}) and
\code{"callr"} (start in external session via \CRANpkg{callr}).
If \code{NA}, encapsulation is not changed, i.e. the settings of the
individual learner are active.
Additionally, if encapsulation is set to \code{"evaluate"} or \code{"callr"},
the fallback learner is set to the featureless learner if the learner
does not already have a fallback configured.}

\item{allow_hotstart}{(\code{logical(1)})\cr
Determines if learner(s) are hot started with trained models in
\verb{$hotstart_stack}. See also \link{HotstartStack}.}

\item{clone}{(\code{character()})\cr
Select the input objects to be cloned before proceeding by
providing a set with possible values \code{"task"}, \code{"learner"} and
\code{"resampling"} for \link{Task}, \link{Learner} and \link{Resampling}, respectively.
Per default, all input objects are cloned.}

\item{unmarshal}{\code{\link{Learner}}\cr
Whether to unmarshal learners that were marshaled during the execution.
If \code{TRUE} all models are stored in unmarshaled form.
If \code{FALSE}, all learners (that need marshaling) are stored in marshaled form.}

\item{callbacks}{(list of \link[mlr3misc:Callback]{mlr3misc::Callback} | \code{NULL})\cr
Callbacks to be executed during the resampling process.
See \link{CallbackResample} and \link{ContextResample} for details.}
}
\value{
\link{ResampleResult}.
}
\description{
Runs a resampling (possibly in parallel):
Repeatedly apply \link{Learner} \code{learner} on a training set of \link{Task} \code{task} to train a model,
then use the trained model to predict observations of a test set.
Training and test sets are defined by the \link{Resampling} \code{resampling}.
}
\note{
The fitted models are discarded after the predictions have been computed in order to reduce memory consumption.
If you need access to the models for later analysis, set \code{store_models} to \code{TRUE}.
}
\section{Stochasticity}{

Note that uninstantiated \code{\link{Resampling}}s are instantiated on the task, making
the procedure stochastic even in case of a deterministic learner.
}

\section{Predict Sets}{

If you want to compare the performance of a learner on the training with the performance
on the test set, you have to configure the \link{Learner} to predict on multiple sets by
setting the field \code{predict_sets} to \code{c("train", "test")} (default is \code{"test"}).
Each set yields a separate \link{Prediction} object during resampling.
In the next step, you have to configure the measures to operate on the respective Prediction object:

\if{html}{\out{<div class="sourceCode">}}\preformatted{m1 = msr("classif.ce", id = "ce.train", predict_sets = "train")
m2 = msr("classif.ce", id = "ce.test", predict_sets = "test")
}\if{html}{\out{</div>}}

The (list of) created measures can finally be passed to \verb{$aggregate()} or \verb{$score()}.
}

\section{Parallelization}{


This function can be parallelized with the \CRANpkg{future} or \CRANpkg{mirai} package.
One job is one resampling iteration.
All jobs are send to an apply function from \CRANpkg{future.apply} or \code{mirai::mirai_map()} in a single batch.
To select a parallel backend, use \code{\link[future:plan]{future::plan()}}.
To use \code{mirai}, call \code{mirai::daemons(.compute = "mlr3_parallelization")} before calling this function.
The \code{future} package guarantees reproducible results independent of the parallel backend.
The results of \code{mirai} will not be the same but can be made reproducible by setting a \code{seed} when calling \code{mirai::daemons()}.
More on parallelization can be found in the book:
\url{https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html}
}

\section{Progress Bars}{

This function supports progress bars via the package \CRANpkg{progressr}.
Simply wrap the function call in \code{\link[progressr:with_progress]{progressr::with_progress()}} to enable them.
Alternatively, call \code{\link[progressr:handlers]{progressr::handlers()}} with \code{global = TRUE} to enable progress bars
globally.
We recommend the \CRANpkg{progress} package as backend which can be enabled with
\code{progressr::handlers("progress")}.
}

\section{Logging}{


The \CRANpkg{mlr3} uses the \CRANpkg{lgr} package for logging.
\CRANpkg{lgr} supports multiple log levels which can be queried with
\code{getOption("lgr.log_levels")}.

To suppress output and reduce verbosity, you can lower the log from the
default level \code{"info"} to \code{"warn"}:

\if{html}{\out{<div class="sourceCode">}}\preformatted{lgr::get_logger("mlr3")$set_threshold("warn")
}\if{html}{\out{</div>}}

To get additional log output for debugging, increase the log level to \code{"debug"}
or \code{"trace"}:

\if{html}{\out{<div class="sourceCode">}}\preformatted{lgr::get_logger("mlr3")$set_threshold("debug")
}\if{html}{\out{</div>}}

To obtain a more informative traceback when an iteration fails, set
\code{options(mlr3.debug = TRUE)}. This disables parallelization via \CRANpkg{future}
so that errors are raised in the main process.
Note that results computed in debug mode use a different seeding mechanism and are \strong{not reproducible}.

To log to a file or a data base, see the documentation of \link[lgr:lgr-package]{lgr::lgr-package}.
}

\examples{
task = tsk("penguins")
learner = lrn("classif.rpart")
resampling = rsmp("cv")

# Explicitly instantiate the resampling for this task for reproducibility
set.seed(123)
resampling$instantiate(task)

rr = resample(task, learner, resampling)
print(rr)

# Retrieve performance
rr$score(msr("classif.ce"))
rr$aggregate(msr("classif.ce"))

# merged prediction objects of all resampling iterations
pred = rr$prediction()
pred$confusion

# Repeat resampling with featureless learner
rr_featureless = resample(task, lrn("classif.featureless"), resampling)

# Convert results to BenchmarkResult, then combine them
bmr1 = as_benchmark_result(rr)
bmr2 = as_benchmark_result(rr_featureless)
print(bmr1$combine(bmr2))
}
\seealso{
\itemize{
\item \code{\link[=as_benchmark_result]{as_benchmark_result()}} to convert to a \link{BenchmarkResult}.
\item Chapter in the \href{https://mlr3book.mlr-org.com/}{mlr3book}:
\url{https://mlr3book.mlr-org.com/chapters/chapter3/evaluation_and_benchmarking.html#sec-resampling}
\item Package \CRANpkg{mlr3viz} for some generic visualizations.
}

Other resample:
\code{\link{ResampleResult}}
}
\concept{resample}
