# maestro 1.2.0

### Breaking changes

- `show_network()` and `MaestroSchedule$show_network()` are fully removed. Use `get_network()` instead.

### New features

- `build_schedule()` gains a `cores` argument, enabling parallel parsing of pipeline scripts via `furrr`. Usage mirrors `run_schedule()`: set `future::plan(future::multisession)` before calling `build_schedule(cores = n)`.

- New `@maestroMap` tag enables dynamic fan-out (scatter). Use `@maestroInputs upstream_name` combined with `@maestroMap` on the downstream pipeline. An empty tag (`@maestroMap`) iterates over every element of the upstream return value. Providing one or more `.input$field` expressions (space-separated) selects which fields to scatter over, keeping the full list available as `.input` in each branch. Multiple expressions zip across several fields simultaneously, similar to `purrr::pmap()`.

- Fan-in (collect): a downstream pipeline declared with `@maestroInputs collect(upstream1, upstream2, ...)` gathers the return values of all listed upstream pipelines into a single named list passed as `.input`. Works with both static multi-source fan-in and dynamic fan-out followed by fan-in (`@maestroMap` → `collect()`). In `get_status()`, `input_run_id` shows comma-separated IDs of all contributing upstream runs and `lineage` joins distinct upstream paths with `&` (e.g. `letter_a&letter_b->ab`). Note that `collect()` is not an actual maestro function, but syntactic sugar.

- New `@maestroLabel` tag allows adding of metadata to a pipeline with key-value pairs. Multiple instances of `@maestroLabel` can be used to add any number of key-value pairs:
  ```
  #' @maestroLabel domain transportation
  #' @maestroLabel database enterprise1
  ```
  `@maestroLabel` is not to be confused with `@maestroFlags` which is an arbitrary list of unnamed or unkeyed strings.

- Added helper function `get_labels()` to extract all labelled key-value pairs as a data.frame from a schedule.

# maestro 1.1.1

### Bug fixes

- Fixed the calculation of `next_run`

- Pipeline functions with required arguments not supplied via `resources` now give an informative error message (e.g. `` missing argument `my_arg` - pass via `resources = list(my_arg = ...)` ``) instead of the cryptic `unused argument (NA = NULL)`.

# maestro 1.1.0

### Breaking changes

- The default value for `maestroStartTime` is no longer fixed to 2024-01-01 00:00:00 but is now dynamic based on the frequency of the pipeline. Daily or more frequent is midnight of the current build day, weekly is beginning of the current week, and so on for lower frequencies.

### New features

- `get_network()` is a new standalone function equivalent to `schedule$get_network()`, following the same pattern as `get_status()`, `get_flags()`, etc. Replaces the deprecated `show_network()`.

- `get_run_sequence()` now returns an `is_primary` column: `TRUE` for pipelines that are not downstream nodes in a DAG (i.e., root or standalone pipelines), `FALSE` for downstream DAG nodes.

- `get_run_sequence()` gains an `include_skipped` argument (default `TRUE`). Set to `FALSE` to exclude pipelines tagged with `@maestroSkip` from the output.

- `maestroStartTime` now accepts partial anchor formats, making it easier to express natural cycle points without picking a specific date:
  - `Mon HH:MM:SS` (weekday + time) — for weekly pipelines, e.g. `Mon 04:00:00` runs every Monday at 4am.
  - `DD HH:MM:SS` or `DD` (month-day + optional time) — for monthly pipelines, e.g. `15 04:00:00` runs on the 15th of every month at 4am.

### Deprecated functionality

- `suggest_orch_frequency()` is deprecated with no replacement.

- `show_network()` (and `MaestroSchedule$show_network()`) is deprecated due to
  its reliance on `DiagrammeR`, which brings in a large number of little-used dependencies. Use `get_network()` to retrieve the pipeline dependency edge list directly.

### Minor changes

- Some optimizations in schedule building and running.

- Schedules no longer store run_sequences, but are rather built lazily as needed. This greatly improves the portability of schedule objects making it more practical in production to cache a schedule as an RDS to avoid re-parsing tags.

- Added option `maestro.check_datetime_override` which takes a POSIXct and overrides the `check_datetime` argument of `run_schedule()`. This should only be used for development and debugging - NOT in production.

### Bug fixes

- Fixed issue where unique identifiers from `get_artifacts()` were not the same as the corresponding `run_id` from `get_status()` for fan-in style DAGs.

- Fixed alignment of DAG cli output

# maestro 1.0.1

### Minor changes

- Log timestamps now use a %Y-%m-%d %H:%M:%S format.

- Some efficiency improvements to `build_schedule()` particularly for high frequency pipelines.

- Console output for conditional pipelines now uses the `(?)` symbol after the pipeline name instead of before it to help with alignment of DAG tree representation.

### Bug fixes

- Running in multicore (i.e., specifying `cores > 1` in `run_schedule()`) is more stable.

- Logging no longer adds unnecessary new lines in between log statements.

- `get_schedule()` is no longer missing pipelines with inputs.

- Fixed regression where cli output was not representing DAG structure.

# maestro 1.0.0

### Breaking changes

This release brings a handful of breaking changes to functions for getting statuses and pipeline artifacts. Core scheduling and orchestrating functions remain largely unchanged with the exception of some bug fixes - meaning that pipelines should continue to run normally. However, users who rely on `get_status()`, `get_artifacts()`, and `last_run_errors()` should carefully inspect their code for breaking changes prior to upgrading in production. 

- `get_status()` better reflects statuses of pipelines executed multiple times in a single run due to DAG structures where branches converge on a single downstream pipeline. As a result, pipelines executed multiple times now have multiple rows in the status table corresponding to each distinct DAG lineage. Specific changes to output of `get_status()` are described below.

- in `get_status()` each pipeline execution has its own `run_id` which is a random 6-char string. Columns added to the output of `get_status()` include: 
  - run_id: the unique id of the run
  - input_run_id: the unique id of the run that inputted into the pipeline (NA if root pipeline)
  - lineage: string representing the full lineage of pipelines leading to the current execution

- `get_status()` column `success` is initialized to be `NA` if the pipeline has not been invoked. This better characterizes the unknown success of a pipeline that has not yet run.

- `get_artifacts()` now returns artifacts for each unique run_id - allowing for potentially multiple sets of artifacts from one pipeline if it's executed multiple times in a single orchestration run.

- `last_run_errors()` and friends now output vectors for pipelines that are executed multiple times in a single orchestration run.

### New features

- New `get_run_sequence()` function for getting a data.frame of scheduled run times for all pipelines in a `MaestroSchedule` optionally including DAGs. 

### Minor changes

- Pipelines with `@maestroFrequency` of minutely or secondly can now be run with specific hours, days, etc. In general, time specifiers like `@maestroHours`, `@maestroDays`, etc. are less restrictive about the base frequency of the pipeline.

### Bug fixes

- Fixed issue where a pipeline error on one branch of a branching DAG interrupted execution on subsequent independent branches.

- Fixed bug where if `check_datetime` argument to `run_schedule` was a Date it caused a mismatch in the sequence checking (#173). 

# maestro 0.7.1

### Bug fixes

- Fixed issue where info messages with curly brackets interrupted pipeline execution.

# maestro 0.7.0

### New features

- Conditional pipelines: A new `maestroRunIf` tag that takes an R expression returning a boolean and conditionally runs a pipeline. These can use inputs from upstream DAG pipelines, values from the orchestrator via `run_schedule(..., resources = list())`, or any arbitrary TRUE/FALSE expression.

### Minor changes

- `maestro::invoke()` gains argument `log_to_console` allowing it to print messages, warnings, and errors to the console.

- `last_run_errors()` and friends now return named lists where the name corresponds to the pipe name.

# maestro 0.6.3

### Bug fixes

- Fixed issue with scheduler on certain frequencies over 1 month when the start time was less than 1 year ago.

# maestro 0.6.2

### Minor changes

- Console output of `run_schedule()` simplified. Counts of skipped pipelines removed to avoid confusion with `@maestroSkip` tag. 'Total' now refers to the total number of pipelines invoked in a run - not all pipelines in the project.

### Bug fixes

- `maestro::invoke()` only runs the selected pipeline or DAG rather than accidentally running all pipelines in the schedule (#161)

# maestro 0.6.1

### Minor changes

- `maestro::invoke()` on a DAG pipeline successfully executes the full DAG (#156).

### Bug fixes

- `maestro::invoke()` now properly passes resources (arguments) to pipelines (#157).

- Number of errors reported in `get_status()` is now accurately reported.

# maestro 0.6.0

### New features

- New `maestroFlags` tag that allows passing arbitrary pipeline tags, which is useful for documentation and labeling. Tags are now accessible via `get_flags()`.

- New function `get_slot_usage` to help identify the number of pipelines running on a scheduled time slot.

- New `maestroPriority` tag that determines the order in which pipelines on the same schedule instance execute. Uses integer values from 1-N where 1 is the highest priority.

- `maestroStartTime` now accepts a timestamp formatted like HH:MM:SS. This is useful for pipelines running on a daily or hourly frequency because the date is often arbitrary in those cases (#143). 

### Removed features

- Arguments `logging` and `log_file` in `run_schedule()`, which were deprecated in maestro 0.5.0 are now fully removed.

### Minor changes

- Reduced the cached length of schedule sequences to 2 years in advance. (This only affects workflows where a schedule is cached instead of rebuilt each orchestration run).

### Bug fixes

- `n_pipelines` attribute of `<MaestroPipelineList>` now corresponds correctly to the number of pipelines.

- `create_pipeline` no longer adds extra line breaks where optional tags would be.

# maestro 0.5.3

### Bug fixes

- Fallback within `build_schedule()` to address CRAN check.

# maestro 0.5.2

### Minor changes

- Pipeline schedule sequences are now stored internally inside of `<MaestroPipeline>` objects instead of generated during `run_schedule()`. This has implications when caching a schedule as the sequence only goes out 3 years in advance.

- Performance improvements to `run_schedule()`.

### Bug fixes

- Specifying `maestroHours`, `maestroDays`, `maestroMonths` now correctly adopts the time zone specified in `maestroTz` (#141).

- When using non UTC time zones, the presence of Daylight Savings Time in the `maestroStartTime` is used to adjust the sequence so that invocations occur on the same time interval.

- Other time zone fixes to deal with differing `maestroTz` and system time checks.

- `maestroHours`, was only valid when `maestroFrequency` was specified as 'hourly', but now '1 hour' is also acceptable (same applies for other specifier tags).

# maestro 0.5.1

### Bug fixes

- Updated error messages and documentation to reflect that a `maestroFrequency` of multiple weeks (e.g., 2 weeks) is invalid.

# maestro 0.5.0

### New features

- Pipeline errors, warnings, and messages can now be printed to the console using `run_schedule(log_to_console = TRUE)`. These logs will be interwoven between messages created by `maestro` (#130).

- `run_schedule()` gains `log_to_file` argument to specify whether to log to a file (replaces `logging` and `log_file` arguments).

### Deprecated functionality

- `run_schedule()` arguments `logging` and `log_file` are deprecated. Use `log_to_file = TRUE` to log to a generic maestro.log file or `log_to_file = '[path-to-your-log-file]'` to log to a specific text file.

### Minor changes

- `run_schedule()` now warns if the unit of `orch_frequency` is lower frequency than the highest frequency pipeline in the project.

- `run_schedule()` enforces a minimum `orch_frequency` of 1 year (e.g., '2 years' or more no longer valid).

### Bug fixes

- Message and warning counts are now properly displayed in the status and output of `run_schedule()`, as well as in `last_run_messages()` and `last_run_warnings()`, even if they are below the `maestroLogLevel`.

- Fixed display of `run_schedule()` to have more accurate next run times for pipelines. This issue was evident when running orchestrator on a frequency of daily or lower.

# maestro 0.4.1

### Bug fixes

- Fixed issue where pipelines with a dependency would run on a time schedule even if the upstream pipeline didn't run (and vice versa).

- Fixed output of next scheduled pipelines to better reflect DAG structures.

# maestro 0.4.0

### New features

- Directed acyclic graph (DAG) pipelines - where the output of one pipeline can feed into another - are now available using the `maestroOutputs` and `maestroInputs` tags. Pipelines that input into a downstream pipeline should use the `maestroOutputs` tag. Pipelines that receive input from an upstream pipeline should use the `maestroInputs` tag (#98).

- New function `show_network` for visualizing the connections between pipelines that are connected in a DAG.

- `MaestroSchedule` gains new methods `get_network()` (returns a data.frame) and `show_network()` (returns a visualization using {DiagrammeR}).

- Added catch-all `maestro` tag to identify a function as a pipeline without specifying other configurations.

### Minor changes

- skip argument added to `create_pipeline` to allow for interactive creation of pipelines that default to skip.

### Bug fixes

- Fixed issue with `suggest_orch_frequency` when using different styles of frequency (e.g., 1 day vs. daily) in a single schedule.

- Fixed issue where pipeline sourcing failures were appearing as successful runs in status outputs.

# maestro 0.3.0

This version refactors much of the code base to rely on R6 classes for pipelines and schedules. Pay careful attention to the breaking changes to see how existing code may be impacted.

### Breaking changes

- Schedules are now represented as an R6 object of class `<MaestroSchedule>`. `build_schedule()` returns a MaestroSchedule object that can be passed to `run_schedule()` as normal. To access the schedule table run `get_schedule()`.

- `run_schedule()` no longer returns a list of `$status` and `$artifacts` but now returns/modifies the MaestroSchedule object. Status can be accessed using `get_status(schedule)` and artifacts via `get_artifacts(schedule)`

- `suggest_orch_frequency()` now takes a `<MaestroSchedule>` object.

- Data `example_schedule` removed from the package.

- Skipped pipelines are no longer shown in the CLI output of `run_schedule()`.

- It is now required that all pipeline names are unique. The names of each maestro pipeline function must be unique across the project to support the implementation of DAGs. `build_schedule()` will abort if any non-unique names are detected.

### New features

- Added functions `get_schedule()`, `get_status()`, and `get_artifacts()` for interacting with `<MaestroSchedule>` objects.

- Added function `invoke()` to instantly run a pipeline in a schedule.

### Bug Fixes

- Error messaging is clearer when running functions that wrap around purrr iterators (#115).

# maestro 0.2.0

### New features

- New tags `maestroHours`, `maestroDays`, and `maestroMonths` allows running of pipelines on specific hours of day, days of week, days of month, or months of year (#100).

- `maestroFrequency` tag now accepts the values hourly, daily, weekly, biweekly, monthly, quarterly, and yearly. Argument `orch_frequency` to `run_schedule()` also accepts these values.

### Minor changes

- Changed from `example_schedule` data the pipeline with a schedule of 1 minute to 30 minutes in keeping with best practices for minimum pipeline frequency.

- `suggest_orch_frequency` now uses the smallest interval between any two pipelines (#99).

### Bug Fixes

- Error messages on unintentional overwrites from `create_*()` functions correctly reference name of path or directory that was to be overwritten.

- Fixed cli output of `run_schedule()` to not show skipped pipelines in the next run portion.

# maestro 0.1.2

### Bug Fixes

- Fixed cli output to correctly handle counting of successful runs when pipelines are skipped.

- Performance improvements to `build_schedule()` (#101).

# maestro 0.1.1

### Breaking changes

- Creater functions `create_pipeline()` and `create_maestro` no longer have default arguments for the path to where the scripts are created. Users must explicitly define these paths.

- Argument `log_file` in `run_schedule()` no longer defaults to `./maestro.log` but instead defaults to `NULL`.

### Minor changes

- Creater functions `create_*` now take a boolean `overwrite` argument to make the overwriting of existing pipelines, projects, and orchestrators more explicit.

# maestro 0.1.0

- Initial CRAN submission

# maestro 0.0.4

### Bug fixes

- Fixed the output of the next run for pipelines (#90)

# maestro 0.0.3

### Breaking changes

- `run_schedule()` now returns a list with status and artifacts instead of just a data.frame of the status. Artifacts are any values returned from pipelines. Pipelines that return nothing will have no artifacts.

### Major changes

- New helper function `suggest_orch_frequency()` to provide a suggestion of what frequency to use for the orchestrator.

### Minor changes

- Start and end times are now reported from functions that result in an error in single core only (#82).

### Bug fixes

- CLI output from `run_schedule()` now correctly outputs the total number of pipelines (#81) and correctly outputs number of errors.

# maestro 0.0.2

### Breaking changes

- maestroFrequency tag now adheres to a more human-readable format like "1 day", "2 hours", "4 weeks", etc.

- `orch_frequency` argument in `run_schedule()` also takes more human-readable format identical to maestroFrequency tag.

- maestroInterval tag removed

- `orch_interval` argument to `run_schedule()` removed.

- `create_maestro()` and `create_orchestrator()` now use the argument `type` instead of `extension` for defining what script type to use for the orchestrator.

- Changed `last_parsing_errors()` to `last_build_errors()`; changed functions of the form `last_runtime_*()` to `last_run_*()`.

### Major changes

- Additional columns added to the output of `run_schedule()`: `pipeline_started` and `pipeline_ended` to indicate the start and end times of a pipeline execution; `next_run` to indicate when the next run should be based on the frequency of the pipeline and orchestrator.

- Pipelines now show as skipped if they are not scheduled.

- Added hex logo

### Minor changes

- Backend improvements to schedule checking

- Timestamps are formatted to specified time zone.

- `run_schedule()` cli output suggests to use `last_run_errors()` or `last_run_warnings()` if any errors or warnings were found.
